A text-condition guided image expansion method based on diffusion model and terminal

Abstract

The present invention discloses a method and terminal for image expansion guided by text conditions based on a diffusion model. The method receives an original image input by a user, generates expansion text conditions for the original image using a pre-trained multimodal large language model, performs feature encoding on the original image to generate image features, and performs feature encoding on the expansion text conditions to generate text features. The image features and text features are input into a pre-trained latent diffusion model based on a dual-UNet network, and an expansion image is generated based on the latent diffusion model. The present invention introduces a multimodal large language model to generate text conditions, so that the expansion content conforms to semantic logic, thereby improving rationality and aesthetics. The dual-UNet structure processes text semantics and original image features in modules, avoiding excessive cross-attention burden of a single UNet and enhancing semantic coherence between the expansion content and the original image. Through data enhancement and a dual-UNet architecture, expansion of any pixel in the image is supported, breaking through the expansion range limitation of the generative adversarial network.

Description

Technical Field

[0001] The present invention relates to the technical field of model-based image expansion, and in particular to a text condition-guided image expansion method and terminal based on a diffusion model. Background Art

[0002] Currently, there are two main technical routes in the field of image expansion: one is based on the generative adversarial network method, which constructs an adversarial training framework between the generator and the discriminator, so that the generator learns the distribution characteristics of the boundary area of ​​the input image to generate a visually coherent expansion area; the other is based on the diffusion model method, which uses the encoder to map the sub-image area of ​​the input image to the latent space, uses the diffusion process to gradually denoise, and uses the decoder to reconstruct the latent space features into the expansion image.

[0003] However: (1) methods based on generative adversarial networks cannot perform arbitrary pixel expansion of images;

[0004] (2) The denoising process of the diffusion model is less guided by the original sub-graph information, and the semantic connection between the expanded content and the original content is incoherent;

[0005] (3) The generated external expansion content lacks rationality and the final generated result is not beautiful enough. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an image expansion method and terminal guided by text conditions based on a diffusion model, which supports expansion of any pixel point and is reasonable, beautiful, and semantically coherent.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A text condition-guided image expansion method based on a diffusion model comprises the following steps:

[0009] S1. Receive an original image input by a user, and generate an expanded text condition for the original image using a pre-trained multimodal large language model;

[0010] S2. Feature encoding is performed on the original image to generate image features, and feature encoding is performed on the expanded text conditions to generate text features;

[0011] S3. Input the image features and the text features into a pre-trained latent diffusion model based on a dual UNet network, and generate an outward-expanded image based on the latent diffusion model.

[0012] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0013] A terminal for image expansion based on text conditional guidance of a diffusion model comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-described method for image expansion based on text conditional guidance of a diffusion model are implemented.

[0014] The beneficial effects of the present invention are as follows: a text-condition guided image expansion method and terminal based on a diffusion model of the present invention introduces a multimodal large language model to generate text conditions, replacing the discarded text guidance in the traditional diffusion model, so that the expanded content conforms to the semantic logic, improving rationality and aesthetics; the dual UNet structure processes text semantics and original image features in modules, avoiding the excessive cross-attention burden of a single UNet, and enhancing the semantic coherence between the expanded content and the original image; through data enhancement and the dual UNet architecture, it supports the expansion of any pixel of the image, breaking through the expansion range limitation of the generative adversarial network. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a simplified flowchart of an image expansion method guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0016] Figure 2 This is an example diagram of the overall method flow architecture of an image expansion method guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of the structure of a hybrid expert module of an image expansion method guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0018] Figure 4 This is a diagram illustrating the interaction structure of a Q-former module of an image expansion method guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0019] Figure 5 A connection diagram of Q-former and LLM decoder training for an image expansion method guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0020] Figure 6 This is a structural example diagram of an image expansion terminal guided by text conditions based on a diffusion model according to an embodiment of the present invention;

[0021] Description of labels:

[0022] 1. A text-condition guided image expansion terminal based on a diffusion model; 2. A processor; 3. A memory. DETAILED DESCRIPTION

[0023] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0024] Please refer to Figures 1 to 5 , a text condition guided image expansion method based on a diffusion model, comprising the steps of:

[0025] S1. Receive an original image input by a user, and generate an expanded text condition for the original image using a pre-trained multimodal large language model;

[0026] S2. Feature encoding is performed on the original image to generate image features, and feature encoding is performed on the expanded text conditions to generate text features;

[0027] S3. Input the image features and the text features into a pre-trained latent diffusion model based on a dual UNet network, and generate an outward-expanded image based on the latent diffusion model.

[0028] From the above description, it can be seen that the beneficial effects of the present invention are: the present invention's method and terminal for image expansion guided by text conditions based on a diffusion model introduce a multimodal large language model to generate text conditions, replacing the text guidance discarded in the traditional diffusion model, so that the expanded content conforms to semantic logic, improving rationality and aesthetics; the dual UNet structure processes text semantics and original image features in modules, avoiding the excessive cross-attention burden of a single UNet, and enhancing the semantic coherence between the expanded content and the original image; through data enhancement and a dual UNet architecture, it supports expansion of any pixel in the image, breaking through the expansion range limitation of the generative adversarial network.

[0029] Furthermore, for the original image, generating an expanded text condition using a pre-trained multimodal large language model includes the following steps:

[0030] The original image is input into the multimodal large language model, image features are extracted through the CLIP visual encoder, and outward text conditions are generated through the hybrid expert module and the Q-former module.

[0031] As can be seen from the above description, CLIP extracts local block features of the image, ConvNeXt extracts global semantics, and the hybrid expert module dynamically weights and fuses local and global information, enabling the model to understand the overall structure and local details of the image.

[0032] Q-former strengthens the cross-modal alignment of image features and text through image-text comparison, generation, and pairing tasks, ensuring that the generated expanded text conditions accurately match the semantics of the original image, thereby improving the guidance accuracy of subsequent diffusion models.

[0033] Furthermore, the potential diffusion model adopts a dual UNet network, including a frozen UNet network with frozen original parameters and a pre-trained trainable UNet network;

[0034] Step S3 includes the steps of:

[0035] Injecting the text features into the cross-attention module of the frozen UNet network, inputting the image features into the cross-attention module of the trainable UNet network, and combining the features of the two layers of UNet through a zero convolution layer to gradually denoise and generate an outward expansion latent variable;

[0036] The expanded latent variable is decoded into an expanded image by a decoder.

[0037] As can be seen from the above description, the frozen UNet leverages the pre-trained text-to-image capability to focus on text semantics to guide the expansion direction; the trainable UNet combines original image features (such as edges and colors) to constrain the visual coherence of the expanded content; the zero convolution layer dynamically adjusts the interaction strength between the two UNets, balancing semantic guidance and content constraints to prevent the expanded area from deviating from the original image style or causing semantic discontinuity.

[0038] Furthermore, the training data construction of the multimodal large language model includes the following steps:

[0039] Obtain a certain amount of image data, generate corresponding text descriptions for each image using an open source model, and input the text embedding features generated by the text embedding model to obtain image-text pairs;

[0040] For each image-text pair, data augmentation processing is used to generate an enhanced image-text pair to obtain a training dataset;

[0041] The data augmentation processing options include instance cropping of image data, random cropping with a preset ratio, and random horizontal or vertical mixing of image-text pairs.

[0042] As can be seen from the above description, instance cropping forces the model to learn the expansion logic of "local instance → global scene" (such as cropping a sheep's head to generate a complete sheep); random cropping with preset proportions forces the model to infer large areas of missing content, improving the flexibility of the expansion range; image-text hybrid enhancement simulates the needs of multi-scene fusion and enhances the model's processing of complex expansion semantics (such as splicing "sheep" and "grassland" images to generate mixed text descriptions).

[0043] Furthermore, the training of the multimodal large language model includes the steps of:

[0044] For each image-text pair after data augmentation, the image data is input into the CLIP visual encoder to perform image encoding to obtain image embedding features, and the original image data is extracted through a convolutional neural network to obtain global features;

[0045] Inputting the image embedding feature and the global feature into a hybrid expert module to generate a new image embedding feature;

[0046] The image embedding features and the text embedding features obtained by the hybrid expert module are input into the Q-former module, and the first stage of training is carried out through three tasks: image-text comparison learning, text generation, and image-text pairing.

[0047] The vector output by the Q-former module is linked to a layer of MLP and then input into the decoder of the large language model. The cross entropy loss of the context window is calculated and the parameters of the Q-former model are fine-tuned.

[0048] As can be seen from the above description, the hybrid expert module dynamically allocates feature weights through K MLP expert networks to adapt to the feature fusion requirements under different cropping ratios (such as enhancing global feature weights when cropping at large ratios). The Q-former optimizes image-text alignment in the first stage, and in the second stage, the large language model decoder calibrates the text logic, ensuring that the generated expanded text conforms to the image semantics and has natural language fluency.

[0049] Furthermore, the loss parameter of the Q-former model is:

[0050] ;

[0051] in , ,..., is a given text token sequence, and θ is the parameter of the model.

[0052] As can be seen from the above description, the cross-entropy loss based on the context window forces the text token sequence generated by the Q-former to conform to the language prior knowledge (such as grammar and semantic coherence), avoiding logical breaks or semantic contradictions in the expanded text and improving the quality of the text condition.

[0053] Furthermore, the hybrid expert module uses K multi-layer perceptrons as the expert network :

[0054] ;

[0055] in, represents the i-th new image embedding feature output by the hybrid expert module, represents the i-th image embedding feature generated by the CLIP visual encoder, and Cat represents the tensor splicing operation used for the weight calculation of the average expert network:

[0056] ;

[0057] in, It is a multi-layer perceptron that converts tensors from feature dimensions into K dimensions.

[0058] As can be seen from the above description, the gating network dynamically calculates expert weights, allowing the model to adaptively select experts based on input features (such as the correlation between image blocks and global semantics), improving the flexibility and efficiency of feature fusion; sparse activation of some experts (non-fully connected) reduces computational complexity while enhancing the model's ability to cluster multimodal features.

[0059] Furthermore, the construction of the potential diffusion model includes:

[0060] Obtain an open source UNet network, generate a copy of the open source UNet network, freeze parameters of the open source UNet network to serve as a frozen UNet network, and use the copy of the open source UNet network for training as a trainable UNet network;

[0061] The frozen UNet network and the trainable UNet network are connected by zero convolution and use different inputs. The frozen UNet network inputs the latent noise vector z t , the trainable UNet network input latent noise vector z t , the potential representation z of the mask image m And the downsampled binary mask m;

[0062] The calculation formula for each layer interaction between the frozen UNet network and the trainable UNet network is:

[0063] ;

[0064] ;

[0065] in, The potential diffusion model is represented by The output features of the layer, and Represent the n-th layer output features of the frozen UNet network and the trainable UNet network, represents the number of steps of noise addition in the diffusion process, represents the zero convolutional layer, is a parameter that adjusts the interaction strength between the two networks. represents the encoder in the variational autoencoder VAE, Indicates that the original image is padded with zeros on all sides to obtain a mask image of the target expanded image size. E I represents the image features generated by feature encoding of the original image, ET Indicates the text features generated by feature encoding of the expanded text condition.

[0066] As can be seen from the above description, the frozen UNet retains the pre-trained generation capability, while the trainable UNet clearly distinguishes the original image from the expanded area through the masked latent representation and binary mask, guiding the diffusion process to prioritize repairing the expanded area; the zero-convolution layer linearly combines the two UNet features to implement a dual denoising mechanism of "semantic guidance (frozen UNet) + structural constraints (trainable UNet)", improving the rationality of the generated content.

[0067] Furthermore, the potential diffusion model adopts the noise loss function of the diffusion model as the loss function:

[0068] ;

[0069] in, The potential diffusion model is represented by The output feature of the layer, t represents the number of steps of noise addition in the diffusion process, is random noise sampled from a standard normal distribution, z t is the latent noise vector obtained by adding noise in the tth step, It means taking the mathematical expectation.

[0070] As can be seen from the above description, by minimizing the L2 distance between the predicted noise and the actual noise, the dual UNet is forced to accurately model the inverse distribution of the diffusion process, ensuring the high fidelity of the outward expansion latent variable. The optimization goal of the noise loss is consistent with the physical meaning of the diffusion model, ensuring that the generated outward expansion image is visually coherent and natural, without obvious noise artifacts.

[0071] The present invention provides a text condition-guided image expansion method and terminal based on a diffusion model, which are applicable to semantic-guided expansion based on a diffusion model in the field of image editing.

[0072] Please refer to Figures 1 to 5 , embodiment 1 of the present invention is:

[0073] A text condition-guided image expansion method based on a diffusion model comprises the following steps:

[0074] S1. Receive an original image input by a user, and generate an expanded text condition for the original image using a pre-trained multimodal large language model;

[0075] For the original image, generating an expanded text condition using a pre-trained multimodal large language model includes the following steps:

[0076] The original image is input into the multimodal large language model, image features are extracted through the CLIP visual encoder, and outward text conditions are generated through the hybrid expert module and the Q-former module.

[0077] The training data construction of the multimodal large language model includes the following steps:

[0078] Obtain a certain amount of image data, generate corresponding text descriptions for each image using an open source model, and input the text embedding features generated by the text embedding model to obtain image-text pairs;

[0079] For each image-text pair, data augmentation processing is used to generate an enhanced image-text pair to obtain a training dataset;

[0080] The data augmentation processing options include instance cropping of image data, random cropping with a preset ratio, and random horizontal or vertical mixing of image-text pairs.

[0081] In this embodiment, a large number of color images I are first collected. G , generate a corresponding text description T for each image through the existing open source model (other open source multimodal language models, used to describe image content) G , and input the text embedding model to get the text embedding E G Instance cropping refers to using the detection model to anchor all instances in the image, randomly selecting an instance bounding box to crop the sub-image to obtain I crop , the preset ratio random cropping means significantly reducing the cropped sub-image I crop With the original image I G The ratio of the original image is 20%-50%, and only 20%-50% of the original image is retained. The text embedding E of the corresponding image under the two cropping schemes G Remain unchanged. Random horizontal or vertical mixing of image-text pairs means that two images are spliced ​​together horizontally or vertically, and their corresponding text embeddings are mixed proportionally. During the training process, one of the three enhancement methods is randomly selected for the original image to obtain the corresponding mixed data enhanced image. With text embedded E G Paired-input multimodal large language model training.

[0082] The training of the multimodal large language model includes the following steps:

[0083] For each image-text pair after data enhancement processing, the image data is input into the CLIP visual encoder to perform image encoding to obtain image embedding features, and the original image data is extracted through a convolutional neural network to obtain global features.

[0084] The image embedding feature and the global feature are input into a hybrid expert module to generate a new image embedding feature.

[0085] In this embodiment, the image obtained by data enhancement Input CLIP visual encoder to perform image encoding, and then embed the resulting image into feature V patch , and the convolutional neural network ConvNeXt extracts image I G The obtained global feature V global , input together into the mixed expert module to get the new image embedding ,like Figure 3 The specific calculation formula is as follows:

[0086] ;

[0087] ;

[0088] The hybrid expert module uses K multi-layer perceptrons (MLPs) as the expert network :

[0089] ;

[0090] in, represents the i-th new image embedding feature output by the hybrid expert module, represents the i-th image embedding feature generated by the CLIP visual encoder, and Cat represents the tensor splicing operation used for the weight calculation of the average expert network:

[0091] ;

[0092] in, It is a multi-layer perceptron that converts tensors from feature dimensions into K dimensions.

[0093] The role of Q-former is to use a learnable vector V Q Link image features and text features, the structure is as follows Figure 4 As shown. The image obtained by the hybrid expert module is embedded into the feature With the text embedding feature E G The Q-former module is input and trained in the first phase through three tasks: image-text comparison learning, text generation, and image-text pairing.

[0094] like Figure 5 As shown, the vector V output by the Q-former module is Q After connecting a layer of MLP, the decoder of the large language model is input, the cross entropy loss of the context window is calculated, and the parameters of the Q-former model are fine-tuned.

[0095] The loss parameter of the Q-former model is:

[0096] ;

[0097] in , ,..., is a given text token sequence, and θ is the parameter of the model.

[0098] S2. Feature encoding is performed on the original image to generate image features, and feature encoding is performed on the expanded text conditions to generate text features;

[0099] S3, inputting the image features and the text features into a pre-trained latent diffusion model based on a dual UNet network, and generating an outward expansion image based on the latent diffusion model;

[0100] The potential diffusion model adopts a dual UNet network, including a frozen UNet network with frozen original parameters and a pre-trained trainable UNet network;

[0101] Step S3 includes the steps of:

[0102] Injecting the text features into the cross-attention module of the frozen UNet network, inputting the image features into the cross-attention module of the trainable UNet network, and combining the features of the two layers of UNet through a zero convolution layer to gradually denoise and generate an outward expansion latent variable;

[0103] The expanded latent variable is decoded into an expanded image by a decoder.

[0104] In this embodiment, text features and image features are injected into different UNets respectively. It aims to accept the guidance of text conditions of the multimodal large language model while making full use of the original sub-graph to generate more reasonable extended content and more aligned with the original image semantics. Specifically, the text condition C T Through CLIP text encoder Get text feature E T , injected into the cross attention module of each layer of the frozen UNet, the original image I G After the image encoder Get image feature E I , after aligning the tensor dimension with the latent space feature dimension d through the multi-layer perceptron, it is injected into the cross attention module of each layer of the trainable UNet, as shown in Figure 2 As shown. It can be expressed as the following calculation:

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] The W here Q 、W K 、W V is a learnable linear layer, , They are the output of the crisscross attention layer of the frozen UNet network with frozen weights and the output of the crisscross attention layer of the trainable UNet network.

[0112] The construction of the potential diffusion model includes:

[0113] An open source UNet network is obtained, and a copy is generated for the open source UNet network. The parameters of the open source UNet network are frozen as a frozen UNet network. The copy of the open source UNet network is used for training as a trainable UNet network.

[0114] In this embodiment, Figure 2 As shown, the UNet network in the pre-trained potential diffusion model is used , the architecture contains residual blocks, self-attention layers and cross-attention layers, and generates a copy of it. The two UNets are connected through zero convolution. During the training process, only the parameters in the copy UNet network are adjusted, and the parameters of the original UNet are frozen.

[0115] The frozen UNet network and the trainable UNet network are connected by zero convolution and use different inputs. The frozen UNet network inputs the latent noise vector z t , the trainable UNet network input latent noise vector z t , the potential representation z of the mask image m And the downsampled binary mask m;

[0116] The calculation formula for each layer interaction between the frozen UNet network and the trainable UNet network is:

[0117] ;

[0118] ;

[0119] in, The potential diffusion model is represented by The output features of the layer, and Represent the n-th layer output features of the frozen UNet network and the trainable UNet network, represents the number of steps of noise addition in the diffusion process, Z represents the zero convolution layer, is a parameter that adjusts the interaction strength between the two networks. represents the encoder in the variational autoencoder VAE, Indicates that the original image is padded with zeros on all sides to obtain a mask image of the target expanded image size. E I represents the image features generated by feature encoding of the original image, E T Indicates the text features generated by feature encoding of the expanded text condition.

[0120] The potential diffusion model adopts the noise loss function of the diffusion model as the loss function:

[0121] ;

[0122] in, The potential diffusion model is represented by The output feature of the layer, t represents the number of steps of noise addition in the diffusion process, is random noise sampled from a standard normal distribution, z t is the latent noise vector obtained by adding noise in the tth step, It means taking the mathematical expectation.

[0123] In the forward process of the diffusion model, Gaussian noise is sampled and added to the data sample z0, resulting in a noisy sample z at time step t t In the reverse process, the model needs to predict the noise intensity at the current time t.

[0124] In this example, the training strategy employed BLIP2 to generate textual conditions for training images. UNet from StableDiffusion 1.5 was used as the pre-trained model. The entire latent diffusion model was trained for 200,000 iterations on eight NVIDIA GeForce RTX4090 GPUs with a learning rate of 1e-5. The gradient accumulation strategy from the Accelerate package was applied to increase the equivalent batch size, and an 8-bit Adam optimizer was used to reduce GPU memory usage. To enable the model to handle conditions without text, textual conditions were randomly dropped with a probability of 20% during training.

[0125] Please refer to Figure 6, the second embodiment of the present invention is:

[0126] A text-condition guided image expansion terminal 1 based on a diffusion model includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps of the text-condition guided image expansion method based on the diffusion model are implemented.

[0127] In summary, the present invention provides a method and terminal for image expansion guided by text conditions based on a diffusion model. The method introduces a multimodal large language model to generate text conditions, replacing the discarded text guidance in the traditional diffusion model, so that the expanded content conforms to semantic logic, improving rationality and aesthetics. The dual UNet structure processes text semantics and original image features in modules to avoid the excessive cross-attention burden of a single UNet and enhance the semantic coherence between the expanded content and the original image. Through data enhancement and the dual UNet architecture, the expansion of any pixel in the image is supported, breaking through the expansion range limitation of the generative adversarial network.

[0128] This paper leverages the text-guided nature of pre-trained text-to-image diffusion models to introduce a multimodal large language model to generate textual conditions for image expansion tasks, thereby generating more reasonable and aesthetically pleasing expanded images. A dual-UNet diffusion model is constructed, in which textual conditions and image features are injected into separate UNets. This allows the diffusion model denoising process to incorporate information from the original subgraph, generating more semantically coherent expanded content.

[0129] Incorporating textual conditions. Currently, most diffusion model-based out-drawing methods utilize pre-trained text-to-image models. However, these methods discard textual conditions during the generation process, resulting in reduced performance of the text-based graph model. In contrast, the present invention employs a dedicated multimodal large language model to generate reasonable textual conditions for image out-drawing, enabling optimal generation performance and producing more reasonable and aesthetically pleasing image out-drawing results.

[0130] Guided by original image information. Understanding the inherent patterns of the original image is a key factor in image expansion tasks. By constructing a dual-UNet diffusion model and injecting original image features into the cross-attention layer of an additional UNet, the diffusion model's denoising process incorporates original image information, reducing the linguistic incoherence between the original image and the generated expanded content. Simultaneously, two networks process textual conditions and original image features separately, alleviating the burden of a single UNet's cross-attention layer on processing multimodal data.

[0131] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A text-condition guided image expansion method based on a diffusion model, characterized in that: Including steps: S1. Receive an original image input by a user, and generate an expanded text condition for the original image using a pre-trained multimodal large language model; S2. Feature encoding is performed on the original image to generate image features, and feature encoding is performed on the expanded text conditions to generate text features; S3, inputting the image features and the text features into a pre-trained latent diffusion model based on a dual UNet network, and generating an outward expansion image based on the latent diffusion model; The potential diffusion model adopts a dual UNet network, including a frozen UNet network with frozen original parameters and a pre-trained trainable UNet network; Step S3 includes the steps of: Injecting the text features into the cross-attention module of the frozen UNet network, inputting the image features into the cross-attention module of the trainable UNet network, and combining the features of the two layers of UNet through a zero convolution layer to gradually denoise and generate an outward expansion latent variable; Decoding the expanded latent variable into an expanded image through a decoder; The construction of the potential diffusion model includes: Obtain an open source UNet network, generate a copy of the open source UNet network, freeze parameters of the open source UNet network to serve as a frozen UNet network, and use the copy of the open source UNet network for training as a trainable UNet network; The frozen UNet network and the trainable UNet network are connected by zero convolution and use different inputs. The frozen UNet network inputs a latent noise vector, and the trainable UNet network inputs a latent noise vector, a potential representation of the mask image, and a downsampled binary mask. The calculation formula for each layer interaction between the frozen UNet network and the trainable UNet network is: ； ； in, The potential diffusion model is represented by The output features of the layer, and Represent the n-th layer output features of the frozen UNet network and the trainable UNet network, represents the number of steps of noise addition in the diffusion process, z t is the latent noise vector obtained by adding noise in step t, represents the zero convolutional layer, is a parameter that adjusts the interaction strength between the two networks. represents the encoder in the variational self-encoder VAE, m represents the binary mask after downsampling, Indicates that the original image is padded with zeros on all sides to obtain a mask image of the target expanded image size. represents the mask image features, E I represents the image features generated by feature encoding of the original image, E T Indicates the text features generated by feature encoding of the expanded text condition.

2. The image expansion method guided by text conditions based on a diffusion model according to claim 1, characterized in that: For the original image, generating an expanded text condition using a pre-trained multimodal large language model includes the following steps: The original image is input into the multimodal large language model, image features are extracted through the CLIP visual encoder, and outward text conditions are generated through the hybrid expert module and the Q-former module.

3. The image expansion method guided by text conditions based on a diffusion model according to claim 1, characterized in that: The training data construction of the multimodal large language model includes the following steps: Obtain a certain amount of image data, generate corresponding text descriptions for each image using an open source model, and input the text embedding features generated by the text embedding model to obtain image-text pairs; For each image-text pair, data augmentation processing is used to generate an enhanced image-text pair to obtain a training dataset; The data augmentation processing options include instance cropping of image data, random cropping with a preset ratio, and random horizontal or vertical mixing of image-text pairs.

4. The image expansion method guided by text conditions based on a diffusion model according to claim 3, characterized in that: The training of the multimodal large language model includes the following steps: For each image-text pair after data augmentation, the image data is input into the CLIP visual encoder to perform image encoding to obtain image embedding features, and the original image data is extracted through a convolutional neural network to obtain global features; Inputting the image embedding feature and the global feature into a hybrid expert module to generate a new image embedding feature; The image embedding features and the text embedding features obtained by the hybrid expert module are input into the Q-former module, and the first stage of training is carried out through three tasks: image-text comparison learning, text generation, and image-text pairing. The vector output by the Q-former module is linked to a layer of MLP and then input into the decoder of the large language model. The cross entropy loss of the context window is calculated and the parameters of the Q-former model are fine-tuned.

5. The text condition guided image expansion method based on diffusion model according to claim 4, characterized in that: The loss function of the Q-former model is: ； in , ,..., is a given text token sequence, and θ is the parameter of the model.

6. The text condition guided image expansion method based on diffusion model according to claim 4, characterized in that: The hybrid expert module uses K multi-layer perceptrons as the expert network : ； in, represents the i-th new image embedding feature output by the hybrid expert module, represents the i-th image embedding feature generated by the CLIP visual encoder, and Cat represents the tensor splicing operation used for the weight calculation of the average expert network: ； in, is the weight of the j-th expert network output, It is a multi-layer perceptron that converts tensors from feature dimensions into K dimensions, V global Represents global features.

7. The text condition guided image expansion method based on diffusion model according to claim 1, characterized in that: The potential diffusion model adopts the noise loss function of the diffusion model as the loss function: ； in, represents the output features of the Nth layer of the latent diffusion model, is random noise sampled from a standard normal distribution, It means taking the mathematical expectation.

8. A text-condition guided image expansion terminal based on a diffusion model, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image expansion method guided by text conditions based on a diffusion model are implemented as described in any one of claims 1 to 7.

Images