Method and device for generating image based on multi-scale feature fusion

Abstract

The invention provides a method and device for generating an image based on multi-scale feature fusion, and the method comprises the steps: obtaining text features and target noise, the text features being used for providing content guidance for generating the image; the target noise is processed through a trained target encoder, a potential vector fused with multi-scale features is generated, the multi-scale features are used for indicating details and overall structure information of different levels in a generated image, and the potential vector is used for indicating image features of the generated image; the text features are fused into the potential vectors through a trained target fusion model, fusion features are generated, and the fusion features are used for achieving generation of a text guide image; and decoding the fusion features into a final image through a trained target decoder. The quality of the generated image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a method and device for generating images based on multi-scale feature fusion. Background Art

[0002] Image generation is an important research direction in the field of artificial intelligence, and its application scenarios are very wide, including art creation, game design, product design, medical image generation, etc. With the development of deep learning technology, image generation technology has made significant progress and can generate more and more realistic images.

[0003] Currently, models can be used to generate text images so that the images can express the meaning of the text. However, the model has some problems in the quality of image generation. It is easy to lose detailed information or the overall structure is unclear, resulting in poor image generation quality. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present application provides a method and device for generating an image based on multi-scale feature fusion.

[0005] In a first aspect, the present application provides a method for generating an image based on multi-scale feature fusion, the method comprising:

[0006] Acquire text features and target noise, wherein the text features are used to provide content guidance for generating an image;

[0007] Processing the target noise through a trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate details and overall structural information at different levels in the generated image;

[0008] The text feature is fused into the latent vector through a trained target fusion model to generate a fusion feature, wherein the fusion feature is used to realize the generation of a text-guided image, and the latent vector is used to indicate the image feature of the generated image;

[0009] The fused features are decoded into the final image by the trained target decoder.

[0010] Optionally, processing the target noise by a trained target encoder to generate a latent vector fused with multi-scale features includes:

[0011] Inputting the target noise into a trained target encoder;

[0012] Performing a convolution operation on the target noise through a plurality of convolution kernels of different sizes to extract feature information of different scales, wherein the convolution kernels of different sizes correspond to different receptive fields;

[0013] The feature information of multiple different scales is fused by using a preset fusion method;

[0014] The fused multi-scale features are convolved and compressed to generate the latent vector.

[0015] Optionally, the target fusion model is a target diffusion model, and the text feature is fused into the latent vector by the target fusion model to generate the fusion feature, including:

[0016] Inputting the text features and the latent vector into the diffusion model, wherein the diffusion model includes a plurality of time steps;

[0017] Calculating cross attention according to the text feature and the potential vector corresponding to the current time step, and updating the potential vector according to the result of the cross attention;

[0018] The updated potential vector is passed to the next time step until the cross-attention calculation of all time steps is completed, and the result of the cross-attention output of the last time step is obtained;

[0019] The result of the cross attention output at the last time step is used as the fusion feature.

[0020] Optionally, the formula for calculating the cross attention includes:

[0021]

[0022] Among them, Attention(Q, K, V) is the cross attention, Q represents the potential vector corresponding to the current time step, K represents each word in the text feature, V represents the encoding vector of each word, and d k is the dimension of K, and T represents transpose.

[0023] Optionally, the training process of the target encoder, the target decoder and the target fusion model includes:

[0024] Get multiple sample image-text pairs;

[0025] Training a VAE model according to the sample image-text pair so that the initial encoder learns to extract and fuse multi-scale features and generate image latent vectors, and the initial decoder learns to decode, wherein the VAE model includes an initial encoder and an initial decoder;

[0026] When the parameters of the VAE model are fixed, the initial fusion model is trained in conjunction with the VAE model so that the initial fusion model learns to fuse text features into the latent vector.

[0027] Optionally, the initial fusion model is an initial diffusion model, and training the initial fusion model in conjunction with the VAE model includes:

[0028] Converting a sample image into a latent vector using the target encoder;

[0029] Performing a forward diffusion process of the initial diffusion model on the latent vector, wherein the forward diffusion process is adding noise to the sample image;

[0030] In the reverse denoising process of the initial diffusion model, the noise is removed to restore the original image features, and the text features are integrated into the image features using a cross-attention mechanism at each time step;

[0031] The trained target fusion model is obtained by adjusting the loss function, wherein the loss function is used to measure the difference between the generated image and the sample image, and whether the generated image conforms to the description of the sample text.

[0032] Optionally, the target noise is random noise or a target image.

[0033] In a second aspect, the present application provides a device for generating an image based on multi-scale feature fusion, the device comprising:

[0034] An acquisition module, used for acquiring text features and target noise, wherein the text features are used for providing content guidance for generating an image;

[0035] A processing module, used for processing the target noise through a trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate details and overall structural information at different levels in the generated image, and the latent vector is used to indicate image features of the generated image;

[0036] A fusion module, used for fusing the text feature into the latent vector through a trained target fusion model to generate a fusion feature, wherein the fusion feature is used to realize the generation of a text-guided image;

[0037] The decoding module is used to decode the fused features into a final image through a trained target decoder.

[0038] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor is used to implement any of the steps of the method for generating an image based on multi-scale feature fusion when executing the program stored in the memory.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any step of the method for generating an image based on multi-scale feature fusion is implemented.

[0042] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0043] The method provided in the embodiment of the present application first extracts feature information of different scales and fuses it into a latent vector, so that the latent vector can contain details of different levels of the image and overall structural information, and then fuses the text features into the latent vector to achieve text-guided image generation. Finally, the final image output by decoding can not only finely control the details and overall structure of the image, but also ensure that the generated image conforms to the text description, thereby improving the quality of images generated based on text. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A schematic diagram of the hardware environment of a method for generating an image based on multi-scale feature fusion provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of a method for generating an image based on multi-scale feature fusion provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the overall process of generating an image based on multi-scale feature fusion provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of a device for generating an image based on multi-scale feature fusion provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0053] In order to solve the problem mentioned in the background technology, according to one aspect of an embodiment of the present application, an embodiment of a method for generating an image based on multi-scale feature fusion is provided.

[0054] Optionally, in the embodiment of the present application, the above-mentioned method for generating an image based on multi-scale feature fusion can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 through a network, the user inputs text through the terminal, and the server generates a corresponding image based on the text information. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0055] The embodiment of the present application provides a method for generating an image based on multi-scale feature fusion, which can be applied to a server.

[0056] The following will describe in detail a method for generating an image based on multi-scale feature fusion provided by an embodiment of the present application in combination with a specific implementation manner. Figure 2 As shown, the specific steps are as follows:

[0057] Step 201: Acquire text features and target noise, wherein the text features are used to provide content guidance for generating an image.

[0058] The user inputs a user-provided text description through the terminal, and the server uses the text encoder in the preset model to convert the input text information into one or more text features Vtext. These text features can capture the semantic information in the text, such as color, shape, scene and other descriptive elements. The text features provide content guidance for generating images to ensure that the generated images meet the user's description. Among them, the text encoder can use BERT or RoBERTa, and this application does not specifically limit the type of text encoder.

[0059] The server also obtains target noise, which can be random noise or a target image input by the user. If the target noise is random noise, then it is possible to create a diverse and realistic new image based on the text description, for example, a new image can be generated based on the text "a white kitten playing on the grass"; if the target noise is random noise, then it is possible to modify the image content in a targeted manner based on the text description while retaining the original image characteristics, for example, based on the text "a white kitten playing on the grass", the color of the kitten in the original image "a black kitten playing on the grass" can be changed to white.

[0060] Step 202: The target noise is processed by the trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate the details and overall structural information at different levels in the generated image, and the latent vector is used to indicate the image features of the generated image.

[0061] The server uses the trained target encoder to process the target noise into a low-dimensional latent vector z. During the encoding process, the target encoder uses convolution kernels of different sizes to extract feature information of different scales and fuses it into the latent vector. Convolution kernels of different sizes are used to capture local details and overall structures. The generated latent vector contains details and overall structural information at different levels, which enhances the model's ability to understand complex images. Then, the fusion of multi-scale features achieves effective compression of the image and reduces the amount of data for subsequent processing.

[0062] The target encoder uses multiple convolution kernels of different sizes (such as 3x3, 5x5, 7x7, etc.) to perform convolution operations on random noise, thereby obtaining a series of visual feature maps of different scales. Each scale corresponds to different levels of image features, such as local texture, edges, contours, and overall structure. Feature maps of different scales allow the model to focus on various aspects of the image during the generation process, from fine-grained details to coarse-grained overall layout, improving the quality of the generated image.

[0063] If the target noise is random noise, multiple convolution kernels of different sizes are used to perform convolution operations on the random noise in order to extract potential structures or patterns at different levels (scales) from random noise that has no specific morphological information. Although there is no specific image at this time, the convolution kernel can be regarded as a filter that can recognize and enhance specific patterns in the input data, even if these patterns are not obvious initially. Although the convolution operation is performed on random noise, this step is actually predicting or presetting various features that may appear in future images. By using convolution kernels of different sizes, features at various scales can be captured. For example, a small convolution kernel may focus on local features such as details or edges, while a large convolution kernel may focus more on overall shapes or patterns in a large range.

[0064] Step 203: The text feature is fused into the latent vector through the trained target fusion model to generate a fused feature, wherein the fused feature is used to realize the generation of the text-guided image.

[0065] The target fusion model obtains the latent vector output by the target encoder and the text features output by the text encoder, and fuses the text features into the latent vector. The text features can effectively guide the update of the latent vector, ensuring that the generated image is not only visually realistic, but also consistent with the text description in terms of content.

[0066] Step 204: Decode the fused features into a final image through the trained target decoder.

[0067] The target decoder is responsible for restoring the low-dimensional fusion features into high-dimensional image data and gradually recovering a clear and realistic image. It is the inverse process of the target encoder.

[0068] In this application, feature information of different scales is first extracted and fused into a latent vector, so that the latent vector can contain details of different levels of the image and overall structural information. Then, text features are fused into the latent vector to achieve text-guided image generation. Finally, the final image output by decoding can not only finely control the details and overall structure of the image, but also ensure that the generated image conforms to the text description, thereby improving the quality of text-based generated images.

[0069] As an optional implementation, in step 202, processing the target noise by using a trained target encoder to generate a latent vector fused with multi-scale features includes:

[0070] S11: Input the target noise into the trained target encoder;

[0071] S12: performing a convolution operation on the target noise through multiple convolution kernels of different sizes to extract feature information of different scales, wherein convolution kernels of different sizes correspond to different receptive fields;

[0072] S13: using a preset fusion method to fuse feature information of multiple different scales;

[0073] S14: Convolve and compress the fused multi-scale features to generate a latent vector.

[0074] The server takes the target noise provided by the user (which can be random noise or a target image specified by the user) as input and sends it to the trained target encoder. The target encoder uses multiple convolution kernels of different sizes to perform convolution operations on the input target noise to extract feature information of different scales, where each convolution kernel of different sizes corresponds to a specific receptive field, thereby capturing different levels of details of the image, that is, from small-scale textures to large-scale structural changes.

[0075] Convolution kernels with larger receptive fields help maintain the overall structure of the image, while smaller receptive fields help preserve local details, which is critical for generating images with high fidelity. Multi-scale feature extraction ensures that the generative model can capture various complex patterns in the image, improving the quality and diversity of the final output. In addition, even in the face of low-quality or incomplete inputs, multi-scale feature extraction can provide more stable performance because information at different scales can compensate for each other's shortcomings to a certain extent.

[0076] Next, the target encoder uses a preset fusion method to integrate the feature information of multiple scales obtained in the above steps. By fusing features of different scales, it can better represent complex image content, thereby generating more delicate and realistic images. The fused features can find a balance between the global and local, ensuring that the generated image is not only reasonable at the macro level, but also consistent in micro details.

[0077] Fusion methods include but are not limited to splicing, weighted averaging, etc. These methods can be flexibly selected according to specific task requirements. The following is a detailed description of these two fusion methods.

[0078] Splicing: Splice features of different scales in the channel dimension to obtain a feature containing multi-scale information. The calculation formula for splicing is: F concat =Concar(F3, F5, F7), where F concat is the fusion feature after splicing, and F3, F5, and F7 represent feature information of different scales respectively.

[0079] Weighted average: Perform weighted average on features of different scales to obtain a fused feature, where the weight can be obtained through learning or set based on experience. The calculation formula for weighted average is:

[0080] F avg=αF3+βF5+γF7, where F avg is the feature fused after weighted averaging, α, β, γ are weight coefficients, and α+β+γ=1.

[0081] The target encoder performs convolution and compression operations on the fused multi-scale features again, converting them into a compact latent vector. This latent vector contains all the information required for generation, but its size is greatly reduced. The compressed latent vector reduces the demand for data storage space, while also speeding up the calculation speed and improving the efficiency of the entire generation process. Despite the reduction in size, the latent vector still retains all the key information, which allows the generated image to retain the original features while meeting new creation requirements.

[0082] During the encoding process of the target encoder, multiple convolution kernels of different sizes are used to capture different levels of details of the image, covering everything from the overall structure of the image to local details, and then generating a latent vector that integrates multi-scale features, so that the output latent vector can represent complex image content, making it easier for the subsequent target fusion model to enhance its ability to understand complex images.

[0083] As an optional implementation, in step 203, the target fusion model is a target diffusion model, and the text features are fused into the latent vector through the target fusion model. The generation of fusion features includes the following process:

[0084] S21: inputting the text features and the latent vector into a diffusion model, wherein the diffusion model includes a plurality of time steps;

[0085] S22: Calculate cross attention based on the text features and the potential vector corresponding to the current time step, and update the potential vector based on the result of the cross attention;

[0086] S23: Pass the updated potential vector to the next time step until the cross attention calculation of all time steps is completed, and the cross attention result outputted at the last time step is obtained;

[0087] S24: Take the cross-attention result output at the last time step as the fusion feature.

[0088] The server takes the encoded text features and latent vectors as input and sends them to the diffusion model. In the diffusion model, there are multiple time steps t, each of which represents a stage of the denoising process. For each time step t, cross attention is calculated based on the latent vector and text features at the current moment. Specifically, the latent vector is regarded as a query, the text features are used as keys and values, and the cross attention mechanism is used to calculate the weight of the influence of the text features on the latent vector. The latent vector of the current time step is updated according to the result of the cross attention. This step is achieved by adjusting the values ​​of each element in the latent vector so that they are closer to the ideal state, that is, closer to the image features required by the text description provided by the user. The updated latent vector is passed to the next time step to continue to participate in the cross attention calculation. This process continues until all cross attention calculations are completed in the last time step. By introducing the latest cross attention results at each time step, the latent vector is continuously optimized and gradually approaches the feature representation of the target image. This gradual optimization process ensures the quality of the final output. When the cross attention calculation is completed in the last time step, the output result is regarded as the final fused feature. These fused features contain information from both text and image, providing a basis for subsequent decoding into the final image.

[0089] Cross-attention allows the model to dynamically focus on key parts of the text description during the generation process, ensuring that the generated image is more in line with the user's intentions and expectations. For example, when generating "a white kitten", the model can place more emphasis on important features such as color and animal species. In this way, the generated image not only looks realistic, but also better reflects the content of the text description, achieving a deep connection between the image and the text. Cross-attention helps capture the subtle differences in the text description and reflect them in the final generated image, thereby improving the refinement and realism of the image.

[0090] Optionally, the diffusion model also performs image denoising during the processing process. Specifically, the noise in the image features is gradually removed in the reverse denoising process. At the same time, the text features are integrated into the latent vector using the cross-attention mechanism to achieve text-guided image generation.

[0091] Figure 3 The flowchart for outputting the final image based on the cross-attention mechanism includes the following steps:

[0092] Step 301: Use a text encoder to convert the text description into a text feature vector Vtext.

[0093] Step 302: Initialize a random noise and use a target encoder to convert the random noise into a latent vector z.

[0094] Step 303: Set the initial time step t = S, where S is the total number of time steps. At each time step t, perform the following operations:

[0095] Step 304: Determine whether t≥1, if so, execute step 305, if not, execute step 308.

[0096] Step 305: Use the image features of the current time step as Query, and the text features Vtex as Key and Value to calculate the cross attention.

[0097] The formula for calculating cross attention includes:

[0098]

[0099] Among them, Attention(Q, K, V) is the cross attention, Q represents the potential vector corresponding to the current time step, K represents each word in the text feature, V represents the encoding vector of each word, and d k is the dimension of K, and T represents transpose.

[0100] Step 306: Update the potential vector of the current time step according to the result of the cross attention.

[0101] Step 307: pass the updated latent vector to the next time step t-1, return to step 304, and repeat steps 304 to 307 until the cross-attention calculations for all time steps are completed.

[0102] Step 308: After the cross-attention calculation is completed at the last time step t=0, the final fused features are obtained, and the target decoder is used to decode the fused features into the final image.

[0103] As an optional implementation, the training process of the target encoder, the target decoder, and the target fusion model includes:

[0104] Step S31: obtaining a plurality of sample image-text pairs;

[0105] Step S32: training a VAE model according to a sample image-text pair, so that the initial encoder learns to extract and fuse multi-scale features and generate image latent vectors, and the initial decoder learns to decode, wherein the VAE model includes an initial encoder and an initial decoder;

[0106] Step S33: When the parameters of the VAE model are fixed, the initial fusion model is trained jointly with the VAE model so that the initial fusion model learns to fuse text features into the latent vector.

[0107] First, a large number of image-text pairs need to be collected as training samples. Image data needs to be preprocessed, such as resizing and cropping, and text data needs to be preprocessed, such as word segmentation and stop word removal.

[0108] The collected sample image-text pairs will then be used to train a VAE model, which consists of two parts: the initial encoder and the initial decoder.

[0109] Initial encoder: responsible for converting the input image into a low-dimensional latent vector z. At the same time, in this process, the encoder learns to capture different levels of details in the image through multi-scale feature extraction and fusion mechanism.

[0110] Inception Decoder: Receives the latent vector z from the encoder and restores it to the original image or an approximate version of it.

[0111] During the training process, both the image and the corresponding text description are considered. Specifically, an auxiliary task can be added to the latent space of VAE, that is, predicting the text features that match the image, which can be achieved by introducing cross entropy loss or other appropriate loss functions. By introducing text information as an auxiliary task, VAE can learn richer and more semantic latent representations, thereby improving the quality of image reconstruction.

[0112] Once the VAE model is trained and its parameters are fixed, the final training phase begins, which is the joint training of the diffusion model. The diffusion model plays the role of the initial fusion model here. The core task of the diffusion model is to gradually recover the original image features from the noise and integrate the text features into the image features using the cross-attention mechanism at each time step. At this point, the VAE has learned good image encoding and decoding capabilities, and the diffusion model can further optimize the generation process on this basis.

[0113] Initialization: First, fix the parameters of the VAE model to ensure that it can stably encode images into latent representations and can effectively decode them back to images.

[0114] Forward diffusion process: Given a latent vector z, the diffusion model performs a series of time-step operations, gradually adding random noise to the latent vector z at each time step. This process simulates the process from a clear image to complete noise.

[0115] Reverse denoising process: Contrary to forward diffusion, the diffusion model attempts to recover the original image features from a noisy state. More importantly, at each time step t, the diffusion model uses a cross-attention mechanism to combine the text features Vtext with the current image features. This combination allows the text information to continue to influence the direction of image generation throughout the denoising process, making the generated results more consistent with the text description.

[0116] Joint training: When training the diffusion model, we also use image-text pairs. For each pair of data, we first use the VAE encoder to get the potential representation of the image, and then generate the image through the reverse denoising process of the diffusion model.

[0117] The design of the loss function is critical throughout the training process. It not only measures the difference between the generated image and the real sample, but also considers whether the generated image accurately reflects the content of the sample text. Specifically, the loss function may include but is not limited to the following components:

[0118] Reconstruction error: used to evaluate the similarity between the decoded image and the original input, such as mean square error or variational lower bound.

[0119] Text consistency loss: Ensuring that the generated image matches the provided text description can be achieved through contrastive learning or other forms of text-image association metrics.

[0120] By continuously adjusting the weights in the above loss function, the model can be guided in the desired direction, and finally an image generation system with high fidelity and faithfulness to the text description can be obtained. The above training process is an iterative optimization process, in which the learning results of the VAE model provide a solid starting point for the diffusion model, and the diffusion model further enhances the quality and flexibility of image generation.

[0121] This application includes four functional modules, and the function of each functional module is as follows.

[0122] 1. Encoder module.

[0123] 1.1 Multi-scale feature extraction and fusion: In the middle layer of the encoder, different sizes of patch_size (such as 3x3, 5x5, 7x7) are used to process image features. Each convolution kernel patch size corresponds to a different receptive field, which can capture features of different scales. For example, a 3x3 patch focuses on local details, while a 7x7 patch focuses on structural information in a larger range. For each patch size, a series of convolutional layers and nonlinear activation functions are used to extract features to obtain feature information of different scales F3, F5, F7.

[0124] 1.2 Feature fusion: Fuse the feature information F3, F5, and F7 of different scales. Common fusion methods include splicing and weighted averaging.

[0125] 1.3 Latent vector generation: The fused multi-scale features are further convolved and compressed to finally generate the mean μ and variance σ of the latent vector. The formula of the latent vector is: where μ and σ are calculated by the encoder network.

[0126] 2. Text encoder module.

[0127] This module is responsible for converting the input text description into a feature vector Vtext. A pre-trained Transformer model, such as BERT or RoBERTa, can be used to extract the semantic features of the text.

[0128] 3. Diffusion model.

[0129] In the reverse denoising process of the diffusion model, the cross-attention mechanism is used to integrate the text features Vtext into the image features (latent vector).

[0130] Specific implementation: At each time step t of the diffusion model, the image features of the current step are used as the query, the text features Vtext are used as the key and value, and the cross attention is calculated to obtain the weighted representation of the text features on the image features.

[0131] Through the cross-attention mechanism, text features can guide the denoising process of image features, making the generated image more consistent with the text description.

[0132] 4. Decoder module.

[0133] This module is responsible for decoding the fused features processed by the diffusion model into an image, which is the inverse process of the encoder.

[0134] This application provides an overall process for generating an image based on multi-scale feature fusion, including the following steps:

[0135] 1. Data preparation.

[0136] Collect a large amount of image-text pair data for training models.

[0137] Perform preprocessing on image data, such as resizing, cropping, etc.

[0138] Preprocess the text data, such as word segmentation and stop word removal.

[0139] 2. Model construction.

[0140] 2.1 Build an image VAE model, including an encoder network and a decoder network. In the encoder, build a multi-scale feature extraction module, use patches of different sizes (such as 3x3, 5x5, 7x7) to process the image features of the intermediate layer, and perform multi-scale feature fusion inside the encoder.

[0141] 2.2 Build a text encoder. You can use a pre-trained Transformer model such as BERT or RoBERTa.

[0142] 2.3 Construct a diffusion model and introduce a cross-attention mechanism in the reverse denoising process to achieve the fusion of text features and image features.

[0143] 3. Model training.

[0144] First, the VAE model is trained to learn the potential representation and decoding capabilities of the image, while also learning the extraction and fusion of multi-scale features.

[0145] Then, the VAE model parameters are fixed and the diffusion model is trained.

[0146] When training the diffusion model, the forward diffusion process is first performed to add noise to the image features; then the reverse denoising process is performed to learn to recover the original image features from the noise, and the cross-attention mechanism is used to integrate the text features into the image features at each time step.

[0147] 4. Image generation.

[0148] 4.1 Image Coding and Multi-Scale Feature Extraction: Use VAE (Variational Autoencoder) encoder to convert the input image into a feature vector. In the VAE encoder, we use patches of different sizes (such as 3x3, 5x5, 7x7) to process the image features of the intermediate layer to extract feature information of different scales. These patches of different sizes can capture the details and structural information of different levels of the image.

[0149] 4.2 Multi-scale feature fusion: In the VAE encoder, the multi-scale features extracted by different patches are fused to obtain a feature representation containing rich multi-scale information, which is finally compressed into a latent variable z.

[0150] 4.3 Text feature extraction: The text description is also converted into a text feature vector.

[0151] Feature fusion based on cross-attention: In the reverse denoising process of the diffusion model, the cross-attention mechanism is used to integrate text features into image features (features from the VAE encoder input) to achieve text-guided image generation.

[0152] 4.4 Image output: Using the fused feature vector, the noise in the image features is gradually removed through the reverse denoising process of the diffusion model. Finally, the image VAE decoder is used to decode the denoised image features into the final image.

[0153] Based on the same technical concept, the embodiment of the present application also provides a device for generating an image based on multi-scale feature fusion, such as Figure 4 As shown, the device comprises:

[0154] An acquisition module 401 is used to acquire text features and target noise, wherein the text features are used to provide content guidance for generating an image;

[0155] A processing module 402 is used to process the target noise through the trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate the details and overall structural information of different levels in the generated image, and the latent vector is used to indicate the image features of the generated image;

[0156] A fusion module 403 is used to fuse text features into latent vectors through a trained target fusion model to generate fusion features, wherein the fusion features are used to realize the generation of text-guided images;

[0157] The decoding module 404 is used to decode the fused features into a final image through a trained target decoder.

[0158] Optionally, the processing module 402 is used to:

[0159] Input the target noise into the trained target encoder;

[0160] The target noise is convolved with multiple convolution kernels of different sizes to extract feature information of different scales, where convolution kernels of different sizes correspond to different receptive fields.

[0161] The feature information of multiple different scales is fused by using a preset fusion method;

[0162] The fused multi-scale features are convolved and compressed to generate a latent vector.

[0163] Optionally, the target fusion model is a target diffusion model, and the fusion module 403 is used to:

[0164] Inputting the text features and the latent vector into a diffusion model, wherein the diffusion model includes multiple time steps;

[0165] Calculate cross attention based on the text features and the latent vector corresponding to the current time step, and update the latent vector based on the result of cross attention;

[0166] The updated potential vector is passed to the next time step until the cross-attention calculation of all time steps is completed, and the result of the cross-attention output of the last time step is obtained;

[0167] The result of the cross attention output at the last time step is used as the fusion feature.

[0168] Optionally, the formula for calculating the cross attention includes:

[0169]

[0170] Among them, Attention(Q, K, V) is the cross attention, Q represents the potential vector corresponding to the current time step, K represents each word in the text feature, V represents the encoding vector of each word, and d k is the dimension of K, and T represents transpose.

[0171] Optionally, the device is also used for:

[0172] Get multiple sample image-text pairs;

[0173] Training a VAE model according to a sample image-text pair so that an initial encoder learns to extract and fuse multi-scale features and generate an image latent vector, and an initial decoder learns to decode, wherein the VAE model includes an initial encoder and an initial decoder;

[0174] When the parameters of the VAE model are fixed, the initial fusion model is trained jointly with the VAE model so that the initial fusion model learns to fuse text features into the latent vector.

[0175] Optionally, the device is also used for:

[0176] Use a target encoder to convert a sample image into a latent vector;

[0177] Performing a forward diffusion process of the initial diffusion model on the latent vector, wherein the forward diffusion process is to add noise to the sample image;

[0178] In the reverse denoising process of the initial diffusion model, the noise is removed to restore the original image features, and the text features are integrated into the image features using the cross-attention mechanism at each time step;

[0179] The trained target fusion model is obtained by adjusting the loss function, where the loss function is used to measure the difference between the generated image and the sample image, and whether the generated image conforms to the description of the sample text.

[0180] Optionally, the target noise is random noise or a target image.

[0181] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0182] Memory 503, used for storing computer programs;

[0183] The processor 501 is used to implement the above steps when executing the program stored in the memory 503.

[0184] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0185] The communication interface is used for communication between the above electronic device and other devices.

[0186] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0187] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0188] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0189] In another embodiment of the present invention, a computer program product including instructions is provided, which enables the computer to execute any one of the methods in the above embodiments when the computer program product is executed on the computer.

[0190] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0191] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0192] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating an image based on multi-scale feature fusion, characterized in that: The method comprises: Acquire text features and target noise, wherein the text features are used to provide content guidance for generating an image; Processing the target noise through a trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate details and overall structural information at different levels in the generated image, and the latent vector is used to indicate image features of the generated image; The text feature is fused into the latent vector through a trained target fusion model to generate a fusion feature, wherein the fusion feature is used to realize the generation of a text-guided image; The fused features are decoded into the final image by the trained target decoder.

2. The method according to claim 1, characterized in that The target noise is processed by the trained target encoder to generate a latent vector that integrates multi-scale features, including: Inputting the target noise into a trained target encoder; Performing a convolution operation on the target noise through a plurality of convolution kernels of different sizes to extract feature information of different scales, wherein the convolution kernels of different sizes correspond to different receptive fields; The feature information of multiple different scales is fused by using a preset fusion method; The fused multi-scale features are convolved and compressed to generate the latent vector.

3. The method according to claim 1, characterized in that The target fusion model is a target diffusion model. The text feature is fused into the latent vector through the target fusion model. The fusion feature is generated by: Inputting the text features and the latent vector into the diffusion model, wherein the diffusion model includes a plurality of time steps; Calculating cross attention according to the text feature and the potential vector corresponding to the current time step, and updating the potential vector according to the result of the cross attention; The updated potential vector is passed to the next time step until the cross-attention calculation of all time steps is completed, and the result of the cross-attention output of the last time step is obtained; The result of the cross attention output at the last time step is used as the fusion feature.

4. The method according to claim 3, characterized in that The formula for calculating cross attention includes: Among them, Attention(Q,K,V) is cross attention, Q represents the potential vector corresponding to the current time step, K represents each word in the text feature, V represents the encoding vector of each word, and d k is the dimension of K, and T represents transpose.

5. The method according to claim 1, characterized in that The training process of the target encoder, the target decoder and the target fusion model includes: Get multiple sample image-text pairs; Training a VAE model according to the sample image-text pair so that the initial encoder learns to extract and fuse multi-scale features and generate image latent vectors, and the initial decoder learns to decode, wherein the VAE model includes an initial encoder and an initial decoder; When the parameters of the VAE model are fixed, the initial fusion model is trained in conjunction with the VAE model so that the initial fusion model learns to fuse text features into the latent vector.

6. The method according to claim 5, characterized in that The initial fusion model is an initial diffusion model, and training the initial fusion model in conjunction with the VAE model includes: Converting a sample image into a latent vector using the target encoder; Performing a forward diffusion process of the initial diffusion model on the latent vector, wherein the forward diffusion process is adding noise to the sample image; In the reverse denoising process of the initial diffusion model, the noise is removed to restore the original image features, and the text features are integrated into the image features using a cross-attention mechanism at each time step; The trained target fusion model is obtained by adjusting the loss function, wherein the loss function is used to measure the difference between the generated image and the sample image, and whether the generated image conforms to the description of the sample text.

7. The method according to claim 1, characterized in that The target noise is random noise or a target image.

8. A device for generating an image based on multi-scale feature fusion, characterized in that: The device comprises: An acquisition module, used for acquiring text features and target noise, wherein the text features are used for providing content guidance for generating an image; A processing module, used for processing the target noise through a trained target encoder to generate a latent vector fused with multi-scale features, wherein the multi-scale features are used to indicate details and overall structural information at different levels in the generated image, and the latent vector is used to indicate image features of the generated image; A fusion module, used to fuse the text feature into the latent vector through a trained target fusion model to generate a fusion feature, wherein the fusion feature is used to realize the generation of a text-guided image; The decoding module is used to decode the fused features into a final image through a trained target decoder.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images