Multi-task visual base model image generation method based on diffusion model

Abstract

The present application relates to the technical fields of image processing and machine learning, and discloses a method for improving the image generation quality of a multi-task visual base model based on a diffusion model, which comprises two parts, namely, a visual low-level semantic information learning adapter expanded according to the number of tasks and a multi-task visual base model with a feedback auxiliary learning mechanism.The visual low-level semantic information learning adapter expanded according to the number of tasks comprises a group of parallel small-weight convolution layer modules specific to different tasks, and a convolution layer module shared among different tasks is connected in series after the convolution layer modules specific to the different tasks; and the feedback auxiliary learning mechanism guides the multi-task visual base model to be better trained to promote the improvement of the final image generation quality through the minimization of the feedback loss.The present application is mainly applied to the field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and machine learning, and particularly relates to a method for improving the generation quality of a multi-task visual image based on a diffusion model. BACKGROUND

[0002] In recent years, the generation method based on diffusion model has made important breakthroughs in the field of image generation and conversion. Its core idea is to restore high-quality images by gradually denoising, showing strong generation ability and diversity. Under this background, multi-task visual basic model has become a research hotspot, and its goal is to support efficient learning and reasoning of multiple downstream visual tasks. However, with the increasing number of model tasks and the complexity of application scenarios, the existing technology faces many challenges in model generation quality, task adaptability and resource efficiency, etc.

[0003] For example, traditional multi-task visual basic models often rely on complex independent modules or full-parameter adjustment to adapt to different tasks. Although this method has a certain flexibility, with the increase of the number of tasks, the parameter size of the model and the training resource demand increase exponentially. In addition, the existing method is difficult to effectively utilize visual low-level semantic information (such as edge, texture, spatial distribution, etc.) in a multi-task environment, resulting in a lack of consistency in the generation quality between tasks, and the model generalization ability is limited. At the same time, most multi-task visual basic models only rely on the reconstruction loss of the latent space, and the latent space lacks supervision based on visual factors in the image space, so there is a big optimization bottleneck in improving the quality of generated images.

[0004] Therefore, in the face of increasingly complex multi-task visual basic models and their high-quality image generation needs, it is of significant academic value and practical significance to research a lightweight and modular method that can effectively improve the image generation quality, especially a solution that can efficiently adapt in a multi-task environment. This also provides new possibilities for the further development and wide application of multi-task visual basic models based on diffusion models. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application aims to provide a method for improving the image generation quality of a multi-task visual basic model based on a diffusion model, which can efficiently and lightly improve the image generation quality of a multi-task visual basic model based on a diffusion model. To this end, the technical solution adopted by the present application is a multi-task visual basic model image generation method based on a diffusion model, which includes a visual low-level semantic information learning adapter expanded according to the number of tasks and a multi-task visual basic model with a feedback auxiliary learning mechanism, wherein:

[0006] A visual low-level semantic information learning adapter that expands according to the number of tasks includes a set of parallel, task-specific, low-weight convolutional layer modules, and a convolutional layer module shared across different tasks connected in series after these task-specific convolutional layer modules. The adapter is inserted into the task information input of a multi-task visual base model and is expanded according to the number of tasks.

[0007] The feedback-assisted learning mechanism refers to: using a perceptual model with fixed weight parameters to extract corresponding conditional images from the input or output images of a multi-task vision foundation model, thus obtaining the conditional images at the input or output end of the multi-task vision foundation model; calculating the consistency information between the input and output conditional images in the image space to obtain a feedback loss; and minimizing this feedback loss to guide the multi-task vision foundation model to train better and improve the quality of the final image generation. In the feedback-assisted learning mechanism, the output image at the output end is obtained through a one-step denoising strategy to consider training efficiency.

[0008] Specifically, the adapter is expanded according to the number of tasks. If the visual base model contains K tasks, the adapter contains K parallel small-weight convolutional modules. Each small-weight convolutional module learns low-level visual semantic information specific to only one task. Different task information is assigned to the corresponding task-specific convolutional module by an indicator preceding the K parallel convolutional modules. Then, a convolutional module shared by the K tasks is inserted after these K parallel convolutional modules. This module can learn from the information of the previous K small-weight convolutional modules. The adapter formula is: Where I(·) is an indicator function that assigns different task information to the corresponding low-weight convolutional modules. k It is a conditional image or query image from task k. It is the i-th small-weight convolutional module in the adapter-parallel convolutional module, F shared-conv It is a convolutional module shared by all parallel small-weight convolutional modules.

[0009] The one-step denoising strategy used in the feedback-assisted learning mechanism refers to the process in which, during the reverse denoising of the diffusion model, an image with a small amount of noise at a smaller time step is denoised in one step to obtain a clear image.

[0010] The perceptual model introduced in the feedback-assisted learning mechanism is a type of model used to extract conditional images from images, specifically one of the following models: the UPerNet semantic segmentation model based on the encoder-decoder structure; the DPT depth estimation model based on the encoder-decoder structure; the edge detection model EDTER edge detection framework; and the edge detection model EDTER.

[0011] The input query image of the multi-task vision basic model is a normal image or a conditional image, including segmentation map, contour map, depth map, normal map, human pose map. The output generated image is a conditional image, including segmentation map, contour map, depth map, normal map, human pose map, or normal image. The conditional image is extracted from the normal image using a perception model with fixed weight parameters.

[0012] For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism is as follows: First, a perceptual model with fixed weight parameters is used to extract the corresponding conditional image from the output image after a one-step denoising strategy, thus obtaining the conditional image at the output end. Then, the mean square error (MSE) between the conditional image at the input end and the conditional image at the output end is calculated in the image space as the feedback loss.

[0013] For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism is as follows: First, the corresponding conditional image is extracted from the input image using a perceptual model with fixed weight parameters to obtain the conditional image of the input. Then, the mean square error (MSE) between the conditional image of the input and the conditional image of the output after applying a one-step denoising strategy is calculated in the image space as the feedback loss.

[0014] Choose a multi-task vision baseline model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task vision baseline model, while ensuring that the newly added visual low-level semantic information learning adapter can be fine-tuned; train the loss function L on the original multi-task vision baseline model. Train In this context, feedback loss L is introduced. Feedback Then, construct the total loss function of the optimization objective as L. Total =L Train +λL Feedback , where λ is the feedback loss weight; set an appropriate learning rate and begin training the entire network, including the perceptual model and the multi-task vision base model.

[0015] The detailed steps are as follows:

[0016] Step 1: Select Prompt Diffusion as the multi-task vision base model;

[0017] Step 2: Insert a designed low-level visual semantic information learning adapter into the task information input end of the ControlNet part of Prompt Diffusion for each of the input example image pair and the query image. The ControlNet part is a trainable copy network obtained by copying the encoder part and intermediate layers of the diffusion model.

[0018] Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image for Prompt Diffusion can be a conditional image or a normal image, depending on the task.

[0019] When the query image is a conditional image, the model outputs a normal image. In this case, a perceptual model with fixed weight parameters is first used to extract the corresponding conditional image from the image output after a denoising step on the image with a small amount of noise during the Prompt Diffusion backdiffusion process. Then, in the image space, the mean square error (MSE) between the input and output conditional images is calculated as the feedback loss L. Feedback In this case:

[0020] L Feedback =L(x query ,D(G(x t ,t,c t ,,c f )))

[0021] Where L is the mean square error (MSE) for calculation, and x query This is the input conditional image at this point, D(·) is the perceptual model used to extract the conditional image from the normal image, G represents the process by which the model uses a one-step denoising strategy to denoise based on the conditional information to generate the output image from an image with a small amount of noise, and c f It is conditional embedding information from the ControlNet part, c t It is the text embedding information input to the model;

[0022] When the query image is a normal image, the model outputs a conditional image. In this case, a perceptual model with fixed weight parameters is first used to extract the corresponding conditional image from the normal image at the Prompt Diffusion input. Then, in the image space, the mean square error (MSE) between the input and output conditional images after a one-step denoising strategy is calculated as the feedback loss L. Feedback In this case:

[0023] L Feedback =L(D(x) query ),G(x t ,t,c t ,,c f ))

[0024] Where x query This is the normal image input at this time;

[0025] Step 4: After constructing the entire network by inserting the visual low-level semantic information learning adapter, only the ControlNet parameter part of the visual low-level semantic information learning adapter and the Prompt Diffusion model is retained for fine-tuning, while all other weight parameters of the Prompt Diffusion are fixed.

[0026] Step 5: Construct the loss function. The original training loss function for Prompt Diffusion was:

[0027]

[0028] Where x is the query image, the diffusion process adds noise to the encoded latent embedding z = ε(x), thus producing a noisy image latent embedding z. t , z t It is the latent embedding of the noisy image at time step t, c t It is text embedding, c f It is a task-specific embedding, ∈ θ ∈ is the training network of the model, and ∈ is a standard normal random variable;

[0029] Introducing the feedback loss L obtained by utilizing a feedback-assisted learning mechanism in image space Feedback Afterwards, the total loss function is:

[0030] L Total =L Train +λL Feedback

[0031] Where λ is the feedback loss weight;

[0032] Step 6: Set an appropriate learning rate and begin training the entire network, including the perception model and the multi-task vision base model.

[0033] The features and beneficial effects of this invention are:

[0034] (1) This invention designs a method to improve the image generation quality of a multi-task visual basic model based on a diffusion model. It not only has important research value in academia, but also shows excellent practicality in practical applications. (2) This invention can support different numbers of tasks in a multi-task visual basic model. The proposed low-level visual semantic information learning adapter can be expanded according to the number of tasks, thus improving the applicability of the invention. (3) The low-level visual semantic information learning adapter or feedback-assisted learning mechanism proposed in this invention also has a positive gain on the model performance when used alone. Therefore, it improves the flexibility of applying this invention in practical application scenarios according to model design requirements, training resources and other conditions.

[0035] This invention provides an effective means to improve the generation of higher-quality images for multi-task vision basic models based on diffusion models, and provides strong support for their better application in real-world scenarios. Attached image description:

[0036] Figure 1 This is a structural diagram of a visual low-level semantic information learning adapter.

[0037] Figure 2 A schematic diagram of a feedback-assisted learning mechanism.

[0038] Figure 3 This is a schematic diagram of the overall method of the present invention. Detailed Implementation

[0039] To address the shortcomings of existing technologies, this invention proposes a method for improving the image generation quality of a multi-task vision foundation model based on a diffusion model. This method can efficiently and lightweightly improve the image generation quality of a multi-task vision foundation model based on a diffusion model.

[0040] To achieve the above objectives, this invention designs a method for improving the image generation quality of a multi-task vision foundation model based on a diffusion model, employing the following technical solution:

[0041] A method for improving the image generation quality of a diffusion-based multi-task vision foundation model includes the following steps:

[0042] S1. A method for improving the image generation quality of a multi-task visual basic model based on a diffusion model includes a visual low-level semantic information learning adapter that can be expanded according to the number of tasks and a feedback-assisted learning mechanism.

[0043] S2. First, design a visual low-level semantic information learning adapter that can be expanded according to the number of tasks. The adapter is then inserted into the task information input end of the multi-task visual base model. It includes a set of parallel, task-specific, low-weight convolutional layer modules, and a convolutional layer module shared between different tasks is connected in series after these task-specific convolutional layer modules.

[0044] S3. A feedback-assisted learning mechanism is then designed. This mechanism introduces a perceptual model with fixed weight parameters (a perceptual model is a type of model capable of extracting conditional images from images, including semantic segmentation models (such as UperNet), depth estimation models (such as DPT), and edge detection models (such as EDTER)) to extract the corresponding conditional images from the input or output images of the multi-task vision foundation model, thus obtaining the input or output conditional images. In the feedback-assisted learning mechanism, the output image at the output end is obtained through a one-step denoising strategy to consider training efficiency (in the reverse denoising process of the diffusion model, images with a small amount of noise at a smaller time step can obtain clear images through one-step denoising). A feedback loss is obtained by calculating the consistency information in the input and output conditional images in the image space. By minimizing this feedback loss, the multi-task vision foundation model can be better trained to improve the final image generation quality.

[0045] S4. Fix all or most of the original pre-trained weight parameters of the multi-task vision base model, while ensuring that the newly added visual low-level semantic information learning adapter can be fine-tuned; introduce feedback loss obtained by feedback-assisted learning mechanism into the original multi-task vision base model training loss function, construct the optimization objective, and train the multi-task vision base model.

[0046] Furthermore, step S2, concerning the construction process of a visual low-level semantic information learning adapter that can be scaled according to the number of tasks, includes:

[0047] A low-level visual semantic information learning adapter is inserted at the task information input of a multi-task visual base model. If the multi-task visual base model contains K tasks, the adapter contains K parallel small-weight convolutional modules. Each small-weight convolutional module learns low-level visual semantic information specific to only one task. Different task information is assigned to the corresponding task-specific convolutional module by an indicator preceding the K parallel convolutional modules. Then, a convolutional module shared by the K tasks is inserted after these K parallel convolutional modules. This module can learn from the information of the previous K small-weight convolutional modules. The adapter formula is: Where I(·) is an indicator function that assigns different task information to the corresponding low-weight convolutional modules. k It is a conditional image or query image from task k. It is the i-th small-weight convolutional module in the adapter-parallel convolutional module, F shared-conv It is a convolutional module shared by all parallel small-weight convolutional modules.

[0048] Furthermore, the construction process of a feedback-assisted learning mechanism in step S3 includes:

[0049] The input image of the multi-task vision basic model is either a normal image or a conditional image (segmentation image, contour image, depth image, plane normal image, human pose image, etc.), and the output image is either a conditional image (segmentation image, contour image, depth image, plane normal image, human pose image, etc.) or a normal image.

[0050] For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism first uses a perceptual model with fixed weight parameters to extract the corresponding conditional image from the output image after a one-step denoising strategy, thus obtaining the output conditional image. Then, in the image space, the MSE (mean squared error) between the input and output conditional images is calculated as the feedback loss L. Feedback ,like Figure 1 As shown.

[0051] For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism works as follows: First, a perceptual model with fixed weight parameters is used to extract the corresponding conditional image from the input image, thus obtaining the input conditional image. Then, in the image space, the mean squared error (MSE) between the input conditional image and the output conditional image after a one-step denoising strategy is calculated as the feedback loss L. Feedback ,like Figure 1 As shown.

[0052] Furthermore, the scheme for model training in step S4 includes:

[0053] Choose a multi-task vision baseline model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task vision baseline model, while ensuring that the newly added visual low-level semantic information learning adapter can be fine-tuned; train the loss function L on the original multi-task vision baseline model. Train In the middle, the feedback loss L obtained by introducing a feedback-assisted learning mechanism is... Feedback Then, construct the total loss function of the optimization objective as L. Total =L Train +λL Feedback , where λ is the feedback loss weight; train the network.

[0054] InstructCV: InstructCV is a unified computer vision task language interface that transforms various computer vision tasks into text-to-image generation problems. It guides task execution through natural language instructions and optimizes pre-trained text-to-image diffusion models using multimodal and multi-task training datasets, transforming them from generative models into instruction-guided multi-task vision learners. It demonstrates excellent generalization ability to unseen data, categories, and user instructions.

[0055] InstructDiffusion is a general-purpose interface for modeling computer vision tasks. It utilizes a Denoising Diffusion Probability Model (DDPM) to transform various computer vision tasks into image manipulation processes based on intuitive human instructions. The output space is a flexible and interactive pixel space. This model can handle a variety of vision tasks, including understanding tasks (such as segmentation and keypoint detection) and generative tasks (such as editing and enhancement), and can even handle tasks not seen during training, outperforming previous methods on new datasets.

[0056] If InstructCV or InstructDiffusion is chosen as the base model for multi-task vision, all original pre-trained weight parameters are fixed, while the adapter parameters remain trainable. If Prompt Diffusion is chosen as the base model for multi-task vision, only the low-level visual semantic information is retained to learn the adapter and the ControlNet parameters of the model for fine-tuning, while all other weight parameters of Prompt Diffusion are fixed.

[0057] The present invention will now be described in further detail with reference to the accompanying drawings and specific examples, but this is not intended to limit the scope of the invention.

[0058] A method for improving image generation quality of a multi-task visual foundation model based on a diffusion model is presented. This method includes a visual low-level semantic information learning adapter that can be scaled according to the number of tasks and a feedback-assisted learning mechanism. First, a visual low-level semantic information learning adapter that can be scaled according to the number of tasks is designed and inserted into the task information input of the multi-task visual foundation model. The adapter can be scaled according to the number of tasks. The adapter includes a set of parallel, task-specific, low-weight convolutional layer modules, followed by a shared convolutional layer module. Then, a feedback-assisted learning mechanism is designed. This mechanism introduces a perceptual model with fixed weight parameters (a perceptual model is a type of model that can extract conditional images from images, including semantic segmentation models (such as UperNet), depth estimation models (such as DPT), edge detection models (such as EDTER), etc.) to extract the corresponding conditional images from the input or output images of the multi-task visual foundation model, thus obtaining the conditional images at the input or output end. In the feedback-assisted learning mechanism, the output image at the output end is obtained through a one-step denoising strategy to consider training efficiency. A feedback loss is obtained by calculating the consistency information in the input and output conditional images in the image space. Minimizing this feedback loss can guide the multi-task vision foundation model to be trained better, thereby improving the quality of the final image generation. Finally, all or most of the original pre-trained weight parameters of the multi-task vision foundation model are fixed, while the newly added visual low-level semantic information learning adapter can be fine-tuned. In the original multi-task vision foundation model training loss function, a feedback loss obtained by the feedback-assisted learning mechanism is introduced to construct the optimization objective and train the entire network. The specific implementation steps are described below using a specific multi-task vision foundation model as an example:

[0059] Step 1: Select Prompt Diffusion, a multi-task vision foundation model, as the backbone of this example network. Prompt Diffusion is a diffusion-based framework that enables contextual learning in image generation tasks. This model automatically understands the underlying task by using visual-language cues, including a pair of task-specific example images and textual guidance, and performs the same task on new query images while following the textual guidance. The Prompt Diffusion model can handle multiple visual-language tasks and is jointly trained on six different tasks, including forward and reverse tasks such as image segmentation, edge detection, and depth map generation.

[0060] Step 2: Insert a designed low-level visual semantic information learning adapter into the task information input end of the ControlNet part of Prompt Diffusion (the ControlNet part is a trainable copy network obtained by copying the encoder part and intermediate layers of the diffusion model), for both the input example image pair and the query image, as shown in the figure.

[0061] Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image (input image) for Prompt Diffusion can be a conditional image or a normal image, depending on the task.

[0062] When the query image is a conditional image, the model outputs a normal image. In this case, the corresponding conditional image is first extracted from the image output after a denoising step on the image with a small amount of noise from the Prompt Diffusion backdiffusion process. Then, the MSE (mean squared error) loss between the input and output conditional images is calculated as the feedback loss L in the image space. Feedback In this case:

[0063] L Feedback =L(x query ,D(G(x t ,t,c t, ,c f )))

[0064] Where L is the value used to calculate MSE (mean squared error), and x query This is the input conditional image at this point, D(·) is the perceptual model used to extract the conditional image from the normal image, and G represents the process by which the model uses a one-step denoising strategy to denoise based on the conditional information to generate the output image from an image with a small amount of noise. f It is conditional embedding information from the ControlNet part, c t It is the text embedding information input into the model.

[0065] When the query image is a normal image, the model outputs a conditional image. In this case, the perceptual model first extracts the corresponding conditional image from the normal image at the Prompt Diffusion input. Then, in the image space, the MSE (mean squared error) loss between the input and output conditional images after a one-step denoising strategy is calculated as the feedback loss L. Feedback In this case:

[0066] L Feedback =L(D(x) query ),G(x t ,t,c t, ,cf ))

[0067] Where L is the value used to calculate MSE (mean squared error), and x query This is the normal input image at this time. D(·) uses a perceptual model that can extract the conditional image from the normal image. G represents the process by which the model uses a one-step denoising strategy to denoise based on the conditional information to generate the output image from an image with a small amount of noise. f It is conditional embedding information from the ControlNet part, c t It is the text embedding information input into the model.

[0068] Step 4: After constructing the entire network by inserting the visual low-level semantic information learning adapter, fix most of the original pre-trained weight parameters of PromptDiffusion, and only keep the visual low-level semantic information learning adapter and the ControlNet part of the model for fine-tuning.

[0069] Step 5: Construct the loss function. The original training loss function for Prompt Diffusion was:

[0070]

[0071] Where x is the query image, the diffusion process adds noise to the encoded latent embedding z = ε(x), thus producing a noisy image latent embedding z. t , z t It is the latent embedding of the noisy image at time step t, c t It is text embedding, c f It is a task-specific embedding, ∈ θ is the training network of the model, and ∈ is a standard normal random variable.

[0072] Introducing the feedback loss L obtained by utilizing a feedback-assisted learning mechanism in image space Feedback Afterwards, the total loss function is:

[0073] L Total =L Train +λL Feedback

[0074] Where λ is the feedback loss weight.

[0075] Step 6: Set an appropriate learning rate and begin training the entire network.

[0076] Perceptual models are a collective term for models such as depth estimation models, semantic segmentation models, and edge detection models. In this invention, perceptual models are used as a tool to extract corresponding conditional images (depth maps, segmentation maps, and contour maps, which are referred to as conditional images) from images.

[0077] Taking the specific example of Prompt Diffusion mentioned in this invention as an example, this model involves a total of 6 tasks, 3 of which are defined as forward tasks and the other 3 as reverse tasks.

[0078] For the forward task defined in the model (generating depth map, segmentation map, and contour map from normal images respectively), since the output is already a conditional image, the conditional image of the input image is extracted using the perceptual image. In this way, conditional images are available at both the input and output ends, and the MSE loss can be calculated between the conditional images at the output and input ends to guide the model to train better.

[0079] For the inverse task defined in the model (generating a normal image from a depth map, segmentation map, and contour map), the input is a conditional image and the output is a normal image. Therefore, this invention uses a perceptual model as a tool to extract the conditional image from the output normal image. In this way, both the input and output ends have conditional images, and the MSE loss can be calculated between the conditional images at the output and input ends to guide the model to train better.

[0080] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments described above, and any obvious improvements, substitutions, or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A multi-task visual basic model image generation method based on a diffusion model, characterized in that, It consists of two parts: a visual low-level semantic information learning adapter that expands according to the number of tasks, and a multi-task visual base model with a feedback-assisted learning mechanism, wherein: The visual low-level semantic information learning adapter, which is expanded according to the number of tasks, includes a set of parallel low-weight convolutional layer modules for different tasks, and a shared convolutional module shared between different tasks, which is connected in series after these low-weight convolutional layer modules for different tasks. The visual low-level semantic information learning adapter is inserted into the task information input end of the multi-task visual base model and is expanded according to the number of tasks. The feedback-assisted learning mechanism refers to: using a perceptual model with fixed weight parameters to extract corresponding conditional images from the input or output images of a multi-task vision basic model, obtaining conditional images at the input or output end of the multi-task vision basic model; calculating the consistency information between the input and output conditional images in the image space to obtain a feedback loss; and guiding the training of the multi-task vision basic model by minimizing this feedback loss; in the feedback-assisted learning mechanism, the output images at the output end are all obtained through a one-step denoising strategy.

2. The image generation method for a multi-task visual basic model based on a diffusion model as described in claim 1, characterized in that, Specifically, the visual low-level semantic information learning adapter is extended according to the number of tasks. If the visual base model contains K tasks, the adapter contains K parallel small-weight convolutional layer modules. Each small-weight convolutional layer module is used only for learning the visual low-level semantic information of one task. Different task information is assigned to the corresponding small-weight convolutional layer module by an indicator preceding the K parallel small-weight convolutional layer modules. Then, a shared convolutional module, shared by the K tasks, is inserted after these K parallel small-weight convolutional layer modules. The shared convolutional module learns from the information of the previous K small-weight convolutional layer modules. The adapter formula is: ， Where I(·) is an indicator function that assigns different task information to the corresponding low-weight convolutional layer modules. It is a conditional image or query image from task k. It is the i-th small-weight convolutional layer module in the adapter parallel convolutional module. It is a shared convolutional module shared by all parallel small-weight convolutional layer modules.

3. The image generation method for a multi-task visual basic model based on a diffusion model as described in claim 1, characterized in that, The one-step denoising strategy used in the feedback-assisted learning mechanism refers to the process in which, during the reverse denoising of the diffusion model, an image with a small amount of noise at a smaller time step is denoised in one step to obtain a clear image. The perceptual model introduced in the feedback-assisted learning mechanism is a type of model used to extract conditional images from images, specifically one of the following models: the UPerNet semantic segmentation model based on the encoder-decoder structure; the DPT depth estimation model based on the encoder-decoder structure; the edge detection model EDTER edge detection framework; and the edge detection model EDTER.

4. The image generation method for a multi-task visual basic model based on a diffusion model as described in claim 1, characterized in that, The input image of the multi-task vision basic model is either a normal image or a conditional image. The conditional image includes a segmentation map, a contour map, a depth map, a normal map, and a human pose map. The output image is either a conditional image or a normal image. The conditional image is extracted from the normal image using a perception model with fixed weight parameters. For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism is as follows: First, a perceptual model with fixed weight parameters is used to extract the corresponding conditional image from the output image after a one-step denoising strategy, thus obtaining the conditional image at the output end. Then, the mean square error (MSE) between the conditional image at the input end and the conditional image at the output end is calculated in the image space as the feedback loss. For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism is as follows: First, the corresponding conditional image is extracted from the input image using a perceptual model with fixed weight parameters to obtain the conditional image of the input. Then, the mean square error (MSE) between the conditional image of the input and the conditional image of the output after applying a one-step denoising strategy is calculated in the image space as the feedback loss. Choose a multi-task vision baseline model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task vision baseline model, while ensuring that the newly added visual low-level semantic information learning adapter can be fine-tuned; train the loss function on the original multi-task vision baseline model. In this context, feedback loss is introduced. Then, the total loss function of the optimization objective is constructed as follows: ; where λ is the feedback loss weight; set the learning rate and start training the entire network, including the perceptual model and the multi-task vision base model.

5. The image generation method for a multi-task visual basic model based on a diffusion model as described in claim 1, characterized in that, The detailed steps are as follows: Step 1: Select Prompt Diffusion as the base model for multi-task vision; Step 2: Insert a designed low-level visual semantic information learning adapter into the task information input end of the ControlNet part of Prompt Diffusion for each of the input example image pair and the query image. The ControlNet part is a trainable copy network obtained by copying the encoder part and intermediate layers of the diffusion model. Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image of Prompt Diffusion is either a conditional image or a normal image depending on the task. When the query image is a conditional image, the model outputs a normal image. In this case, the corresponding conditional image is first extracted from the image output after a denoising step on the image with a small amount of noise from the Prompt Diffusion backdiffusion process using a perceptual model with fixed weight parameters. Then, the mean square error (MSE) between the input and output conditional images is calculated as the feedback loss in the image space. In this case: ； Where L is the mean square error (MSE) for calculation; D is the input conditional image at this point; D(·) is the perceptual model used to extract the conditional image from the normal image; G represents the process by which the model uses a one-step denoising strategy to denoise based on the conditional information to generate the output image from an image with a small amount of noise. It is conditional embedding information from the ControlNet part. It is the text embedding information input to the model; When the query image is a normal image, the model outputs a conditional image. In this case, a perceptual model with fixed weight parameters is first used to extract the corresponding conditional image from the normal image at the Prompt Diffusion input. Then, in the image space, the mean square error (MSE) between the input and output conditional images after a one-step denoising strategy is calculated as the feedback loss. In this case: ； in This is the normal image input at this time; Step 4: After constructing the entire network by inserting the visual low-level semantic information learning adapter, only the ControlNet parameter parts of the visual low-level semantic information learning adapter and the Prompt Diffusion model are retained for fine-tuning; Step 5: Construct the loss function. The original training loss function for Prompt Diffusion was: ； Where x is the query image, and the diffusion process adds noise to the encoded latent embedding z= This generates a latent embedding for a noisy image. , It is the latent embedding of the noisy image at time step t. It is the training network of the model. It is a standard normal random variable; Introducing a feedback loss obtained by utilizing a feedback-assisted learning mechanism in image space. Afterwards, the total loss function is: ； in It is the feedback loss weight; Step 6: Set the learning rate and begin training the entire network, including the perception model and the multi-task vision base model.

Images