Multitask visual basic model image generation method based on diffusion model

Abstract

The invention relates to the technical field of image processing and machine learning, provides a method for improving the image generation quality of a multitask visual basis model based on a diffusion model, and provides a multitask visual basis model image generation method based on the diffusion model. Comprising a visual low-level semantic information learning adapter expanded according to the number of tasks and a multi-task visual basic model with a feedback auxiliary learning mechanism, and the visual low-level semantic information learning adapter expanded according to the number of tasks comprises a group of parallel small-weight convolutional layer modules specific to different tasks. A convolutional layer module shared among different tasks is connected in series behind the convolutional layer modules of the specific tasks; and the feedback auxiliary learning mechanism guides the multi-task visual basic model to better train through the minimum feedback loss so as to promote the improvement of the final image generation quality. The method is mainly applied to image processing occasions.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and machine learning, and in particular to a method for improving the quality of multi-task visual image generation based on a diffusion model. Background Art

[0002] In recent years, generative methods based on diffusion models have made important breakthroughs in the field of image generation and transformation. The core idea is to restore high-quality images by gradually denoising, showing strong generation capabilities and diversity. In this context, multi-task visual base models have become a research hotspot, with the goal of supporting efficient learning and reasoning of multiple downstream visual tasks at the same time. However, with the increasing number of model tasks and the complexity of application scenarios, existing technologies face many challenges in terms of model generation quality, task adaptability, and resource efficiency.

[0003] For example, traditional multi-task visual base models often rely on complex independent modules or full parameter adjustments to adapt to different tasks. Although this method has a certain degree of flexibility, as the number of tasks increases, the parameter scale and training resource requirements of the model grow exponentially. In addition, existing methods are difficult to effectively utilize low-level visual semantic information (such as edges, textures, spatial distribution, etc.) in a multi-task environment, resulting in a lack of consistency in the quality of generation between tasks and limited model generalization capabilities. At the same time, most multi-task visual base models only rely on the reconstruction loss of the latent space, and the latent space lacks supervision based on visual factors in the image space, so it shows a large optimization bottleneck in improving the quality of generated images.

[0004] Therefore, in the face of increasingly complex multi-task visual base models and their high-quality image generation requirements, it is of great academic value and practical significance to study a lightweight, modular method that can effectively improve the quality of image generation, especially a solution that can be efficiently adapted in a multi-task environment. This also provides new possibilities for the further development and widespread application of multi-task visual base models based on diffusion models. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention aims to propose a method for improving the image generation quality of a multi-task visual base model based on a diffusion model, which can efficiently and lightly improve the image generation quality of a multi-task visual base model based on a diffusion model. To this end, the technical solution adopted by the present invention is that the multi-task visual base model image generation method based on a diffusion model includes a visual low-level semantic information learning adapter expanded according to the number of tasks and a multi-task visual base model with a feedback-assisted learning mechanism, wherein:

[0006] A visual low-level semantic information learning adapter that is expanded according to the number of tasks, comprising a set of parallel small-weight convolutional layer modules specific to different tasks, and a convolutional layer module shared between different tasks connected in series after these task-specific convolutional layer modules, the adapter is inserted into the task information input end of the multi-task visual base model, and the adapter is expanded according to the number of tasks;

[0007] The feedback-assisted learning mechanism refers to: using a perception model with fixed weight parameters to extract the corresponding conditional image from the input image or output image of the multi-task visual base model, obtaining the conditional image of the input or output end of the multi-task visual base model, calculating the consistency information in the input and output conditional images in the image space to obtain a feedback loss, and by minimizing the feedback loss, guiding the multi-task visual base model to be better trained to promote the improvement of the final image generation quality; in the feedback-assisted learning mechanism, the output image of the output end is obtained by a one-step denoising strategy in consideration of training efficiency.

[0008] The adapter is expanded according to the number of tasks. Specifically, if the visual base model contains K tasks, the adapter contains K parallel small weight convolution modules, each small weight convolution module is only specific to the visual low-level semantic information of one task for learning, and different task information is assigned to the corresponding task-specific convolution module by an indicator before the K parallel convolution modules, and then a convolution module shared by K tasks is inserted after the K parallel convolution modules, which can learn from the information of the previous K small weight convolution modules. The adapter formula is: Where I(·) is the indicator function that assigns different task information to the corresponding small-weight convolutional modules, I k is the conditional image or query image from task k, is the i-th small weight convolution module in the parallel convolution module of the adapter, F shared-conv It is a convolution module shared by all parallel small-weight convolution modules.

[0009] The one-step denoising strategy used in the feedback-assisted learning mechanism means that in the reverse denoising process of the diffusion model, an image with a small amount of noise at a smaller time step is denoised in one step to obtain a clear image;

[0010] The perceptual model introduced in the feedback-assisted learning mechanism is a type of model used to extract conditional images from images, specifically one of the following models: a UPerNet semantic segmentation model based on an encoder-decoder structure; a DPT depth estimation model based on an encoder-decoder structure, and an edge detection model edge detection framework EDTER.

[0011] The query image at the input end of the multi-task visual basic model is a normal image or a conditional image including a segmentation map, a contour map, a depth map, a normal map, and a human posture map. The generated image at the output end is a conditional image including a segmentation map, a contour map, a depth map, a normal map, a human posture map, or a normal image. The conditional image is extracted from the normal image using a perception model with fixed weight parameters.

[0012] For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism is to first use a perception model with fixed weight parameters to extract the corresponding conditional image from the image output by the one-step denoising strategy at the output end, and then calculate the mean square error (MSE) between the conditional image at the input end and the conditional image at the output end in the image space as the feedback loss.

[0013] For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism is to first extract the corresponding conditional image from the image at the input end using the perceptual model with fixed weight parameters to obtain the conditional image at the input end, and then calculate the mean square error (MSE) between the conditional image at the input end and the conditional image output at the output end after using the one-step denoising strategy in the image space as the feedback loss;

[0014] Select a multi-task visual base model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task visual base model, while keeping the newly added visual low-level semantic information learning adapter able to be fine-tuned; train the loss function L in the original multi-task visual base model Train In the above example, we introduce feedback loss L Feedback After that, the total loss function of the optimization objective is constructed as L Total =L Train +λL Feedback , where λ is the feedback loss weight; set an appropriate learning rate and start training the entire network including the perception model and the multi-task visual base model.

[0015] The detailed steps are as follows:

[0016] Step 1: Select the multi-task visual base model Prompt Diffusion as the multi-task visual base model;

[0017] Step 2: Insert a designed visual low-level semantic information learning adapter into the task information input of the ControlNet part of Prompt Diffusion for the input sample image pair and the query image. The ControlNet part is a trainable replica network obtained by copying the encoder part and the middle layer of the diffusion model.

[0018] Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image of Prompt Diffusion can be a conditional image or a normal image according to different tasks.

[0019] When the query image is a conditional image, the model outputs a normal image. In this case, the perceptual model with fixed weight parameters is first used to extract the corresponding conditional image from the image with a small amount of noise in the Prompt Diffusion back diffusion process after one-step denoising. Then, the mean square error MSE loss between the input and output conditional images is calculated in the image space as the feedback loss L Feedback , in this case:

[0020] L Feedback =L(x query ,D(G(x t ,t,c t ,,c f )))

[0021] Where L is the mean square error MSE, x query is the input conditional image at this time, D(·) is the perceptual model used to extract the conditional image from the normal image, G represents the process of the model using a one-step denoising strategy to denoise the conditional information to generate an output image from an image with a small amount of noise, c f is the conditional embedding information from the ControlNet part, c t is the text embedding information of the model input;

[0022] When the query image is a normal image, the model outputs a conditional image. In this case, the corresponding conditional image is first extracted from the normal image at the prompt diffusion input end using the perception model with fixed weight parameters. Then, the mean square error (MSE) loss between the conditional image output by the one-step denoising strategy at the input and output ends is calculated in the image space as the feedback loss L. Feedback , in this case:

[0023] L Feedback =L(D(x query ),G(x t ,t,c t ,,c f ))

[0024] where x query is the normal image of the input at this time;

[0025] Step 4: After inserting the visual low-level semantic information learning adapter to build the entire network, only the ControlNet parameter part of the visual low-level semantic information learning adapter and the Prompt Diffusion model is retained for fine-tuning, and all other weight parameters of the Prompt Diffusion are fixed;

[0026] Step 5: Construct the loss function. The original training loss function of Prompt Diffusion is:

[0027]

[0028] Where x is the query image, the diffusion process adds noise to the encoded latent embedding z = ε(x), resulting in a noisy image latent embedding z t , z t is the latent embedding of the noisy image at time step t, c t is the text embedding, c f is a task-specific embedding, ∈ θ is the training network of the model, ∈ is a standard normal random variable;

[0029] Introduce the feedback loss L obtained by using the feedback-assisted learning mechanism in the image space Feedback After that, the total loss function is:

[0030] L Total =L Train +λL Feedback

[0031] Where λ is the feedback loss weight;

[0032] Step 6: Set an appropriate learning rate and start training the entire network including the perception model and the multi-task visual base model.

[0033] The characteristics and beneficial effects of the present invention are:

[0034] (1) The present invention designs a method for improving the image generation quality of a multi-task visual base model based on a diffusion model, which not only has important research value in academia, but also shows excellent practicality in practical applications; (2) The present invention can support different numbers of tasks of a multi-task visual base model, and the proposed visual low-level semantic information learning adapter can be expanded according to the number of tasks, thereby improving the applicability of the invention; (3) The visual low-level semantic information learning adapter or feedback-assisted learning mechanism proposed in the present invention also has a positive gain on the model performance when acting alone, thereby improving the flexibility of applying the present invention in actual application scenarios according to model design requirements, training resources and other conditions.

[0035] The present invention provides an effective means to improve the generation of higher quality images of the multi-task visual basic model based on the diffusion model, and provides strong support for its better application in real-life scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Architecture of learning adapters for visual low-level semantic information.

[0037] Figure 2 Schematic diagram of the feedback-assisted learning mechanism.

[0038] Figure 3 It is an overall schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0039] In view of the shortcomings of the prior art, the present invention proposes a method for improving the image generation quality of a multi-task visual base model based on a diffusion model, which can efficiently and lightly improve the image generation quality of a multi-task visual base model based on a diffusion model.

[0040] In order to achieve the above object, the present invention designs a method for improving the image generation quality of a multi-task visual basic model based on a diffusion model, which adopts the following technical solutions:

[0041] A method for improving the image generation quality of a multi-task visual base model based on a diffusion model comprises the following steps:

[0042] S1. A method for improving the image generation quality of a multi-task visual base model based on a diffusion model includes a visual low-level semantic information learning adapter that can be expanded according to the number of tasks and a feedback-assisted learning mechanism;

[0043] S2. First, design a visual low-level semantic information learning adapter that can be expanded according to the number of tasks; insert the adapter into the task information input of the multi-task visual base model; include a set of parallel small-weight convolutional layer modules specific to different tasks, and connect a convolutional layer module shared between different tasks in series after these task-specific convolutional layer modules.

[0044] S3. Then design a feedback-assisted learning mechanism; this mechanism extracts the corresponding conditional image from the input image or output image of the multi-task visual basic model by introducing a perception model with fixed weight parameters (a perception model is a type of model that can extract conditional images from images, including semantic segmentation models (such as UPerNet models), depth estimation models (such as DPT models), edge detection models (such as EDTER), etc.), and obtains the conditional image at the input or output end. In the feedback-assisted learning mechanism, the output image at the output end is obtained by a one-step denoising strategy (in the reverse denoising process of the diffusion model, an image with a small amount of noise at a smaller time step can be obtained by one-step denoising) in consideration of training efficiency. A feedback loss is obtained by calculating the consistency information in the input and output conditional images in the image space; by minimizing the feedback loss, the multi-task visual basic model can be guided to be better trained to promote the improvement of the final image generation quality;

[0045] S4. Fix all or most of the original pre-trained weight parameters of the multi-task visual base model, while keeping the newly added visual low-level semantic information learning adapter able to be fine-tuned; introduce the feedback loss obtained by the feedback-assisted learning mechanism into the original multi-task visual base model training loss function, build the optimization target, and train the multi-task visual base model.

[0046] Furthermore, the construction process of the visual low-level semantic information learning adapter in step S2 that can be expanded according to the number of tasks includes:

[0047] Insert a low-level visual semantic information learning adapter at the task information input of the multi-task visual base model. If the multi-task visual base model contains K tasks, the adapter contains K parallel small-weight convolution modules, each of which is specific to learning low-level visual semantic information of only one task. Different task information is assigned to the corresponding task-specific convolution module by an indicator before the K parallel convolution modules, and then a convolution module shared by K tasks is inserted after these K parallel convolution modules, which can learn from the information of the previous K small-weight convolution modules. The adapter formula is: Where I(·) is the indicator function that assigns different task information to the corresponding small-weight convolutional modules, I k is the conditional image or query image from task k, is the i-th small weight convolution module in the parallel convolution module of the adapter, F shared-conv It is a convolution module shared by all parallel small-weight convolution modules.

[0048] Furthermore, the process of constructing a feedback-assisted learning mechanism in step S3 includes:

[0049] The query image at the input end of the multi-task visual basic model is a normal image or a conditional image (segmentation map, contour map, depth map, plane normal map, human posture map, etc.), and the generated image at the output end is a conditional image (segmentation map, contour map, depth map, plane normal map, human posture map, etc.) or a normal image.

[0050] For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism is to first use the perceptual model with fixed weight parameters to extract the corresponding conditional image from the image output by the one-step denoising strategy at the output end, and then calculate the MSE (mean square error) between the conditional image at the input end and the conditional image at the output end in the image space as the feedback loss L. Feedback ,like Figure 1 shown.

[0051] For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism is to first extract the corresponding conditional image from the image at the input end using the perceptual model with fixed weight parameters to obtain the conditional image at the input end, and then calculate the MSE (mean square error) between the conditional image at the input end and the conditional image output at the output end after using the one-step denoising strategy in the image space as the feedback loss L Feedback ,like Figure 1 shown.

[0052] Furthermore, the scheme for model training in step S4 includes:

[0053] Select a multi-task visual base model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task visual base model, while keeping the newly added visual low-level semantic information learning adapter able to be fine-tuned; train the loss function L in the original multi-task visual base model Train In the above example, the feedback loss L obtained by introducing the feedback-assisted learning mechanism is Feedback After that, the total loss function of the optimization objective is constructed as L Total =L Train +λL Feedback , where λ is the feedback loss weight; train the network.

[0054] InstructCV: InstructCV is a unified computer vision task language interface that converts a variety of computer vision tasks into text-to-image generation problems, guides task execution through natural language instructions, and uses multimodal multi-task training datasets to perform instruction tuning on the pre-trained text-to-image diffusion model, transforming it from a generation model to an instruction-guided multi-task visual learner, demonstrating excellent generalization capabilities for unseen data, categories, and user instructions.

[0055] InstructDiffusion: InstructDiffusion is a general computer vision task modeling interface that uses the denoising diffusion probability model (DDPM) to transform various computer vision tasks into image manipulation processes based on human intuitive instructions. The output space is a flexible and interactive pixel space. The model can handle a variety of visual tasks including understanding tasks (such as segmentation and key point detection) and generation tasks (such as editing and enhancement). It can even handle tasks that have not been seen in training and surpass previous methods on new data sets.

[0056] If InstructCV or InstructDiffusion is selected as the multi-task visual base model, all the original pre-trained weight parameters are fixed while the adapter parameters can be trained. If Prompt Diffusion is selected as the multi-task visual base model, only the visual low-level semantic information learning adapter and the ControlNet parameter part of the model are retained for fine-tuning, and all other weight parameters of Prompt Diffusion are fixed.

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but they are not intended to limit the present invention.

[0058] A method for improving the image generation quality of a multi-task visual base model based on a diffusion model, the method comprising a visual low-level semantic information learning adapter that can be expanded according to the number of tasks and a feedback-assisted learning mechanism; firstly, a visual low-level semantic information learning adapter that can be expanded according to the number of tasks is designed, and the adapter is inserted into the task information input end of the multi-task visual base model, and the adapter can be expanded according to the number of tasks, and the adapter comprises a set of parallel small-weight convolutional layer modules specific to different tasks, and then a shared convolutional layer module connected in series. Then, a feedback-assisted learning mechanism is designed, which extracts the corresponding conditional image from the input image or output image of the multi-task visual base model by introducing a perception model with fixed weight parameters (the perception model is a type of model that can extract conditional images from images, including semantic segmentation models (such as UPerNet models), depth estimation models (such as DPT models), edge detection models (such as EDTER), etc.), and obtains the conditional image at the input end or output end. In the feedback-assisted learning mechanism, the output image at the output end is obtained by a one-step denoising strategy in consideration of training efficiency. In the image space, the consistency information in the input and output conditional images is calculated to obtain a feedback loss. By minimizing the feedback loss, the multi-task visual base model can be guided to be better trained to promote the improvement of the final image generation quality; finally, all or most of the original pre-trained weight parameters of the multi-task visual base model are fixed, while keeping the newly added visual low-level semantic information learning adapter able to be fine-tuned; in the original multi-task visual base model training loss function, the feedback loss obtained by the feedback-assisted learning mechanism is introduced to construct the optimization target and train the entire network. The following takes a specific multi-task visual base model as an example to introduce the specific implementation steps:

[0059] Step 1: Select the Prompt Diffusion multi-task visual base model as the network backbone for this example. Prompt Diffusion is a diffusion model-based framework that enables contextual learning in image generation tasks. The model automatically understands the underlying task by using visual-linguistic cues, including a pair of example images and text guidance for a specific task, and performs the same task on a new query image while following the text guidance. The Prompt Diffusion model is capable of handling a variety of visual-linguistic tasks and is jointly trained on six different tasks, including forward and reverse tasks such as image segmentation, edge detection, and depth map generation.

[0060] Step 2: Insert a designed visual low-level semantic information learning adapter into the task information input end of the ControlNet part of Prompt Diffusion (the ControlNet part is a trainable replica network obtained by copying the encoder part and the middle layer of the diffusion model), for each input example image pair and query image, as shown in the figure.

[0061] Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image of Prompt Diffusion, i.e., the input image, can be a conditional image or a normal image according to different tasks.

[0062] When the query image is a conditional image, the model outputs a normal image. In this case, the perceptual model is first used to extract the corresponding conditional image from the image with a small amount of noise in the Prompt Diffusion back diffusion process after one-step denoising. Then, the MSE (mean square error) loss between the input and output conditional images is calculated in the image space as the feedback loss L. Feedback , in this case:

[0063] L Feedback =L(x query ,D(G(x t ,t,c t, ,c f )))

[0064] Where L is the mean square error (MSE), x query is the input conditional image at this time, D(·) is the perceptual model used, which can extract the conditional image from the normal image, and G represents the process of the model using a one-step denoising strategy to denoise the conditional information to generate an output image from an image with a small amount of noise. f is the conditional embedding information from the ControlNet part, c t It is the text embedding information of the model input.

[0065] When the query image is a normal image, the model outputs a conditional image. In this case, the perceptual model is first used to extract the corresponding conditional image from the normal image at the prompt diffusion input end. Then, the MSE (mean square error) loss between the conditional image output by the one-step denoising strategy at the input end and the output end is calculated in the image space as the feedback loss L. Feedback , in this case:

[0066] L Feedback =L(D(x query ),G(x t ,t,c t, ,cf ))

[0067] Where L is the mean square error (MSE), x query is the normal image of the input at this time, the perceptual model used by D(·) can extract the conditional image from the normal image, and G represents the process of the model using a one-step denoising strategy to denoise according to the conditional information to generate the output image from the image with a small amount of noise. f is the conditional embedding information from the ControlNet part, c t It is the text embedding information of the model input.

[0068] Step 4: After inserting the visual low-level semantic information learning adapter to build the entire network, fix most of the original pre-trained weight parameters of PromptDiffusion, and only keep the visual low-level semantic information learning adapter and the ControlNet part of the model for fine-tuning.

[0069] Step 5: Construct the loss function. The original training loss function of Prompt Diffusion is:

[0070]

[0071] Where x is the query image, the diffusion process adds noise to the encoded latent embedding z = ε(x), resulting in a noisy image latent embedding z t , z t is the latent embedding of the noisy image at time step t, c t is the text embedding, c f is a task-specific embedding, ∈ θ is the training network of the model, ∈ is a standard normal random variable.

[0072] Introduce the feedback loss L obtained by using the feedback-assisted learning mechanism in the image space Feedback After that, the total loss function is:

[0073] L Total =L Train +λL Feedback

[0074] where λ is the feedback loss weight.

[0075] Step 6: Set an appropriate learning rate and start training the entire network.

[0076] The perceptual model is a general term for models such as depth estimation models, semantic segmentation models, edge detection models, etc. In the present invention, the perceptual model is used as a tool to extract the corresponding conditional images (depth maps, segmentation maps, contour maps, etc. are called conditional images) from the image.

[0077] Take the specific example Prompt Diffusion cited in the present invention as an example. This model involves a total of 6 tasks, 3 of which are defined as forward tasks, and the other 3 are defined as reverse tasks.

[0078] For the forward task defined in the model (generating depth maps, segmentation maps, and contour maps from normal images), in this case, because the output is already a conditional image, the perceived image is used to extract the conditional image of the input image. In this way, there are conditional images at both the input and output ends, and the MSE loss can be calculated between the conditional images at the output and input ends to guide better training of the model.

[0079] For the reverse task defined in the model (generating a normal image from a depth map, a segmentation map, and a contour map), in this case, the input is a conditional image and the output is a normal image. Therefore, the present invention uses a perceptual model as a tool to extract a conditional image from the output normal image. In this way, both the input and output ends have conditional images, and the MSE loss can be calculated between the conditional images at the output and input ends to guide better training of the model.

[0080] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs can also change and modify the above embodiment. Therefore, the present invention is not limited to the above specific embodiment, and any obvious improvement, replacement or modification made by those skilled in the art on the basis of the present invention belongs to the protection scope of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.

Claims

1. A multi-task visual base model image generation method based on a diffusion model, characterized in that: It consists of two parts: a visual low-level semantic information learning adapter that is expanded according to the number of tasks and a multi-task visual basic model with a feedback-assisted learning mechanism. A visual low-level semantic information learning adapter that is expanded according to the number of tasks, comprising a set of parallel small-weight convolutional layer modules specific to different tasks, and a convolutional layer module shared between different tasks connected in series after these task-specific convolutional layer modules, the adapter is inserted into the task information input end of the multi-task visual base model, and the adapter is expanded according to the number of tasks; The feedback-assisted learning mechanism refers to: using a perception model with fixed weight parameters to extract the corresponding conditional image from the input image or output image of the multi-task visual base model, obtaining the conditional image of the input or output end of the multi-task visual base model, calculating the consistency information in the input and output conditional images in the image space to obtain a feedback loss, and by minimizing the feedback loss, guiding the multi-task visual base model to be better trained to promote the improvement of the final image generation quality; in the feedback-assisted learning mechanism, the output image of the output end is obtained by a one-step denoising strategy in consideration of training efficiency.

2. The multi-task visual base model image generation method based on the diffusion model as claimed in claim 1, characterized in that: The adapter is expanded according to the number of tasks. Specifically, if the visual base model contains K tasks, the adapter contains K parallel small weight convolution modules, each small weight convolution module is only specific to the visual low-level semantic information of one task for learning, and different task information is assigned to the corresponding task-specific convolution module by an indicator before the K parallel convolution modules, and then a convolution module shared by K tasks is inserted after the K parallel convolution modules, which can learn from the information of the previous K small weight convolution modules. The adapter formula is: Where I(·) is the indicator function that assigns different task information to the corresponding small-weight convolutional modules, I k is the conditional image or query image from task k, is the i-th small weight convolution module in the parallel convolution module of the adapter, F shared-conv It is a convolution module shared by all parallel small-weight convolution modules.

3. The multi-task visual base model image generation method based on the diffusion model as claimed in claim 1, characterized in that: The one-step denoising strategy used in the feedback-assisted learning mechanism means that in the reverse denoising process of the diffusion model, an image with a small amount of noise at a smaller time step is denoised in one step to obtain a clear image; The perceptual model introduced in the feedback-assisted learning mechanism is a type of model used to extract conditional images from images, specifically one of the following models: a UPerNet semantic segmentation model based on an encoder-decoder structure; a DPT depth estimation model based on an encoder-decoder structure, and an edge detection model edge detection framework EDTER.

4. The multi-task visual base model image generation method based on the diffusion model as claimed in claim 1, characterized in that: The query image at the input end of the multi-task visual basic model is a normal image or a conditional image including a segmentation map, a contour map, a depth map, a normal map, and a human posture map. The generated image at the output end is a conditional image including a segmentation map, a contour map, a depth map, a normal map, a human posture map, or a normal image. The conditional image is extracted from the normal image using a perception model with fixed weight parameters. For tasks where the input image is a conditional image and the output image is a normal image, the feedback-assisted learning mechanism is to first use a perception model with fixed weight parameters to extract the corresponding conditional image from the image output by the one-step denoising strategy at the output end, and then calculate the mean square error (MSE) between the conditional image at the input end and the conditional image at the output end in the image space as the feedback loss. For tasks where the input image is a normal image and the output image is a conditional image, the feedback-assisted learning mechanism is to first extract the corresponding conditional image from the image at the input end using the perceptual model with fixed weight parameters to obtain the conditional image at the input end, and then calculate the mean square error (MSE) between the conditional image at the input end and the conditional image output at the output end after using the one-step denoising strategy in the image space as the feedback loss; Select a multi-task visual base model as the network skeleton; fix all or most of the original pre-trained weight parameters of the multi-task visual base model, while keeping the newly added visual low-level semantic information learning adapter able to be fine-tuned; train the loss function L in the original multi-task visual base model Train In the above example, we introduce feedback loss L Feedback After that, the total loss function of the optimization objective is constructed as L Total =L Train +L Feedback , where λ is the feedback loss weight; set an appropriate learning rate and start training the entire network including the perception model and the multi-task visual base model.

5. The multi-task visual base model image generation method based on the diffusion model as claimed in claim 1, characterized in that: The detailed steps are as follows: Step 1: Select the multi-task visual base model Prompt Diffusion as the multi-task visual base model; Step 2: Insert a designed visual low-level semantic information learning adapter into the task information input of the ControlNet part of Prompt Diffusion for the input sample image pair and the query image. The ControlNet part is a trainable replica network obtained by copying the encoder part and the middle layer of the diffusion model. Step 3: Introduce a feedback-assisted learning mechanism into the model. The query image of Prompt Diffusion can be a conditional image or a normal image according to different tasks. When the query image is a conditional image, the model outputs a normal image. In this case, the perceptual model with fixed weight parameters is first used to extract the corresponding conditional image from the image with a small amount of noise in the Prompt Diffusion back diffusion process after one-step denoising. Then, the mean square error MSE loss between the input and output conditional images is calculated in the image space as the feedback loss L Feedback , in this case: L Feedback =L(x query ,D(G(x t ,t,c t, ,c f ))) Where L is the mean square error MSE, x query is the input conditional image at this time, D(·) is the perceptual model used to extract the conditional image from the normal image, G represents the process of the model using a one-step denoising strategy to denoise the conditional information to generate an output image from an image with a small amount of noise, c f is the conditional embedding information from the ControlNet part, c t is the text embedding information of the model input; When the query image is a normal image, the model outputs a conditional image. In this case, the corresponding conditional image is first extracted from the normal image at the prompt diffusion input end using the perception model with fixed weight parameters. Then, the mean square error (MSE) loss between the conditional image output by the one-step denoising strategy at the input and output ends is calculated in the image space as the feedback loss L. Feedback , in this case: L Feedback =L(D(x query ),G(x t ,t,c t, ,c f )) where x query is the normal image of the input at this time; Step 4: After inserting the visual low-level semantic information learning adapter to build the entire network, only the ControlNet parameter part of the visual low-level semantic information learning adapter and the Prompt Diffusion model is retained for fine-tuning, and all other weight parameters of PromptDiffusion are fixed; Step 5: Construct the loss function. The original training loss function of Prompt Diffusion is: Where x is the query image, the diffusion process adds noise to the encoded latent embedding z = ε(x), resulting in a noisy image latent embedding z t , z t is the latent embedding of the noisy image at time step t, c t is the text embedding, c f is a task-specific embedding, ∈ θ is the training network of the model, ∈ is a standard normal random variable; Introduce the feedback loss L obtained by using the feedback-assisted learning mechanism in the image space Feedback After that, the total loss function is: THE Total =L Train +λL Feedback Where λ is the feedback loss weight; Step 6: Set an appropriate learning rate and start training the entire network including the perception model and the multi-task visual base model.

Images