A text-guided multi-task image inpainting diffusion method, system and medium

Abstract

The present application relates to a kind of text guide-based multi-task image repair diffusion method, system and medium, the method includes: obtaining the degraded image to be repaired and the text guide sentence corresponding to degraded image;Text guide sentence is encoded, and text semantic feature is obtained, the feature extraction is carried out to degraded image, and visual feature is obtained, text semantic feature and visual feature are interactively fused, and fusion feature is obtained;According to fusion feature, the routing weight of multiple expert networks is generated, at least one expert network is selected according to routing weight, and visual feature and fusion feature are forward calculated, and the output result of each expert network is weighted fusion, and repair feature is obtained;Based on diffusion model, image reconstruction is carried out to repair feature, and repair image is output.The present application can process multiple image degradation tasks under unified model framework, can also realize semantic controllable repair using natural language instruction, while considering repair performance, inference efficiency and generalization ability.

Description

Technical Field

[0001] This invention relates to the fields of image processing, computer vision, artificial intelligence and multimodal information processing, and in particular to a text-guided multi-task image restoration and diffusion method, system and medium. Background Technology

[0002] Image restoration technology is widely used in scenarios such as intelligent security, autonomous driving, mobile terminal imaging, remote sensing observation, medical auxiliary analysis, and digital content editing. Actual images are often affected by various degradation factors such as shadow occlusion, motion blur, raindrop adhesion, partial occlusion, uneven lighting, and noise pollution, leading to lost image details, structural distortion, and incomplete semantic expression, which in turn affects the accuracy of subsequent recognition, detection, and decision-making tasks.

[0003] Existing image inpainting methods typically build dedicated models for each specific degradation type. For example, shadow removal models mainly focus on illumination compensation and color restoration, deblurring models focus on high-frequency detail restoration and edge sharpening, while raindrop removal models emphasize local occlusion removal and background texture reconstruction. Although these single-task models can achieve good results on specific tasks, in real-world scenarios, degradation types are complex and diverse, and users cannot predict in advance which dedicated model to use. Therefore, the coexistence of multiple models is not only complex to deploy, but also incurs high model switching costs and poor versatility.

[0004] In recent years, diffusion models have demonstrated strong reconstruction capabilities in image generation and restoration tasks, generating high-quality results through progressive denoising. However, most existing diffusion restoration models are still primarily single-task modeling, lacking a unified processing mechanism applicable to various degradation types. Even when training with a mix of degraded images, a single shared backbone network is prone to parameter competition and representation conflicts between tasks, making it difficult for the model to achieve stable and excellent performance simultaneously across different degradation scenarios.

[0005] Furthermore, most existing image inpainting methods rely primarily on visual input for inference, lacking explicit expression and utilization of the user's inpainting intent. This makes it difficult for the model to flexibly adjust its inpainting strategy based on natural language instructions. Even when some solutions introduce textual conditions, they often treat the text merely as superficial supplementary information, failing to effectively utilize textual semantics in the model's internal dynamic decision-making process. In particular, they lack textual semantic-based task recognition and expert routing mechanisms.

[0006] Therefore, how to construct an image restoration scheme that can handle multiple image degradation tasks under a unified model framework, achieve semantically controllable restoration using natural language instructions, and simultaneously take into account restoration performance, inference efficiency, and generalization ability has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] This invention provides a text-guided multi-task image restoration diffusion method, system, and medium, aiming to achieve both the ability to handle multiple image degradation tasks within a unified model framework and the ability to achieve semantically controllable restoration using natural language instructions, while simultaneously considering restoration performance, inference efficiency, and generalization ability.

[0008] This invention provides a text-guided multi-task image inpainting diffusion method, the method comprising the following steps: Step S10, construct input information: obtain the degraded image to be repaired and the text guidance statement corresponding to the degraded image, wherein the text guidance statement is used to describe the degradation type, degradation location, degradation degree and expected repair target; Step S20, Feature Extraction and Cross-Modal Fusion: The text guidance statement is encoded to obtain text semantic features, the degraded image is feature extracted to obtain visual features, and the text semantic features and the visual features are interactively fused through the cross-modal fusion module to obtain fused features; Step S30, Dynamic expert route repair: Using a cross-modal gating network, route weights of multiple expert networks are generated based on the fusion features. At least one expert network is selected based on the route weights to perform forward computation on the visual features and the fusion features. The outputs of each expert network are then weighted and fused to obtain the repaired features. Step S40, Diffusion Reconstruction Output: Perform image reconstruction on the repair features based on the diffusion model, and output the repaired image.

[0009] A further technical solution of the present invention is that step S10 specifically includes the following steps: Step S101: Obtain the degraded image to be repaired. ; Step S102, acquire the degraded image Corresponding text guidance statement The text guidance statement This includes information on degradation type, degradation location, degradation degree, and desired repair target. Step S103, the degraded image and the text guidance statement As a conditional input for the same repair task, based on the text guidance statement Determine the degraded image The corresponding degenerate semantics.

[0010] A further technical solution of the present invention is that, in step S20, a pre-trained text encoder is used to encode the text guidance statement, and the text guidance statement is represented as follows: ,in express The first in Each word element is encoded to obtain the text semantic features. : (1) Furthermore, regarding the degraded image Visual feature extraction is performed to obtain visual features. : (2) A further technical solution of the present invention is that, in step S20, the text semantic features are processed through a cross-attention mechanism. and the aforementioned visual features The integration process specifically includes: Step S201: Calculate the attention weight matrix of visual features to text semantic features. : (3) Step S202, according to the attention weight matrix Generate enhanced text features : (4) Step S203, for the visual features and the enhanced text features Perform residual fusion to obtain fusion features. : (5) in, , and All are learnable projection matrices. Represents the feature dimension; Representing text semantic features transpose, Represents the projection matrix transpose, This represents the normalization factor used to scale the attention weights.

[0011] A further technical solution of the present invention is that step S30 includes the following steps: Step S301, the fusion feature Input a cross-modal gating network to generate the weight distributions of multiple expert networks. : (6) in, This represents the parameter matrix of the gated network. Indicates the bias term. , Let represent the number of expert networks, and satisfy the following condition: (7) in, This represents an expert network index. Indicates the first The routing weights corresponding to each expert network; Step S302, the visual features and the fusion features Enter the first A network of experts Expert restoration features obtained : (8) Step S303: Distribute the outputs of multiple expert networks according to the weights. Weighted fusion is performed to obtain the repair features. : (9) A further technical solution of the present invention is that step S40 includes the following steps: Step S401, the repair feature The reconstruction phase of the input diffusion model; Step S402, using the diffusion model on the repair features Under the conditional constraints, stepwise denoising and image reconstruction are performed to obtain the reconstructed image. ; Step S403, the reconstructed image As a repaired image Output; The reconstruction process of the diffusion model is expressed as follows: (10) in, Indicates the first The diffusion state of the step, This indicates the diffusion and reconstruction of the network. Indicates the first The reconstruction status of the step.

[0012] A further technical solution of the present invention is to optimize the model using a joint loss function during the training process, wherein the joint loss function satisfies: (11) in, Indicates the total loss. This indicates the loss of the repair task. This indicates the expert load balancing loss. The balance coefficient is represented by the expert load balancing loss, which satisfies the following:

[0013] in, This represents the utilization vector of each expert network. Indicates the first The average probability of an expert network being selected in a training batch. KL(·∥·) represents the target utilization distribution; KL(·∥·) represents the Kullback-Leibler divergence. Indicates the number of expert networks. This represents an expert network index. Represents the distribution of target utilization rate The first in Each component.

[0014] To achieve the above objectives, the present invention also proposes a text-guided multi-task image restoration and diffusion system, the system comprising a memory, a processor, and a text-guided multi-task image restoration and diffusion program stored in the memory, wherein the text-guided multi-task image restoration and diffusion program is executed by the processor to perform the steps of the method described above.

[0015] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a text-guided multi-task image restoration and diffusion program, which, when run by a processor, executes the steps of the method described above.

[0016] The beneficial effects of the text-guided multi-task image inpainting diffusion method, system, and medium of this invention are: This invention directly incorporates textual semantics into the cross-modal fusion and expert routing process of a multi-task image restoration diffusion model, enabling the unified model to identify the degradation type and restoration target corresponding to the current degraded image based on textual guidance statements, thereby alleviating representation conflicts between different degradation tasks in multi-task mixed training. This invention establishes a connection between textual semantics and visual features through a cross-modal fusion module, improving the model's responsiveness to user restoration intentions. This invention achieves adaptive restoration path selection for different degradation types through a cross-modal gating network and multiple expert networks, reducing the complexity of deploying multiple dedicated models for different degradation tasks. This invention outputs restored images through the diffusion reconstruction process, which helps improve the structural consistency and naturalness of the restoration results. This invention also constrains expert utilization through expert load balancing loss, which helps avoid the overuse of a few experts, improving model training stability and generalization ability. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the text-guided multi-task image restoration and diffusion method of the present invention. Figure 2 This is a schematic diagram of the cross-modal fusion module and the degenerate text-hybrid expert module of the present invention; Figure 3 This is a schematic diagram of the hardware architecture of the text-guided multi-task image restoration and diffusion system of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0019] This invention proposes a text-guided multi-task image inpainting diffusion method, such as... Figure 1 As shown, a preferred embodiment of the text-guided multi-task image inpainting diffusion method of the present invention includes the following steps: Step S10, construct input information: obtain the degraded image to be repaired and the text guidance statement corresponding to the degraded image.

[0020] The degraded image can be a shadow degraded image, a motion-blurred degraded image, a raindrop occlusion degraded image, or a noise, haze, scratch, compression artifact, or composite degraded image.

[0021] The text guidance statement is used to describe the degradation type, degradation location, degradation degree, and expected repair target.

[0022] For example, for a shadow-degraded image, the text guidance statement could be "Shadows exist in the image; please remove the shadows and restore normal lighting"; for a motion-blurred degraded image, the text guidance statement could be "The image is blurred due to motion; please restore sharp edges and textures"; for a raindrop-occluded degraded image, the text guidance statement could be "Raindrops exist in the image; please remove the raindrops and restore background details." In practical use, the text guidance statements can be input by the user or automatically generated by the multimodal large model based on the degraded image and then manually reviewed.

[0023] Step S10 specifically includes the following steps: Step S101: Obtain the degraded image to be repaired. ; Step S102, acquire the degraded image Corresponding text guidance statement The text guidance statement This includes information on degradation type, degradation location, degradation degree, and desired repair target. Step S103, the degraded image and the text guidance statement As a conditional input for the same repair task, based on the text guidance statement Determine the degraded image The corresponding degenerate semantics.

[0024] Step S20, Feature Extraction and Cross-Modal Fusion: The text guidance statement is encoded to obtain text semantic features, the degraded image is feature extracted to obtain visual features, and the text semantic features and the visual features are interactively fused through the cross-modal fusion module to obtain fused features.

[0025] like Figure 2 As shown, in this embodiment, step S20 includes three stages: text encoding, visual feature extraction, and cross-modal fusion.

[0026] In step S20, a pre-trained text encoder is used to encode the text guidance statement, which is represented as follows: ,in express The first in Each word element is encoded to obtain the text semantic features. : (1) Furthermore, regarding the degraded image Visual feature extraction is performed to obtain visual features. : (2) In this embodiment, the text guidance statement is first... Input text encoder to obtain text semantic features The text encoder preferably uses a pre-trained BERT encoder, but other Transformer text encoders or lightweight text encoding networks, such as RoBERTa, CLIP text encoder, or other pre-trained language models, can also be used. (Text guiding statements) After word segmentation, word embedding and multi-layer self-attention encoding, text semantic features that can express degradation type and repair target are formed. The calculation process is shown in Equation (1).

[0027] Then, the degraded image Input the visual feature extraction network in the image encoder or diffusion model to obtain visual features. .

[0028] Visual feature extraction networks are not limited to a single convolutional structure; they can also employ U-Net, Restormer, SwinTransformer, or other image recovery backbones. A visual feature extraction network can consist of convolutional layers, residual blocks, Transformer blocks, U-Net coding layers, or a combination of these structures, used to extract low-level texture information, high-level structural information, and degradation region information from degraded images. Its calculation process is shown in equation (2).

[0029] In step S20, the text semantic features are processed through a cross-attention mechanism. and the aforementioned visual features The fusion process is as follows: Step S201: Calculate the attention weight matrix of visual features to text semantic features. : (3) Representing text semantic features transpose, Represents the projection matrix transpose, This represents the normalization factor used to scale the attention weights.

[0030] This embodiment utilizes a cross-attention mechanism to fuse text semantic features. and visual features In a preferred embodiment, visual features are... As a query vector, text semantic features As key vectors and value vectors, they are learned through projection matrices. , and Attention weights of computational visual features to text semantics .

[0031] Step S202, according to the attention weight matrix Generate enhanced text features : (4) This embodiment is based on attention. Weighting the semantic features of the text yields enhanced text features that are relevant to the current visual content. .

[0032] Step S203, for the visual features and the enhanced text features Residual fusion and nonlinear activation are performed to obtain fusion features. : (5) in, , and All are learnable projection matrices. Indicates the feature dimension.

[0033] In this embodiment, through the aforementioned cross-modal fusion process, semantic information such as "shadow," "blur," "raindrop," "remove," and "restore clarity" in the text guidance statement can participate in the expression of visual features, enabling the model to not only repair based on pixel information but also to clarify the current repair task based on natural language instructions.

[0034] Step S30, Dynamic Expert Route Repair: A cross-modal gating network is used to generate routing weights for multiple expert networks based on the fusion features. At least one expert network is selected based on these routing weights to perform forward computation on the visual features and the fusion features. The outputs of each expert network are then weighted and fused to obtain the repaired features. Number of expert networks. It can be adjusted according to the complexity of the task.

[0035] In this embodiment, step S30 includes three stages: cross-modal gating, expert forward computation, and expert output fusion.

[0036] Step S30 includes the following steps: Step S301, the fusion feature Input a cross-modal gating network to generate the weight distributions of multiple expert networks. : (6) in, This represents the parameter matrix of the gated network. Represents the bias term and weight distribution. Represented as: , Let represent the number of expert networks, and satisfy the following condition: (7) in, This represents an expert network index. Indicates the first The routing weights corresponding to each expert network.

[0037] Cross-modal gating networks can be implemented using fully connected layers, multilayer perceptrons, attention routing networks, or combinations thereof. This is because the input to a gating network is a fusion of textual semantics and visual content. Therefore, its output expert weights can simultaneously reflect the visual degradation pattern of the degraded image and the restoration intent in the text guidance statement. For example, when the text guidance statement contains semantics such as "shadow" or "remove shadow", the gating network tends to increase the weight of experts related to shadow restoration; when the text guidance statement contains semantics such as "blur" or "sharpen", the gating network tends to increase the weight of experts related to deblurring; when the text guidance statement contains semantics such as "raindrop" or "remove raindrop", the gating network tends to increase the weight of experts related to raindrop removal.

[0038] Step S302, the visual features and the fusion features Enter the first A network of experts Expert restoration features obtained : (8) Expert Network It can be composed of multiple Transformer blocks, each of which may include a self-attention layer, a normalization layer, an activation function, and residual connections; expert network It can also be composed of convolutional networks, residual networks, U-Net submodules, attention enhancement networks, or combinations of the above structures. Multiple expert networks can have the same structure, or different structures can be used depending on the type of degradation. During training, each expert network gradually develops its specialization capabilities through samples allocated by the gating network, making some experts more suitable for illumination compensation and color restoration, some more suitable for edge sharpening and high-frequency texture restoration, and some more suitable for local occlusion removal and background reconstruction.

[0039] Step S303: Distribute the outputs of multiple expert networks according to the weights. Weighted fusion is performed to obtain the repair features. : (9) Step S40, Diffusion Reconstruction Output: Perform image reconstruction on the repair features based on the diffusion model, and output the repaired image.

[0040] Step S40 specifically includes the following steps: Step S401, the repair feature The reconstruction phase of the input diffusion model.

[0041] Step S402, using the diffusion model on the repair features Under the conditional constraints, stepwise denoising and image reconstruction are performed to obtain the reconstructed image. .

[0042] Step S403, the reconstructed image As a repaired image Output.

[0043] This embodiment will repair features. Input the diffusion model into the reconstruction stage, enabling the diffusion model to repair features. Under the conditional constraints, stepwise denoising and image reconstruction are performed to obtain the reconstructed image. and will reconstruct the image As a repaired image Output. The diffusion reconstruction process can be represented as: (10) in, Indicates the first The diffusion state of the step, This indicates the diffusion and reconstruction of the network. Indicates the first The reconstruction state is determined step by step. Diffusion reconstruction networks can be implemented using U-Net structures, residual diffusion networks, or Transformer diffusion networks. During sampling, the model starts from the initial noisy state or the intermediate state corresponding to the degraded image, and at each step, it reconstructs the state based on the repair features. Predict the denoising direction or residual components, gradually restore the image structure and texture details, and finally output the repaired image. .

[0044] During the training phase, this invention can employ a multi-task hybrid degradation training approach. Training samples can include at least two of the following tasks: shadow removal, motion blur removal, and raindrop removal. Each sample includes a degraded image, a textual guidance statement, and a clear target image. To enable the model to learn the correspondence between textual semantics and degradation types, a textual guidance statement describing its degradation type and restoration target can be configured for each degraded image. The textual guidance statement can be generated by a multimodal large language model and undergo manual review or rule-based filtering to ensure semantic accuracy and concise expression. During training, samples of different degradation types are input into the same image restoration diffusion model, allowing the model to learn multiple degradation restoration rules within a unified parameter framework.

[0045] In one specific embodiment, the ISTD dataset can be selected as the sample source for the shadow degradation task, the GoPro dataset as the sample source for the motion blur degradation task, and the RainDrop dataset as the sample source for the raindrop occlusion degradation task. The input image can be cropped or scaled to 256×256 pixels, and the pixel values ​​can be normalized to a preset range. For shadow images, text guidance statements such as "Shadows exist in the image; please remove the shadows and restore the original colors" can be generated; for motion-blurred images, text guidance statements such as "The image has obvious motion blur; please restore the clear image" can be generated; for raindrop-occluded images, text guidance statements such as "Raindrops cover the image; please remove the raindrops and restore background details" can be generated. The specific datasets and text examples described above are only for illustrating the implementation of the present invention and do not limit the data sources and degradation types applicable to the present invention.

[0046] During training, this invention employs a joint loss function to optimize the model. The joint loss function includes repair task loss and expert load balancing loss, and satisfies the following: (11) in, Indicates the total loss; The loss represents the restoration task loss, which is used to constrain the difference between the output restored image and the target clear image. It can include L1 loss, L2 loss, perceptual loss, structural similarity loss, or a combination of the above losses. This represents the expert load balancing loss, used to constrain the distribution of expert network utilization and prevent the gating network from being biased towards a few experts in the long term. This represents the balancing coefficient, a hyperparameter used to balance the losses from repair tasks and expert load balancing. The expert load balancing loss can be defined as:

[0047] KL(·∥·) represents the Kullback-Leibler divergence. Indicates the number of expert networks. This represents an expert network index. Represents the distribution of target utilization rate The first in Each component.

[0048] Among them, the utilization vector of each expert network in a training batch Represented as:

[0049] in, Indicates the first The average probability of an expert network being selected in a training batch, and the distribution of target utilization. Represented as: .

[0050] When the expert utilization rate is close to a uniform distribution, the load balancing loss is small; when a few experts are overused, the load balancing loss increases, thus guiding the gating network to adjust its expert allocation strategy.

[0051] In a preferred training configuration of this embodiment, the text encoder is not limited to BERT, and can also use RoBERTa, CLIP text encoder, or other pre-trained language models; the visual feature extraction network is not limited to a single convolutional structure, and can also use U-Net, Restormer, Swin Transformer, or other image recovery backbones; the number of expert networks... Adjustable based on task complexity; Top-k sparse activation The model can be adjusted according to the inference speed and restoration quality requirements; the diffusion model can adopt the residual diffusion model, conditional diffusion model, latent space diffusion model or other image restoration diffusion framework; the degradation types are not limited to shadows, motion blur and raindrops, but can also be extended to noise, haze, scratches, low light, compression artifacts and composite degradation.

[0052] Preferably, in this embodiment, the batch size can be set to 8, the number of experts $n$ can be set to 4, and the Top-k sparse activation... The number of training steps can be set to 2, the number of sampling steps can be set to 1000, the number of sampling steps during the inference phase can be reduced to improve efficiency, the Adam optimizer can be used, and the initial learning rate can be set to... The above parameters can be adjusted based on video memory conditions, image resolution, training sample size, and actual application scenarios.

[0053] During the inference phase, after the user inputs the image to be repaired and the text guidance statement, this invention first encodes the text guidance statement and extracts visual features from the image to be repaired; then, it fuses the text semantics and visual features through a cross-attention module; next, a cross-modal gating network selects an appropriate expert network to perform the repair based on the fused features; finally, a diffusion model outputs the repaired image based on the repair features. When the user inputs "remove shadows from the image and restore normal lighting," the model tends to select an expert suitable for lighting restoration; when the user inputs "eliminate motion blur and restore sharp edges," the model tends to select an expert suitable for detail sharpening; when the user inputs "remove raindrops and restore background texture," the model tends to select an expert suitable for local occlusion removal.

[0054] The beneficial effects of the text-guided multi-task image inpainting diffusion method of this invention are: This invention directly incorporates textual semantics into the cross-modal fusion and expert routing process of a multi-task image restoration diffusion model, enabling the unified model to identify the degradation type and restoration target corresponding to the current degraded image based on textual guidance statements, thereby alleviating representation conflicts between different degradation tasks in multi-task mixed training. This invention establishes a connection between textual semantics and visual features through a cross-modal fusion module, improving the model's responsiveness to user restoration intentions. This invention achieves adaptive restoration path selection for different degradation types through a cross-modal gating network and multiple expert networks, reducing the complexity of deploying multiple dedicated models for different degradation tasks. This invention outputs restored images through the diffusion reconstruction process, which helps improve the structural consistency and naturalness of the restoration results. This invention also constrains expert utilization through expert load balancing loss, which helps avoid the overuse of a few experts, improving model training stability and generalization ability.

[0055] To achieve the above objectives, the present invention also proposes a text-guided multi-task image restoration and diffusion system.

[0056] like Figure 3 As shown, the system includes a processor 1001, a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002, and a text-guided multi-task image restoration and diffusion program stored on the processor. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art will understand that Figure 3 The system structure shown does not constitute a limitation on the system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0058] like Figure 3 As shown, the memory 1005, which serves as a computer storage medium, may include an operating device, a network communication module, a user interface module, and a text-guided multi-task image restoration and diffusion program.

[0059] exist Figure 3In the system shown, the network interface 1004 is mainly used to connect to the network server and communicate with the network server; the user interface 1003 is mainly used to interact with the user terminal and receive user input instructions; and the processor 1001 can be used to call the text-guided multi-task image restoration and diffusion program stored in the memory 1005.

[0060] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a text-guided multi-task image restoration and diffusion program, wherein the text-guided multi-task image restoration and diffusion program is executed by a processor to perform the steps of the method described above, which will not be repeated here.

[0061] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A text-guided multi-task image inpainting diffusion method, characterized in that, The method includes the following steps: Step S10, construct input information: obtain the degraded image to be repaired and the text guidance statement corresponding to the degraded image, wherein the text guidance statement is used to describe the degradation type, degradation location, degradation degree and expected repair target; Step S20, Feature Extraction and Cross-Modal Fusion: The text guidance statement is encoded to obtain text semantic features, the degraded image is feature extracted to obtain visual features, and the text semantic features and the visual features are interactively fused through the cross-modal fusion module to obtain fused features; Step S30, Dynamic expert route repair: Using a cross-modal gating network, route weights of multiple expert networks are generated based on the fusion features. At least one expert network is selected based on the route weights to perform forward computation on the visual features and the fusion features. The outputs of each expert network are then weighted and fused to obtain the repaired features. Step S40, Diffusion Reconstruction Output: Based on the diffusion model, perform image reconstruction on the repair features and output the repaired image; Step S40 includes the following steps: Step S401, repair features The reconstruction phase of the input diffusion model; Step S402, using the diffusion model on the repair features Under the conditional constraints, stepwise denoising and image reconstruction are performed to obtain the reconstructed image. ; Step S403, the reconstructed image As a repaired image Output; The reconstruction process of the diffusion model is expressed as follows: (10) in, Indicates the first The diffusion state of the step, This indicates the diffusion and reconstruction of the network. Indicates the first The reconstruction status of the step; Step S30 includes the following steps: Step S301, the fusion feature Input a cross-modal gating network to generate the weight distributions of multiple expert networks. : (6) in, This represents the parameter matrix of the gated network. Indicates the bias term. , Let represent the number of expert networks, and satisfy the following condition: (7) in, This represents an expert network index. Indicates the first The routing weights corresponding to each expert network; Step S302, the visual features and the fusion features Enter the first A network of experts Expert restoration features obtained : (8) Step S303: Distribute the outputs of multiple expert networks according to the weights. Weighted fusion is performed to obtain the repair features. : (9)。 2. The text-guided multi-task image inpainting diffusion method according to claim 1, characterized in that, Step S10 specifically includes the following steps: Step S101: Obtain the degraded image to be repaired. ; Step S102, acquire the degraded image Corresponding text guidance statement The text guidance statement This includes information on degradation type, degradation location, degradation degree, and desired repair target. Step S103, the degraded image and the text guidance statement As a conditional input for the same repair task, based on the text guidance statement Determine the degraded image The corresponding degenerate semantics.

3. The text-guided multi-task image inpainting diffusion method according to claim 1, characterized in that, In step S20, a pre-trained text encoder is used to encode the text guidance statement, which is represented as follows: ,in express The first in Each word element is encoded to obtain the text semantic features. : (1) Furthermore, regarding the degraded image Visual feature extraction is performed to obtain visual features. : (2)。 4. The text-guided multi-task image inpainting diffusion method according to claim 3, characterized in that, In step S20, the text semantic features are processed through a cross-attention mechanism. and the aforementioned visual features The integration process specifically includes: Step S201: Calculate the attention weight matrix of visual features to text semantic features. : (3) Step S202, according to the attention weight matrix Generate enhanced text features : (4) Step S203, for the visual features and the enhanced text features Perform residual fusion to obtain fusion features. : (5) in, , and All are learnable projection matrices. Represents the feature dimension; Representing text semantic features transpose, Represents the projection matrix transpose, This represents the normalization factor used to scale the attention weights.

5. The text-guided multi-task image inpainting diffusion method according to claim 1, characterized in that, During training, a joint loss function is used to optimize the model, and the joint loss function satisfies: (11) in, Indicates the total loss. This indicates the loss of the repair task. This indicates the expert load balancing loss. The balance coefficient is represented by the expert load balancing loss, which satisfies the following: in, This represents the utilization vector of each expert network. Indicates the first The average probability of an expert network being selected in a training batch. KL(·∥·) represents the target utilization distribution; KL(·∥·) represents the Kullback-Leibler divergence. Indicates the number of expert networks. This represents an expert network index. Represents the distribution of target utilization rate The first in Each component.

6. A text-guided multi-task image restoration and diffusion system, characterized in that, The system includes a memory, a processor, and a text-guided multitasking image restoration and diffusion program stored in the memory, the text-guided multitasking image restoration and diffusion program being executed by the processor to perform the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a text-guided multi-task image restoration and diffusion program, which, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 5.

Images