Lightweight multi-task context control image processing system, method and device

Abstract

This invention discloses a lightweight multi-task context control image processing system, including a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected sequentially from high to low spatial resolution. The first and second image encoders are used to extract features and downsample images from sample paths and control paths. The multi-scale fusion units are used to fuse query sample features and task control features at different spatial resolutions. Each multi-scale fusion unit includes a residual convolution module, a sliding window cross-attention module, a feature linear modulation module, a refinement module, and a downsampling module. This invention uses lightweight modules such as depthwise separable convolution and group normalization, significantly reducing the number of parameters compared to traditional control networks. The output feature channel is compatible with downstream diffusion models and can be plugged into existing diffusion models without extensive modifications to the original model architecture.

Description

Technical Field

[0001] This invention relates to the fields of image processing and machine learning, and specifically to a lightweight image processing system, method, and device with multi-task context control. Background Technology

[0002] In recent years, diffusion models have made significant progress in image generation and transformation, demonstrating strong diversity and generative capabilities by generating high-quality images through progressive denoising. Meanwhile, visual tasks such as semantic segmentation, depth estimation, and edge detection have also become research hotspots. To enable diffusion models to perform multiple visual tasks simultaneously, a common approach is to introduce an additional "control network" (such as ControlNet) into the model. This network provides conditional signals for different tasks, offering task-related information to guide the model in generating results specific to that task. However, existing control networks typically suffer from the following shortcomings: for example, traditional methods often construct separate control branches for each task, leading to a sharp increase in model parameter size and an inability to share low-level visual features; furthermore, these methods struggle to utilize contextual example images for task guidance, only providing control signals for a single task, thus limiting the adaptability and generalization ability of diffusion models in multi-task scenarios.

[0003] Therefore, it is necessary to provide a lightweight and general control network structure that can efficiently combine the information of the query image with multi-task example pairs without introducing too many additional parameters, so that a single diffusion model can simultaneously complete multiple visual tasks, including semantic segmentation, depth estimation and contour detection, while maintaining lightweight parameters and computational resources. Summary of the Invention

[0004] This invention provides a lightweight multi-task context control image processing system, method, and device to solve the technical problems existing in the prior art.

[0005] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0006] A lightweight multi-task context control image processing system includes a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution.

[0007] The first image encoder is used for feature extraction and downsampling of images from the sample path; its input is the query sample image, and its output is the query sample input features. ;

[0008] The second image encoder is used to extract features and downsample control example image pairs from the control path; its input consists of control example image pairs stitched together from example source images and target images, and its output is control input features. ;

[0009] The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module;

[0010] The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal embedding information; its inputs are query sample features and task control features, and its output is query sample features fused with temporal information. and task control features ;

[0011] The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ;

[0012] The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ;

[0013] The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ;

[0014] The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; it takes as input the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution downscaling are used as the input features for the query samples of the next multi-scale fusion unit; these features control the task. The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit.

[0015] The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

[0016] Furthermore, the depthwise separable convolutional module extracts spatial features according to the following formula. ;

[0017] ;

[0018] The compression and excitation modules are based on the following formulas from spatial features. Generate spatial features :

[0019] ;

[0020] ;

[0021] In the formula:

[0022] This represents the spatial features extracted by the depthwise separable convolutional module;

[0023] This represents the spatial features generated by the compression and excitation modules;

[0024] This represents the modulation fusion feature output by the characteristic linear modulation module;

[0025] This represents the channel attention weight vector generated by the compression and excitation module;

[0026] This represents the first transformation matrix in the compression and excitation module;

[0027] This represents the second transformation matrix in the compression and excitation module.

[0028] This represents a depthwise separable convolution function;

[0029] This represents a pointwise convolution operation;

[0030] This represents the Sigmoid activation function;

[0031] This represents the global average pooling function;

[0032] Represents the ReLU activation function;

[0033] This represents a shape reshaping function;

[0034] This represents channel-by-channel multiplication;

[0035] Furthermore, the sliding window cross-attention module generates Q, K, and V tensors according to the following formula:

[0036] ;

[0037] ;

[0038] ;

[0039] In the formula:

[0040] Represents a query tensor;

[0041] Represents the key tensor;

[0042] Represents a value tensor;

[0043] This represents the query sample features output by the residual convolution module;

[0044] This represents the task control features output by the residual convolution module;

[0045] Represents the learnable query projection matrix;

[0046] Represents the learnable bond projection matrix;

[0047] The projection matrix represents the learnable values;

[0048] This represents the group normalization function.

[0049] Furthermore, during the window partitioning process, the sliding window cross-attention module alternates between standard window partitioning and partitioning by translating half a window size along the spatial dimension between sequentially connected multi-scale fusion units. This allows attention calculation to capture local contextual information within the window at each scale. Within each window, the cross-attention module calculates cross-attention according to the following formula:

[0050] ;

[0051] In the formula:

[0052] The dimension representing the projected feature;

[0053] This indicates the query characteristics within a single window;

[0054] Represents key features within a single window;

[0055] This represents the characteristics of values ​​within a single window;

[0056] This represents the attention calculation function;

[0057] This represents the softmax function;

[0058] The sliding window cross-attention module generates attention fusion features according to the following formula:

[0059] ;

[0060] In the formula:

[0061] This represents the query sample features output by the residual convolution module;

[0062] This represents the attention fusion features generated by the sliding window cross-attention module;

[0063] Indicates the learnable scaling factor;

[0064] This represents the learnable output projection matrix;

[0065] This represents the inverse window transform function.

[0066] Furthermore, the characteristic linear modulation module generates modulation parameters using a multilayer perceptron according to the following formula:

[0067] ;

[0068] In the formula:

[0069] Indicates the scaling parameter;

[0070] Indicates the offset parameter;

[0071] Represents the time step embedding vector;

[0072] This represents the task control features output by the residual convolution module;

[0073] This represents the global average pooling function;

[0074] Indicates the concatenation function;

[0075] Represents the function of a multilayer perceptron;

[0076] The modulation fusion features are generated according to the following formula:

[0077] ;

[0078] In the formula:

[0079] This represents the fusion features generated by the modulation fusion module;

[0080] This represents the attention fusion features generated by the sliding window cross-attention module;

[0081] Represents the group normalization function;

[0082] This indicates channel-by-channel multiplication.

[0083] The present invention also provides a lightweight multi-task context control image processing method, which constructs an image processing system. The image processing system is configured with a first image encoder, a second image encoder, a multi-scale fusion module and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution.

[0084] The first image encoder is used for feature extraction and downsampling of images from the sample path; its input is the query sample image, and its output is the query sample input features. ;

[0085] The second image encoder is used to extract features and downsample control example image pairs from the control path; its input consists of control example image pairs stitched together from example source images and target images, and its output is control input features. ;

[0086] The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module;

[0087] The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal embedding information; its inputs are query sample features and task control features, and its output is query sample features fused with temporal information. and task control features ;

[0088] The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ;

[0089] The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ;

[0090] The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ;

[0091] The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; it takes as input the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution downscaling are used as the input features for the query samples of the next multi-scale fusion unit; these features control the task. The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit.

[0092] The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

[0093] Furthermore, the time step is mapped to a high-dimensional time representation through a standard temporal projection layer or a sin / cos-based temporal encoder, and then the temporal embedding vector is obtained through linear transformation or MLP. The vector is injected into the residual convolution module and the feature linear modulation module.

[0094] Furthermore, when training the image processing system, the training data consists of multi-task control example image pairs and query images, and the training loss function uses noise prediction loss; data augmentation and regularization strategies are selected to improve the robustness of the model.

[0095] Multitasking is divided into two main categories: forward tasks and reverse tasks. Forward tasks include generating semantic segmentation maps from images, generating depth maps from images, and generating contour maps from images. Reverse tasks include generating images from semantic segmentation maps, generating images from depth maps, and generating images from contour maps. By explicitly providing task context information in the form of controlled example image pairs, a single system can support multiple tasks.

[0096] Furthermore, the output feature processing module is connected to the target diffusion model, which has multiple downsampling blocks. The output features of the output feature processing module are added to the output features of the first downsampling block of the target diffusion model, and then input into the second downsampling block. The output feature processing module processes the fused features finally output by the multi-scale fusion module according to the following formula. Processing:

[0097] ;

[0098] ;

[0099] In the formula:

[0100] η represents the global scaling parameter;

[0101] This represents the fusion feature output by the multi-scale fusion module.

[0102] This represents the intermediate features generated by the output feature processing module;

[0103] This represents the final output features of the image processing system;

[0104] Represents the group normalization function;

[0105] This represents a 3×3 convolution function;

[0106] This represents a 1×1 convolution function.

[0107] This represents the SiLU activation function.

[0108] The present invention also provides an apparatus for a lightweight multi-task context-controlled image processing method, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the lightweight multi-task context-controlled image processing method steps as described above.

[0109] The advantages and positive effects of this invention are:

[0110] (1) By using a dual-path structure and a sliding window cross-attention mechanism, it is possible to achieve context learning and control for multiple tasks while maintaining the model's lightweight nature.

[0111] (2) Using example images as input, the model can explicitly utilize contextual example information, supporting multiple tasks such as semantic segmentation, depth estimation, and edge detection;

[0112] (3) Constructing forward and backward task examples improves the richness of control signals and the generalization ability of the model;

[0113] (4) The model parameters are properly optimized, the computational cost is low, and it is suitable for diffusion model inference in resource-constrained environments.

[0114] This invention provides an effective means to improve the generation of higher-quality images for multi-task vision basic models based on diffusion models, and provides strong support for their better application in real-world scenarios.

[0115] The technical fields of application of this invention include:

[0116] This invention focuses on lightweight, multi-tasking image processing capabilities that leverage contextual information from examples. It is suitable for real-world industrial scenarios requiring simultaneous generation and understanding of visual tasks, and places high demands on model size, computational resources, and generalization ability. Its application areas include, but are not limited to:

[0117] Inspection and maintenance in the power and energy industry: Automated semantic segmentation, defect detection, depth estimation and visualization reconstruction of drone / fixed camera images for transmission lines, substations and wind / solar power plants, assisting in fault location, risk assessment and maintenance decision-making, and supporting resource-constrained edge deployment.

[0118] Transportation and Intelligent Mobility: Used for road / bridge / tunnel monitoring, traffic flow analysis, vehicle and pedestrian detection, depth perception and semantic understanding, as well as multi-task perception (segmentation, depth, edge) and scene completion in autonomous driving perception modules.

[0119] UAV aerial surveying and remote sensing: used for semantic segmentation, terrain depth estimation, feature extraction and image enhancement of aerial and satellite imagery, supporting applications such as agricultural yield estimation, land use monitoring, disaster assessment and urban planning.

[0120] Intelligent security and video surveillance: used for human / object segmentation, contour extraction, etc., to improve the recognition rate and interpretability of monitoring systems for abnormal events.

[0121] Medical image post-processing (auxiliary applications): used for segmentation and contour extraction of medical images, providing assistance for remote diagnosis and image visualization; it can be used in conjunction with compliance assessment and professional verification in clinical applications.

[0122] Cultural Heritage Conservation and Digital Archiving: Used for detailed restoration, semantic segmentation, and high-quality reconstruction of images of cultural relics / ancient buildings, assisting in digital restoration and long-term preservation.

[0123] Film and television special effects and media production: Used for scene segmentation, background replacement, detail enhancement and image stylization, improving post-production efficiency and reducing the amount of manual background removal work.

[0124] Agricultural intelligent sensing: used for pest and disease detection, crop segmentation, canopy depth estimation and precision fertilization / spraying decision support, and adapted to image data collected by drones and ground sensors.

[0125] Urban digital twins and infrastructure monitoring: used for automatic extraction, in-depth modeling and fault early warning of elements such as roads, pipelines and green spaces, supporting operation and maintenance and planning decisions.

[0126] Robot Vision and Automation: Enables industrial / service robots to simultaneously perform multi-task perception capabilities such as segmentation, depth estimation, and boundary detection in grasping, path planning, and environmental understanding, thereby improving robustness and versatility. Attached Figure Description

[0127] Figure 1 This is a structural diagram of a lightweight, multi-task context-controlled image processing system.

[0128] Figure 2 This is a structural diagram of a multi-scale fusion unit.

[0129] Figure 3 This is a schematic diagram illustrating the application of a lightweight multi-task context control image processing method according to the present invention.

[0130] In the picture:

[0131] Indicates the query sample image;

[0132] Indicates a control instance pair;

[0133] x represents a sample feature;

[0134] c represents the control input feature;

[0135] This represents the query sample features output by the residual convolution module;

[0136] This represents the task control features output by the residual convolution module;

[0137] This indicates the output characteristics of the refining module;

[0138] This represents the fusion feature output by the multi-scale fusion module.

[0139] This represents the final output features of the image processing system;

[0140] This represents the time step embedding vector. Detailed Implementation

[0141] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0142] The terms “connected” and “linked” used in this invention should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; they can refer to direct connections or indirect connections through intermediate components; they can also refer to electrical connections or signal transmissions. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0143] The Chinese definitions of the following English words, phrases, and abbreviations in this invention are as follows:

[0144] Stable Diffusion: A name for an AI image generation model, whose core is to generate images step by step from random noise using a diffusion model.

[0145] ControlNet: A network structure used to introduce additional conditional signals into a diffusion model to guide the model in generating images that meet specific task requirements.

[0146] ControlLiteNet: A lightweight image processing system with multi-task context control.

[0147] VAE: Variational Autoencoder; a generative model structure that achieves image compression and reconstruction by learning a latent distribution.

[0148] ResBlock: Residual convolutional block; a convolutional structure composed of residual connections, used to alleviate the gradient vanishing problem in deep network training and improve feature representation capabilities.

[0149] Window Unpatch: Inverse window transformation; used to reassemble local features after window partitioning and attention calculation into a global feature map.

[0150] FiLM: Feature Linear Modulation; a modulation method that performs affine transformations on features using scaling and offset parameters, used to implement conditional control in visual models.

[0151] GroupNorm: Group normalization; a normalization method that divides feature channels into groups and normalizes them within each group to stabilize training and reduce batch dependencies.

[0152] ReLU: Modified Linear Unit Activation Function; used to introduce nonlinearity and improve the expressive power of neural networks.

[0153] MLP: Multilayer Perceptron; a fully connected neural network structure consisting of several linear layers and nonlinear activation functions.

[0154] Sigmoid: A sigmoid activation function that maps input values ​​to the (0,1) interval. It is often used to generate outputs that require normalization, such as weights or probabilities.

[0155] Please see Figures 1 to 3 A lightweight multi-task context control image processing system includes a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected sequentially from high to low spatial resolution.

[0156] The first image encoder is used for feature extraction and downsampling of images from the sample path; its input is the query sample image, and its output is the query sample input features. .

[0157] The second image encoder is used to extract features and downsample control example image pairs from the control path; its input consists of control example image pairs stitched together from example source images and target images, and its output is control input features. .

[0158] The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module.

[0159] The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal embedding information; its inputs are query sample features and task control features, and its output is query sample features fused with temporal information. and task control features .

[0160] The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. .

[0161] The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. .

[0162] The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. .

[0163] The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; it takes as input the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution downscaling are used as the input features for the query samples of the next multi-scale fusion unit; these features control the task. The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit.

[0164] The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

[0165] Preferably, the depthwise separable convolutional module can extract spatial features according to the following formula. ;

[0166] ;

[0167] The compression and excitation modules can be derived from spatial features using the following formula. Generate spatial features :

[0168] ;

[0169] ;

[0170] In the formula:

[0171] This represents the spatial features extracted by the depthwise separable convolutional module;

[0172] This represents the spatial features generated by the compression and excitation modules;

[0173] This represents the modulation fusion feature output by the characteristic linear modulation module;

[0174] This represents the channel attention weight vector generated by the compression and excitation module;

[0175] This represents the first transformation matrix in the compression and excitation module;

[0176] This represents the second transformation matrix in the compression and excitation module.

[0177] This represents a depthwise separable convolution function;

[0178] This represents a pointwise convolution operation;

[0179] This represents the Sigmoid activation function;

[0180] This represents the global average pooling function;

[0181] Represents the ReLU activation function;

[0182] This represents a shape reshaping function;

[0183] This represents channel-by-channel multiplication;

[0184] Preferably, the sliding window cross-attention module can generate Q, K, and V tensors according to the following formula:

[0185] ;

[0186] ;

[0187] ;

[0188] In the formula:

[0189] Represents a query tensor;

[0190] Represents the key tensor;

[0191] Represents a value tensor;

[0192] This represents the query sample features output by the residual convolution module;

[0193] This represents the task control features output by the residual convolution module;

[0194] Represents the learnable query projection matrix;

[0195] Represents the learnable bond projection matrix;

[0196] The projection matrix represents the learnable values;

[0197] This represents the group normalization function.

[0198] Preferably, during the window partitioning process, the sliding window cross-attention module can alternate between standard window partitioning and partitioning methods that shift by half a window size along the spatial dimension among sequentially connected multi-scale fusion units, allowing attention calculation to capture local contextual information within the window at each scale; within each window, the cross-attention module can calculate cross-attention according to the following formula:

[0199] ;

[0200] In the formula:

[0201] The dimension representing the projected feature;

[0202] This indicates the query characteristics within a single window;

[0203] Represents key features within a single window;

[0204] This represents the characteristics of values ​​within a single window;

[0205] This represents the attention calculation function;

[0206] This represents the softmax function;

[0207] The sliding window cross-attention module generates attention fusion features according to the following formula:

[0208] ;

[0209] In the formula:

[0210] This represents the query sample features output by the residual convolution module;

[0211] This represents the attention fusion features generated by the sliding window cross-attention module;

[0212] Indicates the learnable scaling factor;

[0213] This represents the learnable output projection matrix;

[0214] This represents the inverse window transform function.

[0215] Preferably, the characteristic linear modulation module can generate modulation parameters using a multilayer perceptron according to the following formula:

[0216] ;

[0217] In the formula:

[0218] Indicates the scaling parameter;

[0219] Indicates the offset parameter;

[0220] Represents the time step embedding vector;

[0221] This represents the task control features output by the residual convolution module;

[0222] This represents the global average pooling function;

[0223] Indicates the concatenation function;

[0224] Represents the function of a multilayer perceptron;

[0225] The characteristic linear modulation module can generate modulation fusion features according to the following formula:

[0226] ;

[0227] In the formula:

[0228] This represents the fusion features generated by the modulation fusion module;

[0229] This represents the attention fusion features generated by the sliding window cross-attention module;

[0230] Represents the group normalization function;

[0231] This indicates channel-by-channel multiplication.

[0232] The present invention also provides a lightweight multi-task context control image processing method, which constructs an image processing system. The image processing system is configured with a first image encoder, a second image encoder, a multi-scale fusion module and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution.

[0233] The first image encoder is used for feature extraction and downsampling of images from the sample path; its input is the query sample image, and its output is the query sample input features. ;

[0234] The second image encoder is used to extract features and downsample control example image pairs from the control path; its input consists of control example image pairs stitched together from example source images and target images, and its output is control input features. ;

[0235] The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module;

[0236] The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal embedding information; its inputs are query sample features and task control features, and its output is query sample features fused with temporal information. and task control features ;

[0237] The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ;

[0238] The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ;

[0239] The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ;

[0240] The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; it takes as input the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution downscaling are used as the input features for the query samples of the next multi-scale fusion unit; these features control the task. The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit.

[0241] The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

[0242] Preferably, the time step can be mapped to a high-dimensional time representation through a standard temporal projection layer or a sin / cos-based temporal encoder, and then the time embedding vector can be obtained through linear transformation or MLP. The vector is injected into the residual convolution module and the feature linear modulation module.

[0243] Preferably, when training the image processing system, the training data can be multi-task control example image pairs and query images, and the training loss function can be noise prediction loss; data augmentation and regularization strategies are selected to improve the robustness of the model.

[0244] Multitasking is divided into two main categories: forward tasks and reverse tasks. Forward tasks include generating semantic segmentation maps from images, generating depth maps from images, and generating contour maps from images. Reverse tasks include generating images from semantic segmentation maps, generating images from depth maps, and generating images from contour maps. By explicitly providing task context information in the form of controlled example image pairs, a single system can support multiple tasks.

[0245] The source and target images can be: image → segmentation map, i.e., generating a semantic segmentation map from an image, corresponding to the semantic segmentation task; image → depth map, i.e., generating a depth map from an image, corresponding to the depth estimation task; image → contour map, i.e., generating a contour map from an image, corresponding to the edge detection task; segmentation map → image, i.e., generating an image from a semantic segmentation map, corresponding to the inverse task from segmentation to a normal image; depth map → image, i.e., generating an image from a depth map, corresponding to the inverse task from depth to a normal image; contour map → image, i.e., generating an image from a contour map, corresponding to the inverse task from contour to a normal image.

[0246] Preferably, the output feature processing module can be connected to the target diffusion model, which has multiple downsampling blocks; the first to the mth downsampling blocks can be respectively referred to as downsampling module A1, downsampling module A2, ..., downsampling module Am, where m is the number of downsampling modules in the target diffusion model.

[0247] The output features of the output feature processing module can be added to the output features of the first downsampling block of the target diffusion model, and then input into the second downsampling block; the output feature processing module can process the fused features output by the multi-scale fusion module according to the following formula. Processing:

[0248] ;

[0249] ;

[0250] In the formula:

[0251] η represents the global scaling parameter;

[0252] This represents the fusion feature output by the multi-scale fusion module.

[0253] This represents the intermediate features generated by the output feature processing module;

[0254] This represents the final output features of the image processing system;

[0255] Represents the group normalization function;

[0256] This represents a 3×3 convolution function;

[0257] This represents a 1×1 convolution function.

[0258] This represents the SiLU activation function.

[0259] Both the downsampling block and the downsampling module are components that reduce the size of the feature map through pooling or convolution operations, in order to reduce computation, increase the receptive field, and retain key information.

[0260] This invention discloses an apparatus for a lightweight multi-task context control image processing method, comprising a memory and a processor. The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the lightweight multi-task context control image processing method as described above.

[0261] The downsampling module of the first multi-scale fusion unit outputs the following features after processing the features from the refinement module: The downsampling module of the first multi-scale fusion unit outputs the following features after processing the task control features: ;

[0262] Let the feature output by the downsampling module of the k-th multi-scale fusion unit after processing the output feature of the refining module be: The downsampling module of the k-th multi-scale fusion unit outputs the following features after processing the task control features: ; k is the multi-scale fusion unit number, k=1,2,…,N-1;

[0263] The query sample features are used as input to the residual convolution module of the (k+1)th multi-scale fusion unit;

[0264] The task control features are used as input to the residual convolution module of the (k+1)th multi-scale fusion unit;

[0265] The residual convolution module of the first multi-scale fusion unit receives query sample features from the first image encoder in parallel. and task control features from the second image encoder The downsampling module output of the first multi-scale fusion unit and ;

[0266] The residual convolutional module of the (k+1)th multi-scale fusion unit receives parallel input from the kth multi-scale fusion unit. and k=1,3,…,N-1; the downsampling module output of the (k+1)th multi-scale fusion unit. and .

[0267] The first and second image encoders may include a shallow coding module, which performs preliminary feature extraction and downsampling to obtain a feature map with an initial number of channels, and reduces the resolution of the feature map to half of the original.

[0268] The workflow and working principle of the present invention are further illustrated below using a preferred embodiment as an example:

[0269] To address the shortcomings of existing technologies, such as large control network parameter scale, task-specific nature, and lack of multi-task collaborative capabilities, this invention proposes a lightweight multi-task context control image processing system (ControlLiteNet), which can efficiently provide multi-task control signals for diffusion models and generate target images for corresponding tasks.

[0270] A lightweight multi-task context control image processing system, hereinafter referred to as ControlLiteNet, includes a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution.

[0271] The first image encoder is used for feature extraction and downsampling of images from the sample path; its input is the query sample image, and its output is the query sample input features. ;

[0272] The second image encoder is used to extract features and downsample control example image pairs from the control path; its input consists of control example image pairs stitched together from example source images and target images, and its output is control input features. ;

[0273] Multi-scale fusion units are used to fuse query sample features and task control features at different spatial resolutions; N multi-scale fusion units connected sequentially by the downsampling module in descending order of spatial resolution are called the first to the Nth multi-scale fusion units.

[0274] Each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module;

[0275] The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal embedding information; its inputs are query sample features and task control features, and its output is query sample features fused with temporal information. and task control features .

[0276] The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ;

[0277] The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ;

[0278] The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ;

[0279] The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; it takes as input the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution downscaling are used as the input features for the query samples of the next multi-scale fusion unit; these features control the task. The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit.

[0280] The downsampling module may include a first downsampling module and a second downsampling module. The first downsampling module takes as input the output features from the refining module. Its output characteristics for the refining module The features output after spatial resolution downsampling are used as the input features for the query samples of the next multi-scale fusion unit. The second downsampling module inputs the task control features. The output reduces the spatial resolution of the task control features, which then serve as the control input features for the next multi-scale fusion unit.

[0281] The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

[0282] The image processing system adopts a dual-channel encoding structure, including a first image encoder and a second image encoder.

[0283] The dual-path encoding structure consists of two input paths: one is the query sample image (a three-channel RGB image), and the other is a control example image pair, which is a 6-channel image formed by concatenating the example source image and the example target image along the channel dimension. The former provides the input basis for generation, while the latter provides contextual task information.

[0284] Multi-scale fusion module: To achieve efficient fusion of sample features and task control features at different spatial resolutions, ControlLiteNet constructs a multi-scale fusion module. At each scale, the multi-scale fusion module first extracts features from the query sample path and control example path through a residual convolution module, and then fuses the time-step embedding information. Subsequently, the output of the residual convolution module is group normalized and Q (Query), K (Key), and V (Value) tensors are generated through linear projection. The tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to achieve local alignment between sample path features and control path features. During window partitioning, to enhance the interaction of window edge features, an alternating translation mechanism is used. That is, standard window partitioning and partitioning by translating half a window size along the spatial dimension are alternately used between multi-scale fusion modules at continuous scales, so that attention computation can capture richer local context information within the same scale. Within each window, cross-attention is calculated to inject the task context information of the control path into the sample path features, thereby achieving local feature alignment and information fusion. Subsequently, the sample features are further modulated using the global semantic information of the control features through a feature linear modulation module, achieving a full fusion of sample and task information. Next, a refinement module is introduced, consisting of a depthwise separable convolution module and a compression and activation module: the former extracts fine-grained spatial features with minimal parameter overhead, while the latter strengthens key features and suppresses irrelevant features through channel weighting, thereby improving the expressive power of the fused features. Finally, a downsampling module reduces the feature space resolution, providing input for the next scale of fusion. This multi-scale fusion module is compact and computationally efficient, enhancing feature expressive power while maintaining a lightweight model.

[0285] Temporal embedding and implicit task modeling: ControlLiteNet also introduces temporal embedding information from the diffusion process during fusion, making it better adaptable to features in different denoising stages. Unlike traditional methods that require manually specifying the task direction, this invention implicitly learns the task direction by controlling the interaction between global features and temporal embeddings of example image pairs, enhancing the model's ability to recognize different tasks (forward and backward tasks) without requiring manual setting of the task type.

[0286] Multi-task example pair construction: This invention constructs six task scenarios, divided into two main categories: forward tasks and reverse tasks. Forward tasks include "image to semantic segmentation map," "image to depth map," and "image to contour map," while reverse tasks are the corresponding inverse tasks, namely "semantic segmentation map to image," "depth map to image," and "contour map to image." The input of the control example image pair is formed by concatenating the source image and the target image. For example, when the source image is a regular RGB image and the target image is a segmentation map, the corresponding task is a forward segmentation task. In this case, the output control signal of ControlLiteNet guides the diffusion model to perform segmentation operations on the query sample image, generating the corresponding segmentation map of the query sample image. Conversely, if the source image is a segmentation map and the target image is a regular image, it corresponds to the reverse task of the segmentation task, where the output control signal guides the diffusion model to generate a normal image from the conditional image. Through this example pair format, this invention can effectively utilize the contextual information in the control path, enabling a single model to support multiple visual tasks simultaneously.

[0287] Output Feature Integration with Model: ControlLiteNet ultimately outputs a control feature map (e.g., 320 channels) that matches the target diffusion model, and provides learnable global scaling factors to flexibly adjust the strength of the control signal. The target diffusion model has multiple downsampling modules. This feature map is added to the output feature map of the first downsampling module of the target diffusion model, and then input into the second downsampling module to guide the target diffusion model to generate an image that corresponds to the query sample image and is consistent with the example image pair task.

[0288] To achieve the above objectives, this invention constructs a lightweight multi-task context-controlled image processing method, which employs the following steps:

[0289] 1. ControlLiteNet adopts a dual-path encoding structure, which includes two input encoders: a sample path encoder and a control path encoder, which process query sample image pairs and control example image pairs, respectively.

[0290] 2. Construct a multi-scale fusion module to achieve efficient fusion of sample features and control features at each scale through a sliding window attention mechanism and feature linear modulation. Then, introduce a refinement module and use a depthwise separable convolution module and a compression and activation module to further extract and enhance feature representation. Finally, reduce the feature space resolution through a downsampling module to provide input for the next scale fusion.

[0291] 3. Integrates time step embedded information, implicitly learns task direction, and supports multiple forward and backward visual tasks.

[0292] 4. By constructing multi-task example pairs, a single network can cover a variety of visual task scenarios.

[0293] 5. Output a control feature map that matches the downstream diffusion model and provide a learnable global scaling factor to adjust the control signal strength.

[0294] 6. A lightweight multi-task context-controlled image processing system and method are used to perform multi-task image processing.

[0295] The construction method of the dual-path coding structure includes the following steps:

[0296] The input path is a three-channel RGB query sample image. The control path input is a six-channel control example image pair. From the example source image and target image The images are concatenated along the channel dimension. Each of the two inputs undergoes preliminary feature extraction and downsampling via a shallow encoding module, resulting in a feature map with an initial number of channels (e.g., 128), and the image resolution is reduced to half of the original. Shallow encoding is implemented through three convolutional layers (two resolution-preserving convolutions and one downsampling convolution with a stride of 2), with GroupNorm and ReLU activation functions applied after each layer. The operation of each convolutional layer can be represented as follows:

[0297] ;

[0298] in, For convolution kernel weights, Divided into convolution kernel bias, The asterisk (*) represents the group normalization function; the asterisk (*) represents the convolution operation; and U represents the input of the current layer. This represents the output of the current layer. The last convolution in the shallow encoding layer uses a stride of 2 to achieve downsampling.

[0299] The method for constructing multi-scale fusion units includes the following steps:

[0300] To achieve efficient fusion of sample features x and control features c at different spatial resolutions, for each scale supported by the network, the sample path features x and control path features c are respectively passed through a residual convolutional module (ResBlock), and the time steps are embedded. Injected into this block to generate features that incorporate temporal information:

[0301] ;

[0302] ;

[0303] This represents the query sample features output by the residual convolution module;

[0304] This represents the task control features output by the residual convolution module;

[0305] Indicates to Perform residual convolution calculation.

[0306] Indicates to and Perform residual convolution calculation.

[0307] The output of the residual convolution module is then normalized and linearly projected to obtain the Q(Query), K(Key), and V(Value) tensors:

[0308] ;

[0309] ;

[0310] ;

[0311] In the formula:

[0312] Represents the learnable query projection matrix;

[0313] Represents the learnable bond projection matrix;

[0314] The projection matrix represents the learnable values;

[0315] This represents the group normalization function.

[0316] Subsequently, the sliding window cross-attention module divides the Q, K, and V tensors into multiple non-overlapping local windows and uses a sliding window cross-attention mechanism to align local features. During window partitioning, to enhance the interaction of window edge features, an alternating translation mechanism is employed. This involves alternating between standard window partitioning and partitioning that translates by half a window size along the spatial dimension between fusion modules at continuous scales, allowing attention computation to capture richer local contextual information within each window at every scale. Within each window, cross-attention is calculated:

[0317] ;

[0318] In the formula:

[0319] The dimension representing the projected feature;

[0320] This indicates the query characteristics within a single window;

[0321] Represents key features within a single window;

[0322] This represents the characteristics of values ​​within a single window;

[0323] This represents the attention calculation function;

[0324] This represents the softmax function;

[0325] After the window attention calculation is completed, the output features of each window are recombined into a global feature map through the window unpatch operation, and then passed through a linear projection layer. Perform feature transformation. The transformed features are then scaled by a learnable scaling parameter. Perform weighting and connect it to the original through residuals. When added together, preliminary fusion characteristics are formed:

[0326] ;

[0327] In the formula:

[0328] This represents the query sample features output by the residual convolution module;

[0329] This represents the attention fusion features generated by the sliding window cross-attention module;

[0330] Indicates the learnable scaling factor;

[0331] This represents the learnable output projection matrix;

[0332] This represents the inverse window transform function.

[0333] The feature linear modulation module outputs the control features from the residual convolution module. Perform global average pooling to obtain its global semantic vector, and then embed this vector with the time step. After stitching, modulation parameters—scaling parameters—are generated using a small multilayer perceptron (MLP). With offset parameter :

[0334] ;

[0335] In the formula:

[0336] Indicates the scaling parameter;

[0337] Indicates the offset parameter;

[0338] Represents the time step embedding vector;

[0339] This represents the task control features output by the residual convolution module;

[0340] This represents the global average pooling function;

[0341] Indicates the concatenation function;

[0342] Represents the function of a multilayer perceptron;

[0343] Then, the attention output features are processed. Perform group normalization (GN), then use the generated and Perform an affine transformation on it:

[0344] ;

[0345] In the formula:

[0346] This represents the fusion features generated by the modulation fusion module;

[0347] This represents the attention fusion features generated by the sliding window cross-attention module;

[0348] Represents the group normalization function;

[0349] This indicates channel-by-channel multiplication.

[0350] This feature linear modulation injects the global semantics and temporal information of the control path into the sample features, thereby achieving a full fusion of sample features and task semantics.

[0351] To enhance the expressive power of the features modulated by the feature linear modulation module, a refinement module is introduced. The refinement module consists of a depthwise separable convolution module and a compression and activation module. The depthwise separable convolution module extracts fine-grained spatial information with minimal parameter cost by combining depthwise convolution and pointwise convolution. The compression and activation module strengthens key channel features through channel-level attention.

[0352] The computation of depthwise separable convolutional modules and compression and activation modules can be represented by a simplified expression as follows:

[0353] ;

[0354] ;

[0355] ;

[0356] In the formula:

[0357] This represents the spatial features extracted by the depthwise separable convolutional module;

[0358] This represents the spatial features generated by the compression and excitation modules;

[0359] This represents the modulation fusion feature output by the characteristic linear modulation module;

[0360] This represents the channel attention weight vector generated by the compression and excitation module;

[0361] This represents the first transformation matrix in the compression and excitation module;

[0362] This represents the second transformation matrix in the compression and excitation module.

[0363] This represents a depthwise separable convolution function;

[0364] This represents a pointwise convolution operation;

[0365] This represents the Sigmoid activation function;

[0366] This represents the global average pooling function;

[0367] Represents the ReLU activation function;

[0368] This represents a shape reshaping function;

[0369] This represents channel-by-channel multiplication;

[0370] The refined features are summed with the residuals from the feature linear adjustment module output to obtain the final output at this scale:

[0371] ;

[0372] In the formula:

[0373] This indicates the output characteristics of the refining module;

[0374] Finally, the output features will be... The spatial resolution is reduced through a downsampling module, serving as the input features for query samples at the next scale. (Control path features) It also performs synchronous downsampling as input for the next scale.

[0375] The time-step embedding and task implicit modeling method includes the following steps:

[0376] Integrating the time-step embedding of the diffusion process into the multi-scale fusion unit enables ControlLiteNet to adaptively process different denoising stages. The time step is mapped to a high-dimensional time representation through a standard temporal projection layer or a sin / cos-based temporal encoder, and then the time embedding vector is obtained through linear transformation or MLP. This vector is injected into the MLP of the residual convolution module and the feature linear modulation module. By jointly using the global semantics and temporal embeddings of the control example pairs as input to the feature linear modulation module MLP, the model can implicitly learn the representation of the task direction (forward task / backward task) during training, thus enabling it to support multiple visual tasks in the same network without manually specifying the task type in advance. The temporal step processing should include tensorization and batch broadcasting of the scalar temporal steps to match the batch dimension of the multi-task processing model and ensure consistent numerical types.

[0377] The multi-tasking example construction method includes the following steps:

[0378] It supports six visual tasks, divided into two main categories: forward tasks and reverse tasks. Forward tasks include: image → semantic segmentation map, image → depth map, and image → contour map; reverse tasks include: semantic segmentation map → image, depth map → image, and contour map → image. Task context information is explicitly provided through controlled example pairs, enabling a single model to support multiple tasks.

[0379] The method for integrating output features with the model includes the following steps:

[0380] After the highest-scale processing is completed, the multi-scale fusion unit outputs features. Several convolutions and activations are performed to increase nonlinear representation and normalization. Finally, a 1×1 convolution maps the number of channels to the required number of channels (e.g., 320 channels) for the downstream diffusion model, serving as the final control signal feature map. Simultaneously, a global scaling parameter is learned to control the intensity of the feature map when it is incorporated into the diffusion model. This feature map can be directly injected into the target diffusion model (e.g., the outputs of the first downsampling module of Stable Diffusion are summed and then input into the second downsampling module), guiding it to generate an image that corresponds to the query sample image and is consistent with the example image pair task. The output projection and scaling can be represented as:

[0381] ;

[0382] ;

[0383] In the formula:

[0384] η represents the global scaling parameter;

[0385] This represents the fusion feature output by the multi-scale fusion module.

[0386] This represents the intermediate features generated by the output feature processing module;

[0387] This represents the final output features of the image processing system;

[0388] Represents the group normalization function;

[0389] This represents a 3×3 convolution function;

[0390] This represents a 1×1 convolution function.

[0391] This represents the SiLU activation function.

[0392] Downstream diffusion models can By adding its own features at a specified location or injecting them through other fusion methods, the diffusion process is guided to generate a target image that corresponds to the query sample image and is consistent with the example image pairing task.

[0393] An application scheme for a lightweight multi-task context control image processing system includes:

[0394] Integrating ControlLiteNet into the diffusion model architecture (e.g., as a new control branch added to Stable Diffusion) creates a new model architecture. During training, most of the pre-trained weights of the original diffusion model can be frozen, leaving only the attention output layer trainable, while the newly added ControlLiteNet is trained simultaneously. The training data includes multi-task example pairs and query images. The training loss function uses the original noise prediction loss of the diffusion model to guide the entire model to learn the correct task mapping relationship. After training, the entire model can generate image outputs that meet the requirements of the example task by taking any query image and corresponding task example pair as input during the inference phase. Data augmentation and regularization strategies can be used to improve the model's robustness.

[0395] The present invention will now be described in further detail with reference to the accompanying drawings and specific examples, but this should not be construed as limiting the invention.

[0396] The ControlLiteNet used in this embodiment can be applied to any diffusion model (such as Stable Diffusion) that employs a control network architecture, and is used to accept query images and control example pairs and generate target images.

[0397] Step 1: Model Skeleton Selection and Input Definition. Select an image generation framework based on a diffusion model as the skeleton (e.g., Stable Diffusion). Add a control branch outside this diffusion model and deploy the ControlLiteNet of this invention. During training or inference, the input includes: a three-channel RGB query sample image. and control examples ,in For example source images, For the target image, the control example pair is a 6-channel image obtained by concatenating a pair of example images along the channel dimension. The example pair consists of a "source image" and a "target image," covering various task combinations. For example, the source and target images can be: image → segmentation map, corresponding to semantic segmentation tasks; image → depth map, corresponding to depth estimation tasks; image → contour map, corresponding to edge detection tasks; segmentation map → image, corresponding to the inverse task from segmentation to a normal image; depth map → image, corresponding to the inverse task from depth to a normal image; contour map → image, corresponding to the inverse task from contour to a normal image. Through this construction, the control example pair can explicitly provide task context to the model. For example, when the control example pair represents an "image to segmentation map" example, it indicates that the current task is a segmentation task, and the output of ControlLiteNet will guide the diffusion model to segment and generate the query sample image.

[0398] Step 2: ControlLiteNet Architecture Construction. ControlLiteNet includes the following main components:

[0399] Time-step embedding block: Used to first map the time steps of the diffusion process to a high-dimensional time representation using a time encoder based on sin / cos, and then obtain the time embedding vector through linear transformation or MLP. This vector is injected into the MLP of the residual convolution module and the feature linear modulation module, enabling ControlLiteNet to be aware of different denoising stages.

[0400] First and Second Image Encoders: Both the first and second image encoders employ shallow encoding modules. The sample path and control path inputs each undergo preliminary feature extraction and downsampling through the shallow encoding module to obtain feature maps with an initial number of channels (e.g., 128), reducing the image resolution to half of the original. Shallow encoding is implemented through three convolutional layers (two resolution-preserving convolutions and one downsampling convolution with a stride of 2), with GroupNorm and ReLU activation functions applied after each layer.

[0401] To achieve efficient fusion of sample features x and control features c at different spatial resolutions, for each scale supported by the network, the sample path features x and control path features c are respectively passed through a residual convolutional module (ResBlock), and the time steps are embedded. Injected into this block to generate features that incorporate temporal information:

[0402] Subsequently, the sliding cross-attention module performs group normalization and linear projection on the output of the residual convolution module to obtain the Q(Query), K(Key), and V(Value) tensors:

[0403] The Q, K, and V tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. During window partitioning, to enhance the interaction of window edge features, an alternating translation mechanism is employed. This involves alternating between standard window partitioning and partitioning that translates by half a window size along the spatial dimension between fusion modules at continuous scales, allowing attention computation to capture richer local contextual information within each window at every scale. Cross-attention is then calculated within each window.

[0404] After the window attention calculation is completed, the output features of each window are recombined into a global feature map through the window unpatch operation, and then passed through a linear projection layer. Perform feature transformation. The transformed features are then scaled by a learnable scaling parameter. Perform weighting and connect it to the original through residuals. Add them together to generate attention fusion features. .

[0405] Subsequently, the feature linear modulation module converts the control features output by the residual convolution module. Perform global average pooling to obtain its global semantic vector, and then embed this vector with the time step. After stitching, modulation parameters—scaling parameters—are generated using a small multilayer perceptron (MLP). With offset parameter .

[0406] Then, attention fusion features are analyzed. Perform group normalization (GN), then use the generated and Perform an affine transformation on it.

[0407] This linear modulation of features injects global semantics and temporal information of the control path into the sample features, thereby achieving a full fusion of sample features and task semantics. For the linearly modulated features, a refinement module is introduced to enhance expressive power. The refinement module consists of a depthwise separable convolution module and a compression and activation module. The depthwise separable convolution module extracts fine-grained spatial information with minimal parameter cost by combining depthwise convolution and pointwise convolution, while the compression and activation module strengthens key channel features through channel-level attention.

[0408] The refined features are added to the residuals of the features output by the feature linear adjustment module to obtain the final output at this scale. .

[0409] Downsampling module: outputs features A downsampling convolutional layer with a stride of 2 is used to reduce its spatial resolution, serving as the input feature for the query sample at the next scale. Control path features. It also performs synchronous downsampling as input for the next scale.

[0410] In summary, ControlLiteNet highly integrates information from query images and control example pairs through multi-scale dual-path processing and fusion modules, thereby outputting matching control features for the diffusion model.

[0411] Step 4: Model Integration and Training. Integrate ControlLiteNet into the architecture of the diffusion model (e.g., as a new control branch added to Stable Diffusion). During training, most of the pre-trained weights of the diffusion model can be frozen, leaving only the attention output layer trainable, while simultaneously training ControlLiteNet to maximize the retention of the original model's generative capabilities. Training data must include multi-task example pairs and query images. The loss function uses the diffusion model's original noise prediction loss to guide the entire model in learning the correct task mapping relationships. After training, the entire model can generate image outputs that meet the requirements of the example tasks by taking any query image and corresponding task example pair as input during the inference phase.

[0412] Through the above steps, this invention constructs a lightweight, efficient, and multi-task-capable lightweight multi-task context-controlled image processing system. Its construction has the following characteristics and advantages: Lightweight construction: Employing lightweight modules such as depthwise separable convolution and group normalization, the number of parameters is significantly reduced compared to traditional control networks; Multi-task versatility: Explicitly guiding multiple task directions through example pairs as input avoids redundancy from multiple independent models; Context awareness: Example pairs provide task context information, enabling the model to refer to specific task examples when executing new queries, enhancing the accuracy of generated results; Strong scalability: The structure can be expanded to support more tasks as needed, requiring only the construction of corresponding example pairs without reconstructing the main network; Easy integration: The output feature channels are compatible with downstream diffusion models and can be plugged into existing diffusion models without extensive modifications to the original model architecture.

[0413] The aforementioned first image encoder, second image encoder, output feature processing module, residual convolution module, sliding window cross attention module, feature linear modulation module, refinement module and downsampling module, depthwise separable convolution module, compression and excitation module, diffusion model and other functional modules can all sample applicable functional modules in the prior art, or adopt functional modules in the prior art and construct them using conventional technical means.

[0414] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments described above, and any obvious improvements, substitutions, or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A lightweight multi-task context control image processing system, characterized in that, It includes a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution. The first image encoder is used to extract features and downsample the image from the sample path; Its input is a query sample image, and its output is the query sample input features. ; The second image encoder is used to extract features and downsample control example image pairs from the control path; Its input consists of a control example image pair, which is a combination of an example source image and a target image, and its output is the control input feature. ; The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module; The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal step embedding information; Its inputs are query sample features and task control features, and its output is query sample features that incorporate time-series information. and task control features ; The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ; The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ; The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ; The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; Their respective inputs are the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution reduction are used as the input features for the query samples of the next multi-scale fusion unit. Its task control features The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit. The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

2. The lightweight multi-task context control image processing system according to claim 1, characterized in that, Depthwise separable convolutional modules extract spatial features using the following formula. ; ； The compression and excitation modules are based on the following formulas from spatial features. Generate spatial features : ； ； In the formula: This represents the spatial features extracted by the depthwise separable convolutional module; This represents the spatial features generated by the compression and excitation modules; This represents the modulation fusion feature output by the characteristic linear modulation module; This represents the channel attention weight vector generated by the compression and excitation module; This represents the first transformation matrix in the compression and excitation module; This represents the second transformation matrix in the compression and excitation module. This represents a depthwise separable convolution function; This represents a pointwise convolution operation; This represents the Sigmoid activation function; This represents the global average pooling function; Represents the ReLU activation function; This represents a shape reshaping function; This indicates channel-by-channel multiplication.

3. The lightweight multi-task context control image processing system according to claim 1, characterized in that, The sliding window cross-attention module generates Q, K, and V tensors according to the following formula: ； ； ； In the formula: Represents a query tensor; Represents the key tensor; Represents a value tensor; This represents the query sample features output by the residual convolution module; This represents the task control features output by the residual convolution module; Represents the learnable query projection matrix; Represents the learnable bond projection matrix; The projection matrix represents the learnable values; This represents the group normalization function.

4. The lightweight multi-task context control image processing system according to claim 1, characterized in that, During the window partitioning process, the sliding window cross-attention module alternates between standard window partitioning and partitioning by shifting half a window size along the spatial dimension between sequentially connected multi-scale fusion units. This allows attention calculation to capture local contextual information within the window at each scale. Within each window, the cross-attention module calculates cross-attention according to the following formula: ； In the formula: The dimension representing the projected feature; This indicates the query characteristics within a single window; Represents key features within a single window; This represents the characteristics of values ​​within a single window; This represents the attention calculation function; This represents the softmax function; The sliding window cross-attention module generates attention fusion features according to the following formula: ； In the formula: This represents the query sample features output by the residual convolution module; This represents the attention fusion features generated by the sliding window cross-attention module; Indicates the learnable scaling factor; This represents the learnable output projection matrix; This represents the inverse window transform function.

5. The lightweight multi-task context control image processing system according to claim 1, characterized in that, The characteristic linear modulation module generates modulation parameters using a multilayer perceptron according to the following formula: ； In the formula: Indicates the scaling parameter; Indicates the offset parameter; Represents the time step embedding vector; This represents the task control features output by the residual convolution module; This represents the global average pooling function; Indicates the concatenation function; Represents the function of a multilayer perceptron; The modulation fusion features are generated according to the following formula: ； In the formula: This represents the fusion features generated by the modulation fusion module; This represents the attention fusion features generated by the sliding window cross-attention module; Represents the group normalization function; This indicates channel-by-channel multiplication.

6. A lightweight multi-task context control image processing method, characterized in that, An image processing system is constructed, which includes a first image encoder, a second image encoder, a multi-scale fusion module, and an output feature processing module. The multi-scale fusion module includes N multi-scale fusion units connected in descending order of spatial resolution. The first image encoder is used to extract features and downsample the image from the sample path; Its input is a query sample image, and its output is the query sample input features. ; The second image encoder is used to extract features and downsample control example image pairs from the control path; Its input consists of a control example image pair, which is a combination of an example source image and a target image, and its output is the control input feature. ; The multi-scale fusion unit is used to fuse query sample features and task control features at different spatial resolutions; each multi-scale fusion unit includes a residual convolution module, a sliding window cross attention module, a feature linear modulation module, a refinement module, and a downsampling module; The residual convolution module is used to fuse query sample features and task control features at different spatial resolutions, and to fuse temporal step embedding information; Its inputs are query sample features and task control features, and its output is query sample features that incorporate time-series information. and task control features ; The sliding window cross-attention module is used to process the features output by the residual convolution module. and The process involves sequentially performing group normalization and linear projection to generate Q, K, and V tensors. These tensors are then divided into multiple non-overlapping local windows, and a sliding window cross-attention mechanism is used to align local features. After window attention calculation, the output features of each window are recombined into a global feature map through inverse window transformation. This global feature map is then linearly projected and scaled for weighting, and finally connected to the query sample features via residual connections. Add them together to generate attention fusion features. ; The feature linear modulation module is used to first convert the task control features output by the residual convolution module. Global average pooling is performed to obtain its global semantic vector. This vector is then concatenated with the time step embedding and passed through a multilayer perceptron to generate the following modulation parameters: scaling parameters. With offset parameter Then, the output features of the sliding window cross-attention module are analyzed. Perform group normalization, then use the generated scaling parameters. With offset parameter After normalization Perform an affine transformation to obtain the modulation fusion features. ; The refining module is used to enhance modulation fusion features. The refining module comprises a depthwise separable convolutional module and a compression and activation module connected sequentially. The depthwise separable convolutional module extracts fine-grained spatial information with minimal parameter cost by combining depthwise separable convolution with pointwise convolution. The compression and activation module enhances key channel features through channel-level attention. The refining module fuses the features processed by the depthwise separable convolutional module and the compression and activation module with the modulation features. The output features are obtained by summing the residuals. ; The downsampling module is used to reduce the spatial resolution of the input features through downsampling convolutional layers; Their respective inputs are the output features from the refining module. and task control features Its output characteristics for the refining module The features output after spatial resolution reduction are used as the input features for the query samples of the next multi-scale fusion unit. Its task control features The features output after spatial resolution reduction are used as the control input features for the next multi-scale fusion unit. The output feature processing module is used to receive the fused features output by the multi-scale fusion module. The system sequentially performs convolution, normalization, and activation processing, and uses 1×1 convolution to unify the number of channels to match the dimension injected by the diffusion model. At the same time, a learnable global scaling parameter is introduced to adjust the intensity of the output features, thereby generating the final output features of the system.

7. The lightweight multi-task context control image processing method according to claim 6, characterized in that, The time step is mapped to a high-dimensional time representation through a standard temporal projection layer or a sin / cos-based temporal encoder, and then the time embedding vector is obtained through linear transformation or MLP. The vector is injected into the residual convolution module and the feature linear modulation module.

8. The lightweight multi-task context control image processing method according to claim 6, characterized in that, When training the image processing system, the training data consists of control example image pairs and query images for multiple tasks, and the training loss function is noise prediction loss. Data augmentation and regularization strategies were employed to improve model robustness; Multitasking is divided into two main categories: forward tasks and reverse tasks; Forward tasks include generating semantic segmentation maps from images, generating depth maps from images, and generating contour maps from images; inverse tasks include generating images from semantic segmentation maps, generating images from depth maps, and generating images from contour maps; by explicitly providing task context information in the form of controlled example image pairs, a single system can support multiple tasks.

9. The lightweight multi-task context control image processing method according to claim 6, characterized in that, The output feature processing module is connected to the target diffusion model, which has multiple downsampling blocks; The output features of the output feature processing module are added to the output features of the first downsampling block of the target diffusion model, and then input into the second downsampling block; The output feature processing module processes the fused features output by the multi-scale fusion module according to the following formula. Processing: ； ； In the formula: η represents the global scaling parameter; This represents the fusion feature output by the multi-scale fusion module. This represents the intermediate features generated by the output feature processing module; This represents the final output features of the image processing system; Represents the group normalization function; This represents a 3×3 convolution function; This represents a 1×1 convolution function; This represents the SiLU activation function.

10. An apparatus for a lightweight multi-task context control image processing method, comprising a memory and a processor, characterized in that, The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the lightweight multi-task context control image processing method as described in any one of claims 6 to 9.

Images