Image shadow removal method and system based on image filling enhancement

Abstract

The present invention provides an image deshadowing method and system based on image filling enhancement, which relates to the field of image processing and includes the following steps: constructing an image restoration model; constructing an image filling task training dataset using random masks based on real images in a deshadowing dataset; training the image restoration model using the image filling task training dataset and saving a model weight file; retraining the trained image restoration model based on the model weight file; optimizing the image restoration model during training; determining and saving a model optimization weight file based on the optimized image restoration model; initializing the image restoration model, and using the initialized image restoration model to perform deshadowing on a shadowed image to be processed, thereby obtaining a deshadowed image. The present invention achieves dynamic task adaptation and significantly improves the model's ability to handle complex degraded scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image shadow removal method and system based on image filling enhancement. Background Art

[0002] In the infrastructure sectors of power inspection and industrial testing, the widespread use of automated equipment such as drones and robots has placed higher demands on image processing technology. For example, transmission line inspections require accurate identification of safety hazards such as damaged insulators and foreign objects on conductors. Substation equipment monitoring also relies on high-definition image analysis to detect signs of partial discharge and corrosion. However, the actual operating environment is complex and dynamic. Occlusion from environmental structures or uneven lighting creates dense shadows. Dynamic lighting changes lead to image contrast imbalances. Inclement weather such as rain, fog, and dust further exacerbates image degradation. These interfering factors not only mask key defect characteristics but also mislead detection algorithms, leading to increased missed detection rates. Traditional image processing methods suffer from low computational efficiency when processing high-resolution images, making them inefficient for real-time inspections and severely restricting the deployment of automated terminals.

[0003] Although image deshadowing technology has made some progress in recent years, existing solutions still have significant limitations. Methods based on physical models (such as illumination separation and reflectance estimation) rely on manually set parameters, have poor robustness in complex scenes, and have difficulty adapting to dynamic environmental changes. Deep learning methods (such as GAN and CNN) improve the efficiency of shadow removal through end-to-end training, but their performance is highly dependent on the quality of labeled data. The power inspection field lacks large-scale professional datasets, and the model's generalization ability is limited. In addition, mainstream technologies focus on single-task optimization and ignore the synergy with multiple tasks such as image restoration and super-resolution. As a result, processing strategies for single tasks can no longer meet the needs of more complex application environments.

[0004] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0005] In view of this, the present invention provides an image shadow removal method and system based on image filling enhancement to solve the above-mentioned problems.

[0006] In order to solve the above problems, the specific technical solutions adopted by the present invention are as follows:

[0007] According to one aspect of the present invention, a method for removing shadows from an image based on image filling enhancement is provided, comprising the following steps:

[0008] S1. Based on the single-task image enhancement model, we build an image restoration model by combining the pre-trained image and text enhancement encoder, text encoder, and task selection module.

[0009] S2. Based on the real images of the shadow removal dataset, use random masks to construct an image filling task training dataset, use the image filling task training dataset to train the image restoration model, and save the model weight file after the training is completed;

[0010] S3. Based on the model weight file, the trained image restoration model is retrained using the image filling task training dataset. The image restoration model during training is optimized by combining the learning rate scheduling method of cosine annealing and hot restart. Based on the optimized image restoration model, the model optimization weight file is determined and saved.

[0011] S4. Based on the model optimization weight file, the image restoration model is initialized, and the shadowed image to be processed is removed by using the initialized image restoration model to obtain an image after the shadows are removed.

[0012] Preferably, the image restoration model is constructed based on a single-task image enhancement model in combination with a pre-trained image and text enhancement encoder and a text encoder, comprising the following steps:

[0013] S11. Based on the pre-configured single-task image enhancement model, the initial model structure is obtained by introducing the image enhancement encoder in the pre-trained image and text enhancement encoder.

[0014] S12. Based on the initial model structure, a multi-level model structure is obtained by introducing a text encoder;

[0015] S13. Based on the multi-level model structure, an image restoration model is obtained by introducing a task selection module and a hybrid attention module;

[0016] wherein the image enhancement encoder is used to extract image features corresponding to image encoder weights;

[0017] The text encoder is used to encode the task description text into an embedding vector to obtain task semantic features, so as to achieve conversion from text to semantic feature vectors;

[0018] The task selection module is used to fuse image features and task semantic features using a cross-attention mechanism to obtain a task-guided feature map;

[0019] The hybrid attention module is used to extract features at different levels and generate scale features by splicing, and dynamically adjust the spatial and channel weights in combination with the task-guided feature map.

[0020] Preferably, the method of fusing image features and task semantic features using a cross-attention mechanism to obtain a task-guided feature map comprises the following steps:

[0021] The task semantic features are input into the preset linear layer. Based on the weight matrix learned by the linear layer during training, the task semantic features are linearly transformed and mapped into projection vectors that match the image feature dimensions.

[0022] According to the image features and projection vectors, the cross-attention mechanism is used to calculate the fusion features and obtain the task-guided feature map;

[0023] The calculation formula of the task-guided feature map is:

[0024] ;

[0025] Where, F ca represents the feature map guided by the task, represents the cross attention calculation, F img represents the image features, E task Represents the projection vector.

[0026] Preferably, extracting features at different levels and generating scale features by splicing, combining the feature maps guided by the task, and dynamically adjusting the spatial and channel weights include the following steps:

[0027] Extract features at different levels from the image enhancement encoder in the image-text enhancement encoder, and generate multi-scale features through upsampling and splicing to obtain the task semantic vector;

[0028] Based on the self-attention mechanism, the task semantic vector is fused with the multi-scale features to obtain the self-attention fusion feature;

[0029] The feedforward network is used to perform residual connections on the self-attention fusion features and the task-guided feature maps to obtain features with enhanced nonlinear expression capabilities through the feedforward network.

[0030] Based on the features after the feedforward network enhances the nonlinear expression ability, the weight of the attention channel is adjusted;

[0031] The calculation formula of the task semantic vector is:

[0032] ;

[0033] Where, F mul represents the task semantic vector, Concat represents concatenation, UpSample represents upsampling, and F low 、F mid 、F high Represents features at different levels respectively;

[0034] The calculation formula of the self-attention fusion feature is:

[0035] ;

[0036] Where, F refined represents the self-attention fusion feature, SelfAttention(·) represents the self-attention mechanism, F mul Represents the task semantic vector, E task Represents the projected text semantic vector;

[0037] The calculation formula of the feature after the feedforward network enhances the nonlinear expression ability is:

[0038] ;

[0039] Where, F fined represents the feature after the feedforward network enhances the nonlinear expression ability, FFN(·) represents the feedforward neural network, F ca represents the task-guided feature map, F refined Represents the self-attention fusion feature.

[0040] Preferably, the method of constructing an image filling task training dataset using random masks based on the real images of the shadow removal dataset, training the image restoration model using the image filling task training dataset, and saving the model weight file after the training is completed comprises the following steps:

[0041] S21. Collect a shadow removal dataset containing real images, preprocess the shadow removal dataset using a shadow detection model, generate a mask using a random mask method based on the preprocessing results, and generate an image filling task training dataset based on the mask;

[0042] S22, filling the task training data set according to the image and configuring the task prompt data;

[0043] S23, based on the task prompt data and in combination with the image filling task training data set, iteratively training the image restoration model until the image restoration model reaches convergence;

[0044] S24. For the trained image restoration model, extract and ensure the trained weight file of the image restoration model.

[0045] Preferably, the step of collecting a shadow removal dataset containing real images, preprocessing the shadow removal dataset using a shadow detection model, generating a mask using a random mask method based on the preprocessing result, and generating an image filling task training dataset based on the mask comprises the following steps:

[0046] S211, collecting a shadow removal dataset containing real images, and performing shadow area recognition processing on the shadow removal dataset using a shadow detection model;

[0047] S212, for a real image containing a shadow area, mapping it to a real image without a shadow area to obtain a corresponding spatial coordinate position;

[0048] S213: Perform operations in the mapped shadow area to randomly generate several geometrically shaped masks. The size of the masks on the real image is calculated to determine whether the masks meet the prediction threshold range. If so, execute step S214. If not, regenerate the geometrically shaped masks.

[0049] S214. Based on the generated mask, pair it with the real image in the shadow removal dataset to generate an image filling task training dataset for training.

[0050] Preferably, the method of retraining the trained image restoration model based on the model weight file using the image filling task training data set, and optimizing the image restoration model during training by combining the learning rate scheduling method of cosine annealing and hot restart, and determining and saving the model optimization weight file according to the optimized image restoration model includes the following steps:

[0051] S31, loading the model weight file into the image restoration model, and performing weight initialization processing on the image restoration model;

[0052] S32, using the image filling task training dataset as the image input for image restoration model training, and configuring task prompt data for the image restoration model;

[0053] S33, using the image filling task training data set and task prompt data to train the image restoration model, and combining the cosine annealing and hot restart learning rate scheduling method to optimize the learning rate of the image restoration model;

[0054] S34. Determine whether the image restoration model has reached convergence conditions in the shadow removal task. If so, stop training and ensure the model optimization weight file after the image restoration model training is completed. If not, return to step S33.

[0055] Preferably, the learning rate scheduling method combining cosine annealing and hot restart to optimize the learning rate of the image restoration model includes:

[0056] The learning rate of the image restoration model is gradually reduced through the cosine function, and the learning rate peak attenuation mechanism is introduced to control the learning rate of the image restoration model, forming a decreasing hot restart;

[0057] The learning rate of the image restoration model is gradually reduced by the cosine function, and the learning rate of the image restoration model is controlled by introducing a learning rate peak attenuation mechanism, forming an expression for a decreasing hot restart:

[0058] ;

[0059] ;

[0060] Where, lr t Indicates the learning rate of the current training times, lr min 、lr max Represent the minimum and maximum values ​​of the learning rate, T cycle Indicates the length of a complete training cycle, t indicates the current number of training times, represents the maximum learning rate for the kth training cycle.

[0061] Preferably, the method of optimizing the weight file based on the model, initializing the image restoration model, and removing the shadows from the image to be processed by using the initialized image restoration model to obtain the image after removing the shadows comprises the following steps:

[0062] S41, based on the model optimization weight file, performing a secondary weight initialization process on the image restoration model trained in step S3;

[0063] S42, obtaining a shadowed image to be processed, and inputting it into the image restoration model after weight secondary initialization processing;

[0064] S43. The image restoration model automatically detects the shadow area through the learned multi-task processing module and generates an image with the shadow removed.

[0065] According to another aspect of the present invention, there is provided an image shadow removal system based on image filling enhancement, comprising:

[0066] The model construction module is used to build an image restoration model based on the single-task image enhancement model, combined with the pre-trained image and text enhancement encoder, text encoder, and task selection module;

[0067] A one-time training module is used to construct an image filling task training dataset using random masks based on the real images of the shadow removal dataset, train the image restoration model using the image filling task training dataset, and save the model weight file after the training is completed;

[0068] The secondary training module is used to retrain the trained image restoration model based on the model weight file and the image filling task training dataset. It also optimizes the image restoration model during training by combining the learning rate scheduling method of cosine annealing and hot restart. Based on the optimized image restoration model, the model optimization weight file is determined and saved.

[0069] The shadow processing module is used to initialize the image restoration model based on the model optimization weight file, and use the initialized image restoration model to perform shadow removal processing on the shadowed image to be processed to obtain the image after shadow removal.

[0070] The beneficial effects of the present invention are:

[0071] 1. This paper introduces a guided multi-task learning mechanism to achieve end-to-end shadow detection and removal, while improving the model's comprehensive processing capabilities for complex degradation. On the basis of the single-task image enhancement model, it adds a multimodal encoder based on CLIP pre-training, and uses the cross-attention mechanism to fuse text guidance and image features, realizing dynamic adaptation of tasks and significantly improving the model's processing capabilities for complex degradation scenarios.

[0072] 2. The present invention uses the shadow removal dataset to construct the image filling task, pre-trains the model through a controllable mask generation mechanism, strengthens the model's global understanding of image structure and texture, and provides high-quality initial weights for fine-tuning of subsequent shadow removal tasks, significantly shortening the training cycle and improving model performance.

[0073] 3. The present invention uses transfer learning optimization to transfer the pre-trained weights to the shadow removal task for model training. Combined with the task prompt mechanism, it ensures that the model can achieve rapid convergence of model parameters and improve the performance of shadow removal in complex lighting environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0075] Figure 1 is a flow chart of an image shadow removal method based on image filling enhancement according to an embodiment of the present invention;

[0076] Figure 2 is a principle block diagram of an image shadow removal system based on image filling enhancement according to an embodiment of the present invention;

[0077] Figure 3 1 is an overall structural diagram of a shadow removal method for an image based on image filling enhancement according to an embodiment of the present invention;

[0078] Figure 4This is a flow chart of high-quality feature map output based on a pre-trained CLIP image encoder in an image shadow removal method based on image filling enhancement according to an embodiment of the present invention;

[0079] Figure 5 The present invention is a flowchart of multi-task processing in an image shadow removal method based on image filling enhancement according to an embodiment of the present invention.

[0080] In the picture:

[0081] 1. Model building module; 2. Primary training module; 3. Secondary training module; 4. Shadow processing module. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0083] According to an embodiment of the present invention, a method and system for image shadow removal based on image filling enhancement are provided.

[0084] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 and Figure 3-5 As shown, according to one embodiment of the present invention, a method for removing shadows from an image based on image filling enhancement is provided, comprising the following steps:

[0085] S1. Based on the single-task image enhancement model, we build an image restoration model by combining the pre-trained image and text enhancement encoder, text encoder, and task selection module.

[0086] As a preferred embodiment, the image restoration model is constructed based on a single-task image enhancement model in combination with a pre-trained image and text enhancement encoder and a text encoder, comprising the following steps:

[0087] S11. Based on the pre-configured single-task image enhancement model, the initial model structure is obtained by introducing the image enhancement encoder in the pre-trained image and text enhancement encoder.

[0088] S12. Based on the initial model structure, a multi-level model structure is obtained by introducing a text encoder;

[0089] S13. Based on the multi-level model structure, an image restoration model is obtained by introducing a task selection module and a hybrid attention module;

[0090] wherein the image enhancement encoder is used to extract image features corresponding to image encoder weights;

[0091] The text encoder is used to encode the task description text into an embedding vector to obtain task semantic features, so as to achieve conversion from text to semantic feature vectors;

[0092] The task selection module is used to fuse image features and task semantic features using a cross-attention mechanism to obtain a task-guided feature map;

[0093] As a preferred embodiment, the cross-attention mechanism is used to fuse image features and task semantic features to obtain a task-guided feature map, which includes the following steps:

[0094] The task semantic features are input into the preset linear layer. Based on the weight matrix learned by the linear layer during training, the task semantic features are linearly transformed and mapped into projection vectors that match the image feature dimensions.

[0095] According to the image features and projection vectors, the cross-attention mechanism is used to calculate the fusion features and obtain the task-guided feature map;

[0096] The calculation formula of the task-guided feature map is:

[0097] ;

[0098] Where, F ca represents the feature map guided by the task, represents the cross attention calculation, F img represents the image features, E task Represents the projection vector.

[0099] The hybrid attention module is used to extract features at different levels and generate scale features by splicing, and dynamically adjust the spatial and channel weights in combination with the task-guided feature map.

[0100] As a preferred embodiment, the extraction of features at different levels and generation of scale features through splicing, combined with the task-guided feature map, and dynamic adjustment of spatial and channel weights include the following steps:

[0101] Extract features at different levels from the image enhancement encoder in the image-text enhancement encoder, and generate multi-scale features through upsampling and splicing to obtain the task semantic vector;

[0102] Based on the self-attention mechanism, the task semantic vector is fused with the multi-scale features to obtain the self-attention fusion feature;

[0103] The feedforward network is used to perform residual connections on the self-attention fusion features and the task-guided feature maps to obtain features with enhanced nonlinear expression capabilities through the feedforward network.

[0104] Based on the features after the feedforward network enhances the nonlinear expression ability, the weight of the attention channel is adjusted.

[0105] It should be noted that based on the general single-task image enhancement model, the image enhancement encoder in the pre-trained CLIP visual encoder is introduced (the image enhancement encoder weights are derived from the pre-trained CLIP model. The network structure of the image enhancement encoder is exactly the same as the CLIP visual encoder, so the CLIP pre-trained weights are directly loaded as the initial parameters) and the image features of the corresponding image encoder weights are extracted:

[0106] ;

[0107] In the formula, x represents image data, Encoder image (x) represents the image encoder, F img Represents weighted image features (contextual information obtained after feature extraction).

[0108] It's important to note that weights are the encoder's "parameters," while context is the encoder's "output." Contextual information guides optimization of weights. Specifically, weights are the encoder's "knowledge carrier," determining how to extract context from the input image; contextual information is the "inference result" of the weights on a specific image. These two factors form a dependency relationship between "model parameters" and "feature outputs."

[0109] Among them, the extracted contextual information (image features) and the internal features of the model are dynamically enhanced through self-attention and cross-attention to enhance the feature response value of the shadow-related area).

[0110] In addition, using the text encoder Encoder text Task description text Task prompt Encoding into semantic vector Task embedding , which can realize the conversion from text to semantic feature vector. The calculation formula for the conversion from text to semantic feature vector is:

[0111] ;

[0112] In the formula, Task embedding Represents semantic vector, Encoder text (·) represents the text encoder, Task prompt Represents the task description text.

[0113] Then, the image features (the image feature map here is obtained by inputting the original image into the CLIP image encoder) and the text vector are simultaneously fed into the task selection module. First, the text encoding is mapped (projected) to the same dimension as the feature map, and the text and image feature spaces are aligned to achieve cross-modal interaction. The calculation formula for mapping (projecting) the text encoding to the same dimension as the feature map is:

[0114] ;

[0115] Where, E task represents the projection vector that matches the image feature dimension, Linear(·) represents the linear layer, Task embedding Represents the task embedding vector (semantic vector).

[0116] It should be noted that the weight matrix of the linear layer is learned during the training process and can adaptively generate different weight distributions based on the semantic features of the input task text (such as "remove shadows" or "repair insulator areas").

[0117] The task semantic features are extracted through the text encoder, and then mapped through the linear layer to make the weight value task-specific, laying the foundation for the subsequent dynamic adjustment of the feature map weight. Specifically: Task embedding Perform linear transformation, the weight matrix of the linear layer is learned during the training process, which is based on the Task embedding The semantic features of the image are mapped into projection vectors that match the dimensions of the image feature map. embedding (Corresponding to different task texts) When passing through the linear layer, due to different input features, the output E task Also different.

[0118] In addition, when dynamically fusing text semantics and image features to generate task-guided feature maps, Q, K, and V in the cross-attention mechanism are first defined, where Q represents the image feature F. img ; K, V are the projected text semantic vectors E task , calculate the fusion features based on the cross attention mechanism:

[0119] ;

[0120] Specifically, the original information can be retained through residual connections, and the feedforward network (FFN) can enhance the nonlinear expression ability, including:

[0121] Extract features at different levels from the CLIP encoder {F low 、F mid 、F high}, generate multi-scale features by upsampling and splicing:

[0122] ;

[0123] Where, F mul represents the task semantic vector, Concat represents concatenation, UpSample represents upsampling, and F low 、F mid 、F high Represents features at different levels respectively;

[0124] Then the task semantic vector F mul Fusion with multi-scale features to enhance key area responses:

[0125] ;

[0126] Where, F refined represents the self-attention fusion feature, SelfAttention(·) represents the self-attention mechanism, F mul Represents the task semantic vector, E task Represents the projected text semantic vector;

[0127] The original information is retained through residual connections, and the feedforward network (FFN) enhances the nonlinear expression ability. The calculation formula of the feature after the feedforward network enhances the nonlinear expression ability is:

[0128] ;

[0129] Where, F final represents the feature after the feedforward network enhances the nonlinear expression ability, FFN(·) represents the feedforward neural network, F ca represents the task-guided feature map, F refined Represents the self-attention fusion feature.

[0130] Specifically, by introducing a hybrid attention module, spatial and channel weights are dynamically adjusted:

[0131] ;

[0132] ;

[0133] ;

[0134] Where M spatial represents the spatial attention weight map, M channel represents the channel attention weight map, SpatialAttention represents the spatial attention module, F final It represents the feature after the feedforward network enhances the nonlinear expression ability, F outputIt represents the features after spatial and channel attention adjustment, and ChannelAttention represents the channel attention module.

[0135] Then, according to the task requirements, the final de-shadowed image is generated through a lightweight convolutional layer:

[0136] ;

[0137] Where OutPut represents the enhanced feature map after task semantics guidance, Conv(·) represents the convolutional layer, Decoder represents the decoder module, and F output Represents the features after spatial and channel attention adjustment.

[0138] S2. Based on the real images of the de-shadowing dataset, use random masks to construct an image filling task training dataset, use the image filling task training dataset to train the image restoration model, and save the model weight file after the training is completed.

[0139] As a preferred embodiment, the method of constructing an image filling task training dataset using random masks based on the real images of the shadow removal dataset, training the image restoration model using the image filling task training dataset, and saving the model weight file after the training is completed includes the following steps:

[0140] S21. Collect a shadow removal dataset containing real images, preprocess the shadow removal dataset using a shadow detection model, generate a mask using a random mask method based on the preprocessing results, and generate an image filling task training dataset based on the mask;

[0141] As a preferred embodiment, the method of collecting a shadow removal dataset containing real images, preprocessing the shadow removal dataset using a shadow detection model, generating a mask using a random mask method based on the preprocessing result, and generating an image filling task training dataset based on the mask includes the following steps:

[0142] S211, collecting a shadow removal dataset containing real images, and performing shadow area recognition processing on the shadow removal dataset using a shadow detection model;

[0143] S212, for a real image containing a shadow area, mapping it to a real image without a shadow area to obtain a corresponding spatial coordinate position;

[0144] S213: Perform operations in the mapped shadow area to randomly generate several geometrically shaped masks. The size of the masks on the real image is calculated to determine whether the masks meet the prediction threshold range. If so, execute step S214. If not, regenerate the geometrically shaped masks.

[0145] S214. Based on the generated mask, pair it with the real image in the shadow removal dataset to generate an image filling task training dataset for training.

[0146] S22, filling the task training data set according to the image and configuring the task prompt data;

[0147] S23, based on the task prompt data and in combination with the image filling task training data set, iteratively training the image restoration model until the image restoration model reaches convergence;

[0148] S24. For the trained image restoration model, extract and ensure the trained weight file of the image restoration model.

[0149] It should be noted that when using the images in the shadow removal task dataset as the material, the idea of ​​random masking was used to create the image filling dataset (in the image filling dataset construction method, for each 512×512 high-quality power equipment image, a mask was generated in the shadow corresponding area with a 60% probability (pre-generated by the shadow detection model).

[0150] The mask shapes are randomly selected as rectangles (40%), ellipses (30%), and polygons (30%), with an area controlled between 1 / 12 and 1 / 4 of the image area. The generated masked image is obtained by setting the pixels in the masked area to zero, with the original image used as the label to meet the data requirements for image infill pre-training in step S22. The masked area is divided into shadow and non-shadow areas (the shadow mask output by the shadow detection model is used to locate the coordinates of the true shadow area. A probabilistic control strategy (6:4 ratio) and logical constraint operations are combined to ensure that the mask is generated only within the target area). The mask is generated in the shadow and non-shadow areas with a ratio of 6:4.

[0151] First, a shadow detection model is used to detect shadow regions in low-quality images (images with shadows) from the shadow removal dataset. Next, a mask is generated on high-quality images (clear images without shadows). If a mask is required in a shadowed area, it is generated within the detected shadow region; otherwise, the mask is generated to avoid the shadowed region. The mask area is kept between 1 / 4 and 1 / 12 of the image size, ensuring the challenge of the training task while avoiding loss of image detail due to an overly large mask. Finally, the masked image is paired with the original image to form training data pairs, generating the image filling dataset D1.

[0152] Specifically, the shadow detection model uses a U-Net architecture, with a pre-trained ResNet50 encoder and a decoder consisting of four layers of transposed convolutions. The input is RGB + HSL three-channels (for a total of six channels). The output is segmented using Otsu thresholding and then subjected to a 3×3 kernel morphological operation to generate a binary mask. Training data uses the ISTD dataset (1,500 image pairs) and a self-built power inspection dataset (800 image pairs). The loss function is BCE + Dice (α = 0.75), and the optimizer is AdamW (learning rate 1e-4). After 100 epochs of training, the Dice coefficient on the validation set reached 0.92, meeting the required accuracy for shadow area positioning.

[0153] In addition, when generating a mask on a high-quality image (a clear image without shadows), the low-quality shadowed image is processed through the shadow detection model to determine the location of the shadow area, output a binary mask (1 for the shadow area, 0 for the non-shadow area), and map it to the corresponding spatial coordinate position of the high-quality image to ensure pixel-level alignment.

[0154] The operation is performed within the mapped shadow region (the target region output by the shadow detection model is element-wise multiplied by the randomly generated region mask, ensuring that the mask is only valid within the target region). A random geometric shape mask is generated and the shape area is calculated, ensuring that it is between 1 / 4 and 1 / 12 of the image size. If it is not satisfied, the geometric shape is regenerated.

[0155] It should be noted that the accuracy and generalization ability of shadow area mask generation can be improved by combining self-supervised contrastive learning and dynamic deformable convolutional networks in an end-to-end data processing process, including: data enhancement and positive and negative sample construction. By randomly cropping, rotating, and perturbing the illumination of unlabeled power equipment images, enhanced image pairs are generated; positive sample pairs are composed of different enhanced versions of the same image; negative sample pairs are composed of enhanced versions of different images.

[0156] Among them, a positive sample pair refers to the same image after enhancement to ensure the semantic consistency of key components (the mask intersection over union (IoU) is calculated by a pre-trained segmentation model, and it is accepted as a positive sample pair only when IoU>80%), but there are differences in the shadow area (such as changes in shadow density and range); a negative sample pair refers to the selection of semantically similar but different types of device images (such as transformers of different models), which are constructed into comparison pairs after enhancement to force the model to learn fine-grained feature differences.

[0157] In addition, the model calculates the confidence level of its predictions for each sample. Samples with high confidence but incorrect predictions are defined as "hard negative samples." A sample difficulty assessment module is introduced during training. Based on the cross-entropy between the model's predicted probability and the true label, samples with a cross-entropy greater than a set threshold are selected as hard negative samples. The model dynamically adjusts sample weights, increasing the weight of hard negative samples by 1.5 times in contrastive learning. Furthermore, a corresponding "adversarially enhanced version" is generated (for example, by adding a small perturbation to the shaded area of ​​the hard negative sample), improving the model's robustness to error-prone samples.

[0158] In addition, when using the deep neural network VIT to extract image features, a high-dimensional feature representation F1 is obtained, wherein dynamic deformable convolution is introduced in the feature extraction layer, and the offset Δp is learned. k Adaptively adjust the convolution kernel sampling position to accurately capture the irregular boundaries and details of the shadow. For the input feature map, the output features are:

[0159] ;

[0160] Among them, P represents the current position, k represents the sampling point index of the convolution kernel, and p k Represents the original convolution kernel position, w k represents the weight, Δm k represents the modulation factor.

[0161] Specifically, by cascading deformable convolutional layers with different expansion rates, a multi-scale receptive field feature pyramid is constructed. Combining the outputs of dynamic deformable convolutions at different scales fuses high-level semantics with underlying details, improving the representation of complex shadow structures. The decoding portion of the U-Net segmentation architecture is used to upsample the features output by the dynamic deformable convolutional network, restore the image resolution, and output a shadow region mask.

[0162] While training the model, we also optimize the contrastive learning loss L contrastive and cross entropy segmentation loss L seg .

[0163] ;

[0164] In the formula, s(·) represents the cosine similarity score, N represents the number of positive samples involved in the calculation, and f i ,f j Both represent the feature vectors of positive sample images, f n represents the features of other images (negative samples), y is the true label (0 or 1), p represents the probability of the model predicting the positive class, τ∈(0,1) is the temperature coefficient, and λ1 represents the hyperparameter.

[0165] It's important to note that after generating dataset D1, the task prompt is set to "Please help me fill in the missing parts of the image," clarifying that the current task is image completion. This prompt allows the model to dynamically adjust its feature extraction and restoration strategies (through the "semantic encoding → weight map generation → feature modulation → strategy adjustment" process). This allows the model to dynamically optimize feature extraction direction and restoration logic for specific tasks (such as "filling in the missing parts of the image"), improving the accuracy and pertinence of task execution and focusing on repairing missing areas.

[0166] Finally, using the task prompts, pre-train the multi-task model (image restoration model) for 50 iterations until the model converges. This concludes the model training for the image filling task. After training, save the resulting weight file, W1, to be used as the initial weights for the subsequent shadow removal task.

[0167] S3. Based on the model weight file, the trained image restoration model is retrained using the image filling task training dataset. The image restoration model during training is optimized by combining the learning rate scheduling method of cosine annealing and hot restart. Based on the optimized image restoration model, the model optimization weight file is determined and saved.

[0168] As a preferred embodiment, the image restoration model is trained again based on the model weight file using the image filling task training data set, and the image restoration model in the training process is optimized by combining the learning rate scheduling method of cosine annealing and hot restart. According to the optimized image restoration model, determining and saving the model optimization weight file includes the following steps:

[0169] S31, loading the model weight file into the image restoration model, and performing weight initialization processing on the image restoration model;

[0170] S32, using the image filling task training dataset as the image input for image restoration model training, and configuring task prompt data for the image restoration model;

[0171] S33, using the image filling task training data set and task prompt data to train the image restoration model, and combining the cosine annealing and hot restart learning rate scheduling method to optimize the learning rate of the image restoration model;

[0172] S34. Determine whether the image restoration model has reached convergence conditions in the shadow removal task. If so, stop training and ensure the model optimization weight file after the image restoration model training is completed. If not, return to step S33.

[0173] It's important to note that the shadow removal dataset used as the image input for model training (image restoration model) contains low-quality images with shadows and their corresponding high-quality, shadow-free images. Furthermore, the task prompt is set to "Please help me remove shadows from this image." This text encoding guides the model to focus on feature extraction and restoration of shadow areas, clarifying the current task objective.

[0174] Use the saved pre-trained weight W1 as the weight parameter for model initialization, use the set data set and task prompts to train the model until the model reaches convergence conditions on the shadow removal task, and save the final optimized weight W2.

[0175] In addition, a dynamic learning rate scheduling method is proposed, combining cosine annealing with hot restarts. This method gradually reduces the learning rate using a cosine function to help the model converge more stably in the later stages of training. A learning rate peak decay mechanism is also introduced. After each restart, the learning rate peak is decayed by 80%, and the peak value is decayed to 80% of the previous cycle, forming a decreasing hot restart.

[0176] As a preferred embodiment, the learning rate scheduling method combining cosine annealing and hot restart to optimize the learning rate of the image restoration model includes:

[0177] The learning rate of the image restoration model is gradually reduced through the cosine function, and the learning rate peak attenuation mechanism is introduced to control the learning rate of the image restoration model, forming a decreasing hot restart;

[0178] The learning rate of the image restoration model is gradually reduced by the cosine function, and the learning rate of the image restoration model is controlled by introducing a learning rate peak attenuation mechanism, forming an expression for a decreasing hot restart:

[0179] ;

[0180] ;

[0181] Where, lr t Indicates the learning rate of the current training times, lr min 、lr max Represent the minimum and maximum values ​​of the learning rate, T cycle Indicates the length of a complete training cycle, t indicates the current number of training times, Indicates the maximum learning rate of the kth training cycle. The initial parameters are set to: lr max =1e-6,lr min =1e-6, T cycle =20.

[0182] S4. Based on the model optimization weight file, the image restoration model is initialized, and the shadowed image to be processed is removed by using the initialized image restoration model to obtain an image after the shadows are removed.

[0183] As a preferred embodiment, the method of optimizing the weight file based on the model, initializing the image restoration model, and using the initialized image restoration model to remove the shadows of the image to be processed to obtain the image after removing the shadows includes the following steps:

[0184] S41, based on the model optimization weight file, performing a secondary weight initialization process on the image restoration model trained in step S3;

[0185] S42, obtaining a shadowed image to be processed, and inputting it into the image restoration model after weight secondary initialization processing;

[0186] S43. The image restoration model automatically detects the shadow area through the learned multi-task processing module and generates an image with the shadow removed.

[0187] It should be noted that the specific implementation steps for initializing the image restoration model based on the model optimization weight file and removing the shadows from the image to be processed using the initialized image restoration model to obtain the image after removing the shadows are as follows:

[0188] Model loading: Load the trained weight file W2 into the image restoration model for image filling enhancement to complete model initialization.

[0189] Input the image to be processed: A low-quality image with shadows to be processed is fed into the model, supporting single-image or batch processing. Combined with a text encoder, the model inputs a task prompt (such as "Please help me remove the shadows in this image"). The model dynamically adjusts the feature extraction strategy based on the text guidance, focusing on restoring the shadowed area without manual intervention.

[0190] End-to-end reasoning: The model automatically detects shadow areas through a learned multi-task processing module (the model extracts global image features through the CLIP encoder and combines it with the cross-attention mechanism to fuse text guidance information. The multi-task processing module dynamically adjusts the feature map according to the task weights, prioritizing the restoration of the texture and structure of the shadow area). Combined with the pre-trained structural texture understanding capabilities of the image filling task, it generates a high-quality image after removing the shadows.

[0191] Output result: Directly output a clear image after removing shadows, which can be used for subsequent power inspection defect identification, industrial inspection and other tasks.

[0192] like Figure 2According to another embodiment of the present invention, there is provided an image shadow removal system based on image filling enhancement, comprising:

[0193] Model construction module 1 is used to build an image restoration model based on a single-task image enhancement model, combined with a pre-trained image and text enhancement encoder, a text encoder, and a task selection module;

[0194] A training module 2 is used to construct an image filling task training dataset using random masks based on the real images of the shadow removal dataset, train the image restoration model using the image filling task training dataset, and save the model weight file after the training is completed;

[0195] Secondary training module 3 is used to train the trained image restoration model again based on the model weight file and the image filling task training dataset, and optimize the image restoration model during training by combining the learning rate scheduling method of cosine annealing and hot restart. According to the optimized image restoration model, the model optimization weight file is determined and saved;

[0196] The shadow processing module 4 is used to initialize the image restoration model based on the model optimization weight file, and use the initialized image restoration model to perform shadow removal processing on the shadowed image to be processed to obtain the image after shadow removal.

[0197] In summary, with the help of the above-mentioned technical solutions, the present invention achieves end-to-end shadow detection and removal by introducing a guided multi-task learning mechanism, while also improving the model's comprehensive handling capabilities for complex degradation. Based on a single-task image enhancement model, this invention adds a multimodal encoder based on CLIP pre-training and utilizes a cross-attention mechanism to fuse text guidance with image features, achieving dynamic task adaptation and significantly improving the model's handling capabilities for complex degradation scenarios. The present invention constructs an image filling task using a shadow removal dataset and pre-trains the model through a controllable mask generation mechanism, strengthening the model's global understanding of image structure and texture. It also provides high-quality initial weights for subsequent fine-tuning of the shadow removal task, significantly shortening the training cycle and improving model performance. Through transfer learning optimization, the present invention transfers pre-trained weights to the shadow removal task for model training. Combined with a task prompt mechanism, this ensures rapid convergence of model parameters and improved shadow removal performance in complex lighting environments.

[0198] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0199] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image shadow removal method based on image filling enhancement, characterized in that: The following steps are involved: S1. Based on the single-task image enhancement model, we build an image restoration model by combining the pre-trained image and text enhancement encoder, text encoder, and task selection module. S2. Based on the real images of the shadow removal dataset, use random masks to construct an image filling task training dataset, use the image filling task training dataset to train the image restoration model, and save the model weight file after the training is completed; S3. Based on the model weight file, the trained image restoration model is retrained using the image filling task training dataset. The image restoration model during training is optimized by combining the learning rate scheduling method of cosine annealing and hot restart. Based on the optimized image restoration model, the model optimization weight file is determined and saved. S4. Initializing the image restoration model based on the model optimization weight file, and using the initialized image restoration model to perform shadow removal on the shadowed image to be processed, to obtain a shadow-removed image; The image restoration model is constructed based on the single-task image enhancement model and combined with the pre-trained image and text enhancement encoder and text encoder, including the following steps: S11. Based on the pre-configured single-task image enhancement model, the initial model structure is obtained by introducing the image enhancement encoder in the pre-trained image and text enhancement encoder. S12. Based on the initial model structure, a multi-level model structure is obtained by introducing a text encoder; S13. Based on the multi-level model structure, an image restoration model is obtained by introducing a task selection module and a hybrid attention module; wherein the image enhancement encoder is used to extract image features corresponding to image encoder weights; The text encoder is used to encode the task description text into an embedding vector to obtain task semantic features, so as to achieve conversion from text to semantic feature vectors; The task selection module is used to fuse image features and task semantic features using a cross-attention mechanism to obtain a task-guided feature map; The hybrid attention module is used to extract features at different levels and generate scale features by splicing, and dynamically adjust the spatial and channel weights in combination with the task-guided feature map.

2. The image shadow removal method based on image filling enhancement according to claim 1, characterized in that: The method of fusing image features and task semantic features using the cross attention mechanism to obtain a task-guided feature map includes the following steps: The task semantic features are input into the preset linear layer. Based on the weight matrix learned by the linear layer during training, the task semantic features are linearly transformed and mapped into projection vectors that match the image feature dimensions. According to the image features and projection vectors, the cross-attention mechanism is used to calculate the fusion features and obtain the task-guided feature map; The calculation formula of the task-guided feature map is: ； Where, F ca represents the feature map guided by the task, represents the cross attention calculation, F img represents the image features, E task Represents the projection vector.

3. The image shadow removal method based on image filling enhancement according to claim 1, characterized in that: The process of extracting features at different levels and generating scale features through splicing, combining the feature maps guided by the task, and dynamically adjusting the spatial and channel weights includes the following steps: Extract features at different levels from the image enhancement encoder in the image-text enhancement encoder, and generate multi-scale features through upsampling and splicing to obtain the task semantic vector; Based on the self-attention mechanism, the task semantic vector is fused with the multi-scale features to obtain the self-attention fusion feature; The feedforward network is used to perform residual connections on the self-attention fusion features and the task-guided feature maps to obtain features with enhanced nonlinear expression capabilities through the feedforward network. Based on the features after the feedforward network enhances the nonlinear expression ability, the weight of the attention channel is adjusted; The calculation formula of the task semantic vector is: ； Where, F mul represents the task semantic vector, Concat represents concatenation, UpSample represents upsampling, and F low 、F mid 、F high Represents features at different levels respectively; The calculation formula of the self-attention fusion feature is: ； Where, F refined represents the self-attention fusion feature, SelfAttention(·) represents the self-attention mechanism, F mul Represents the task semantic vector, E task Represents the projected text semantic vector; The calculation formula of the feature after the feedforward network enhances the nonlinear expression ability is: ； Where, F final represents the feature after the feedforward network enhances the nonlinear expression ability, FFN(·) represents the feedforward neural network, F ca represents the task-guided feature map, F refined Represents the self-attention fusion feature.

4. The image shadow removal method based on image filling enhancement according to claim 1, characterized in that: The method comprises the following steps: constructing an image filling task training dataset using random masks based on real images of the shadow removal dataset, training an image restoration model using the image filling task training dataset, and saving a model weight file after the training is completed: S21. Collect a shadow removal dataset containing real images, preprocess the shadow removal dataset using a shadow detection model, generate a mask using a random mask method based on the preprocessing results, and generate an image filling task training dataset based on the mask; S22, filling the task training data set according to the image and configuring the task prompt data; S23, based on the task prompt data and in combination with the image filling task training data set, iteratively training the image restoration model until the image restoration model reaches convergence; S24. For the trained image restoration model, extract and ensure the trained weight file of the image restoration model.

5. The image shadow removal method based on image filling enhancement according to claim 4, characterized in that: The method of collecting a shadow removal dataset containing real images, preprocessing the shadow removal dataset using a shadow detection model, generating a mask using a random mask method based on the preprocessing result, and generating an image filling task training dataset based on the mask includes the following steps: S211, collecting a shadow removal dataset containing real images, and performing shadow area recognition processing on the shadow removal dataset using a shadow detection model; S212, for a real image containing a shadow area, mapping it to a real image without a shadow area to obtain a corresponding spatial coordinate position; S213: Perform operations in the mapped shadow area to randomly generate several geometrically shaped masks. The size of the masks on the real image is calculated to determine whether the masks meet the prediction threshold range. If so, execute step S214. If not, regenerate the geometrically shaped masks. S214. Based on the generated mask, pair it with the real image in the shadow removal dataset to generate an image filling task training dataset for training.

6. The image shadow removal method based on image filling enhancement according to claim 1, characterized in that: The method of retraining the trained image restoration model based on the model weight file and the image filling task training dataset is combined with the learning rate scheduling method of cosine annealing and hot restart to optimize the image restoration model during training. The method of determining and saving the model optimization weight file according to the optimized image restoration model includes the following steps: S31, loading the model weight file into the image restoration model, and performing weight initialization processing on the image restoration model; S32, using the image filling task training dataset as the image input for image restoration model training, and configuring task prompt data for the image restoration model; S33, using the image filling task training data set and task prompt data to train the image restoration model, and combining the cosine annealing and hot restart learning rate scheduling method to optimize the learning rate of the image restoration model; S34. Determine whether the image restoration model has reached convergence conditions in the shadow removal task. If so, stop training and ensure the model optimization weight file after the image restoration model training is completed. If not, return to step S33.

7. The image shadow removal method based on image filling enhancement according to claim 6, characterized in that: The learning rate scheduling method combining cosine annealing and hot restart to optimize the learning rate of the image restoration model includes: The learning rate of the image restoration model is gradually reduced through the cosine function, and the learning rate peak attenuation mechanism is introduced to control the learning rate of the image restoration model, forming a decreasing hot restart; The learning rate of the image restoration model is gradually reduced by the cosine function, and the learning rate of the image restoration model is controlled by introducing a learning rate peak attenuation mechanism, forming an expression for a decreasing hot restart: ； ； Where, lr t Indicates the learning rate of the current training times, lr min 、lr max Represent the minimum and maximum values ​​of the learning rate, T cycle Indicates the length of a complete training cycle, t indicates the current number of training times, represents the maximum learning rate for the kth training cycle.

8. The image shadow removal method based on image filling enhancement according to claim 1, characterized in that: The method of optimizing the weight file based on the model, initializing the image restoration model, and using the initialized image restoration model to perform shadow removal processing on the shadowed image to be processed to obtain the shadow-removed image includes the following steps: S41, based on the model optimization weight file, performing a secondary weight initialization process on the image restoration model trained in step S3; S42, obtaining a shadowed image to be processed, and inputting it into the image restoration model after weight secondary initialization processing; S43. The image restoration model automatically detects the shadow area through the learned multi-task processing module and generates an image with the shadow removed.

9. An image shadow removal system based on image filling enhancement, used to implement the image shadow removal method based on image filling enhancement according to any one of claims 1 to 8, characterized in that: include: The model construction module is used to build an image restoration model based on the single-task image enhancement model, combined with the pre-trained image and text enhancement encoder, text encoder, and task selection module; A one-time training module is used to construct an image filling task training dataset using random masks based on the real images of the shadow removal dataset, train the image restoration model using the image filling task training dataset, and save the model weight file after the training is completed; The secondary training module is used to retrain the trained image restoration model based on the model weight file and the image filling task training dataset. It also optimizes the image restoration model during training by combining the learning rate scheduling method of cosine annealing and hot restart. Based on the optimized image restoration model, the model optimization weight file is determined and saved. The shadow processing module is used to initialize the image restoration model based on the model optimization weight file, and use the initialized image restoration model to perform shadow removal processing on the shadowed image to be processed to obtain the image after shadow removal.

Images