Shadow removal method and system based on diffusion, segmentation and super-resolution model

Abstract

The present invention discloses a method and system for removing shadows based on diffusion, segmentation and super-resolution models, the method comprising reading an original image and reducing the resolution of the original image; inputting the image after reducing the resolution into a pre-trained multi-scale cascade diffusion model to obtain a de-shadowed image; based on a pre-trained super-resolution model, restoring the resolution of the de-shadowed image to the resolution of the original image; inputting the original image into a pre-trained image segmentation model to segment a set of image slices of shadows in the original image; inputting the image slices and a pre-defined category text list into a pre-trained multi-modal language vision model to obtain a slice subset; fusing the image after resolving the restored resolution and the slice subset to obtain a final image. The present invention realizes efficient shadow removal and image detail enhancement, significantly improves the effect of image shadow removal and the quality of processed images, and meets the image de-shadowing requirements in different scenarios.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and image processing, and discloses a shadow removal method and system based on diffusion, segmentation and super-resolution models. Background Art

[0002] In the field of computer vision and image processing, shadow removal is a key research topic. Shadows affect the quality and visual effects of images, which in turn affects subsequent image analysis tasks such as object detection, image segmentation, and scene understanding. Therefore, removing shadows from images is crucial to improving the accuracy and reliability of these tasks. Shadow removal has not only received much attention in academic research, but also has a wide range of needs in practical applications, such as autonomous driving, video surveillance, medical image processing, and augmented reality.

[0003] Existing shadow removal techniques are mainly divided into the following categories: physical model-based methods, color space transformation-based methods and image segmentation-based methods.

[0004] Physical model-based methods use the physical properties of shadows to remove shadows by modeling the relationship between light sources, objects, and shadows. For example, shadow detection and removal can be achieved by solving a set of equations in the lighting model. However, such methods rely on accurate lighting and geometric information and have poor adaptability to complex scenes and dynamically changing environments. In addition, it is difficult to obtain accurate lighting and geometric information, especially in outdoor and dynamic scenes, which limits the applicability and practicality of the model.

[0005] The method based on color space transformation is based on the distribution characteristics of shadow areas and non-shadow areas in color space, and removes shadows through color transformation. For example, contrast enhancement, color correction and image filtering techniques are used to reduce or eliminate the impact of shadows. This method relies on a predefined color model and is difficult to handle scenes with complex and varied color distributions, and the removal effect is limited. The color model has poor versatility and requires manual adjustment for different scenes. It lacks adaptive capabilities and is prone to losing color and detail information of the image when removing shadows, resulting in poor image quality.

[0006] Image segmentation-based methods remove shadows by segmenting the image into shadow and non-shadow areas and then processing the shadow areas. Common methods include threshold segmentation, region growing, and graph cut techniques. These methods can effectively detect and segment shadow areas, but when removing shadows, they often introduce new artifacts or cause loss of image details, affecting the overall quality of the image. In addition, when dealing with complex scenes, this type of method has insufficient detection and segmentation accuracy for shadow areas, resulting in poor removal effects.

[0007] It can be seen that the existing shadow removal technology has problems such as poor environmental adaptability, poor removal effect and deterioration of image quality. Therefore, further research and innovation are needed to solve the above problems existing in the existing technology. Summary of the invention

[0008] The purpose of the invention is to provide a shadow removal method and system based on diffusion, segmentation and super-resolution models to solve the above-mentioned problems existing in the prior art.

[0009] The technical solution, a shadow removal method based on diffusion, segmentation and super-resolution model, includes the following steps:

[0010] S1, read the original image, perform adaptive resolution adjustment and feature enhancement on the original image, and obtain an image with reduced resolution;

[0011] S2, inputting the image with reduced resolution into the pre-trained multi-scale cascade diffusion model to remove shadows and obtain a de-shadowed image;

[0012] S3, based on the pre-trained super-resolution model, restoring the resolution of the image after the shadow removal to the resolution of the original image, to obtain an image with restored resolution;

[0013] S4, inputting the original image into a pre-trained image segmentation model to obtain a segmentation mask of the shadow, and based on the segmentation mask, segmenting a corresponding set of image slices in the original image;

[0014] S5, inputting the image slice set and the predefined category text list of objects to be retained into the pre-trained multimodal language vision model to obtain a slice subset of objects to be retained;

[0015] S6. Fusing the image after the resolution restoration and the slice subset to obtain a final high-resolution shadow-free image.

[0016] Shadow removal system based on diffusion, segmentation and super-resolution models, including:

[0017] at least one processor; and,

[0018] a memory communicatively connected to at least one of the processors; wherein,

[0019] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the shadow removal method based on diffusion, segmentation and super-resolution models.

[0020] Beneficial effect: The present invention achieves efficient shadow removal and image detail enhancement by introducing a multi-module joint processing mechanism, improves the effect of image shadow removal and the quality of processed images, meets the image shadow removal needs in different scenarios, and has significant advantages in retaining image details and improving image fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the present invention.

[0022] Figure 2 This is a flow chart of step S1 of the present invention.

[0023] Figure 3 This is a flow chart of step S2 of the present invention.

[0024] Figure 4 This is a flow chart of step S3 of the present invention.

[0025] Figure 5 This is a flow chart of step S4 of the present invention.

[0026] Figure 6 This is a flow chart of step S5 of the present invention.

[0027] Figure 7 This is a flow chart of step S6 of the present invention. DETAILED DESCRIPTION

[0028] like Figure 1 As shown, the present application proposes a shadow removal method based on diffusion, segmentation and super-resolution model, comprising the following steps:

[0029] S1, read the original image, perform adaptive resolution adjustment and feature enhancement on the original image, and obtain an image with reduced resolution;

[0030] S2, inputting the image with reduced resolution into the pre-trained multi-scale cascade diffusion model to remove shadows and obtain a de-shadowed image;

[0031] S3, based on the pre-trained super-resolution model, restoring the resolution of the image after the shadow removal to the resolution of the original image, to obtain an image with restored resolution;

[0032] S4, inputting the original image into a pre-trained image segmentation model to obtain a segmentation mask of the shadow, and based on the segmentation mask, segmenting a corresponding set of image slices in the original image;

[0033] S5, inputting the image slice set and the predefined category text list of objects to be retained into the pre-trained multimodal language vision model to obtain a slice subset of objects to be retained;

[0034] S6. Fusing the image after the resolution restoration and the slice subset to obtain a final high-resolution shadow-free image.

[0035] In this embodiment, for any input image, the processing process is mainly divided into two branches: shadow removal and foreground optimization. Its purpose is to retain the corresponding weak parts of the shadow removal model on the basis of the general image shadow removal model to achieve better image fidelity and retain more details.

[0036] This embodiment can effectively remove the shadow part in the image through the multi-scale cascade diffusion model. The multi-scale cascade diffusion model simulates the process of shadow diffusion and dissipation to gradually remove the shadows of the image, thereby retaining more image details. The design of the multi-scale cascade diffusion model enables it to automatically adapt to different types of shadow distribution, improve the shadow removal effect, and not affect the image quality of the non-shadow area; through the image segmentation model (SAM), the image is accurately segmented into foreground and background, so as to further optimize the quality of the foreground area after removing the shadow. By accurately locating and distinguishing the foreground and background, the image after shadow removal retains higher details and clarity in the key area; through the super-resolution model, detail enhancement is performed, and the image after shadow removal is reconstructed at high resolution to further improve the clarity and detail performance of the image. The super-resolution model can convert the low-resolution shadow removal result into a high-resolution image, thereby improving the overall image quality; through the multimodal language vision model (LLAVA), specific area selection is performed, and the language model analyzes different areas in the image and selects the areas that need to be focused on. The model can intelligently identify important areas in the image and perform special processing on these areas to ensure that the shadow removal and detail enhancement effects are optimal in the key parts.

[0037] Through the combination of these technical features, high-quality image de-shading and detail retention are achieved, the effect and quality of image processing are improved, and the image de-shading needs in different scenarios are met.

[0038] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0039] S11, reading the original image and current GPU memory information, using an adaptive block strategy to divide the original image into a predetermined number of sub-blocks of unequal size, and calculating the significance score of each sub-block;

[0040] S12, adaptively assigning a resolution reduction coefficient to each sub-block based on the significance score of each sub-block and current GPU memory information;

[0041] S13, based on the resolution reduction coefficient corresponding to each sub-block, using a deformable convolutional network based on an attention mechanism to reduce the resolution of each sub-block to obtain a sub-block with reduced resolution;

[0042] S14, rejoining the sub-blocks with reduced resolution to obtain a complete image with reduced resolution.

[0043] According to one aspect of the present application, step S11 is further:

[0044] S111, reading the original image and current GPU memory information, and applying Gaussian blur to the original image to obtain a smooth image of a predetermined scale;

[0045] S112, based on each smoothed image, applying the Canny edge detection algorithm to obtain edge maps of different scales; performing weighted fusion on the edge maps of all scales to obtain a final edge map;

[0046] S113, based on the final edge map, using a quadtree segmentation algorithm to divide the original image into a predetermined number of sub-blocks;

[0047] S114: Calculate the square sum of the gradient values ​​and the entropy value of each sub-block, and calculate the significance score of each sub-block based on the square sum of the gradient values ​​and the entropy value.

[0048] In one embodiment of the present application, the original image I and the current GPU memory information M are read. gpu , use the multi-scale edge detection algorithm to extract the edge map E of image I, specifically: apply Gaussian blur to image I to obtain smooth images of multiple scales {I1, I2, ..., I n}; For each smoothed image I i Apply the Canny edge detection algorithm to obtain the edge map {E1, E2,..., E n}; weighted fusion of edge maps of all scales is performed to obtain the final edge map E. Based on the edge map E, the quadtree segmentation algorithm is used to divide the image I into N sub-blocks {B1, B2, ..., B N}: Initialize the root node of the quadtree, corresponding to the entire image I, and recursively segment each node: if the edge density of the area corresponding to the node is higher than the threshold T edge , then split the node into four child nodes; otherwise, mark the node as a leaf node; traverse all leaf nodes, each leaf node corresponds to a sub-block B i For each sub-block B i Calculate its significance score S i , specifically: Calculate sub-block B i The square sum of the gradient values ​​G i =sum(|▽B i |2 );Calculate sub-block B i The entropy value H i = -sum(p i * log(p i )), where p i is the normalized histogram of pixel values; calculate the significance score S i = w1 * G i + w2 * H i , where w1 and w2 are predefined weights.

[0049] Based on the saliency score of each sub-block and the current GPU memory information, a resolution reduction coefficient is adaptively assigned to each sub-block. Specifically, the optimization problem is constructed: the objective function is minΣ(S i * k i 2 ), the constraint is Σ(size(B i ) / k i 2 ) <= M gpu And k min <= k i <= k max , where k i For sub-block B i The resolution reduction factor, k min is the minimum resolution reduction factor, k max is the maximum resolution reduction factor, size(B i ) is sub-block B i The Lagrangian multiplier method is used to solve the optimization problem and construct the Lagrangian function L(k,λ) =Σ(S i * k i 2 ) +λ*(Σ(size(B i ) / k i 2 ) - M gpu ); calculate L(k, λ) for k i and the partial derivative of λ, and set it to 0, where λ represents the Lagrange multiplier; use Newton's method or gradient descent method to solve the equations and get the optimal k i value.

[0050] Construct an attention-based deformable convolutional network (ADCN), which includes L deformable convolutional layers and attention modules, and define the DC of each deformable convolutional layer l , including offset field prediction and convolution operation; define each attention module A l, including channel attention and spatial attention; the deformable convolutional layers and attention modules are stacked alternately to form a complete ADCN network. i , the resolution is reduced through the ADCN network, and the sub-block B i Input the first layer of the ADCN network. In each deformable convolution layer, the sampling position of the convolution kernel is adjusted according to the predicted offset field. In each attention module, the channel attention and spatial attention weights are calculated and applied. After L layers of processing, the sub-block B with reduced resolution is output. i '; Adjust B i ', so that it meets the predetermined resolution reduction factor k i : Calculate the target size: height' = height / k i , width' =width / k i , where height represents the original height of the image sub-block, width represents the original width of the image sub-block, height' represents the adjusted height, and width' represents the adjusted width; use bilinear interpolation to convert B i 'Adjust to the target size to obtain the final reduced resolution sub-block B i '.

[0051] Create a blank canvas with the same size as the original image after scaling by the average reduction factor, and i 'Place the sub-blocks at the corresponding position on the canvas at the relative position in the original image; smooth the sub-block boundary area, identify the boundary area of ​​adjacent sub-blocks, apply weighted averaging or alpha blending in the boundary area, and smooth the transition; output the final reduced-resolution complete image I'.

[0052] This embodiment achieves efficient and accurate image preprocessing by combining an adaptive block strategy, a deformable convolutional network (ADCN) based on an attention mechanism, and multi-scale feature extraction. Using multi-scale edge detection and quadtree segmentation algorithms, image blocks are adaptively divided according to the complexity of the image content, ensuring that the computational complexity is effectively reduced while retaining important details. By calculating the significance score of each sub-block and combining the GPU memory limit, the resolution reduction coefficient is dynamically allocated, which not only takes into account the hardware constraints, but also can flexibly adjust the processing strategy according to the importance of the image content. The introduced ADCN network, through deformable convolution and attention mechanisms, can adaptively adjust the receptive field and focus on important features, effectively retaining key texture and edge information while reducing the resolution, which is crucial for subsequent shadow removal. Compared with traditional fixed-size block or uniform downsampling methods, this adaptive strategy can better balance computational efficiency and image quality, especially when processing images containing complex shadows, it can maximize the retention of shadow edge and texture information in the process of reducing the resolution, providing better quality input for subsequent shadow removal. In addition, by rejoining the sub-blocks in a weighted average manner, artifacts that may be introduced by block processing are further eliminated, ensuring the continuity and naturalness of the image after reducing the resolution. This method, which comprehensively considers image content, hardware limitations, and processing efficiency, not only improves the efficiency of the entire shadow removal process, but also provides an image representation that is more conducive to shadow identification and removal in subsequent steps, thus laying a solid foundation for the entire shadow removal task.

[0053] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0054] S21, construct a multi-scale cascade diffusion model, define predetermined scale levels and corresponding diffusion models, and set noise scheduling for each diffusion model;

[0055] S22, performing multi-scale decomposition on the image after reducing the resolution, generating a multi-scale image set, and calculating the residual of each scale;

[0056] S23, based on the multi-scale image set and the residual of each scale, applying the multi-scale cascade diffusion model step by step to perform diffusion processing to obtain a shadow removal result;

[0057] S24. Based on the de-shadowing result, an adaptive noise adjustment mechanism is introduced to calculate the shadow intensity index of each scale, and the noise scheduling is dynamically adjusted according to the shadow intensity index to obtain the adjusted noise scheduling; based on the adjusted noise scheduling, steps S22 to S23 are re-executed to obtain the final de-shadowed image.

[0058] In one embodiment of the present application, a multi-scale cascade diffusion model (MCDM) is constructed, and L scale levels {l1, l2, ..., l L}, each level corresponds to a diffusion model D l , for each diffusion model D l Define the noise schedule: Set the noise range parameter β min and β max ; Define the number of time steps T; For each time step t, calculate the noise coefficient β t l =β min + t / T *(β max - Beta min ).

[0059] The reduced resolution image I' is decomposed into multiple scales, and the complete image I' is downsampled L times using the Gaussian pyramid to obtain a multi-scale image set {I'1, I'2, ..., I' L}, initialize I'1 = I', for i from 2 to L, execute: I' i = downsample(I' i-1 ), where downsample is a Gaussian downsampling operation; for l from 1 to L-1, the residual of each scale is calculated: R l = I' l - upsample(I' l+1 ), where upsample is an upsampling operation, usually using bilinear interpolation.

[0060] Apply the diffusion model to remove shadows step by step from coarse to fine. Apply the diffusion model to the coarsest scale L: Initialize the noise image x T L ~ N(0, I), denoising step by step from t=T to 1: x t-1 L = D L (x t L , I' L , t), where D L represents the diffusion model corresponding to the Lth scale level, and the shadow removal result O is obtained. L = x0 L ; Conditional diffusion for the intermediate scale l (l = L-1, ..., 2): Initialize the noise image: x T l ~ N(0, I), denoising step by step from t=T to 1: x t-1 l = D l (x tl , I' l , upsample(O l+1 ),t), and get the shadow removal result O l = x0 l ; Conditional diffusion of the finest scale 1 and fusion of residuals: Initialize the noise image: x T 1 ~ N(0, I), denoising step by step from t=T to 1: x t-1 1 = D1(x t 1 , I'1, upsample(O2), R1, t), where O2 represents the image result after de-shading at the second scale, and R1 represents the residual image at the first scale, and the final de-shading result O1 = x0 is obtained 1 .

[0061] Calculate the shadow intensity index S for each scale l l , for image I' l Perform expansion operation: I' ldilated = dilate(I' l ), where dilate() represents the dilation operation function; calculate the difference between the original image and the dilated image: diff = |I' l -I' ldilated |; Calculate the shadow intensity index: S l = mean(diff) / mean(I' l ), where mean( ) represents the average function; according to the shadow intensity index S l Dynamically adjust the noise schedule: For each time step t, update the noise coefficient: β t l =β min + S l * t / T * (β max -β min ), re-execute the above diffusion process using the updated noise schedule.

[0062] In another embodiment of the present application, Gaussian noise is added to the image after reducing the resolution, and the formula is:

[0063] I noisy =I'+N(0,σ 2 )

[0064] Among them, I noisy represents a noisy image, I' represents an image with reduced resolution, N(0,σ 2 ) has a mean of 0 and a variance of σ 2This can simulate the noise that the image may encounter in the actual environment and provide more information for the shadow removal model.

[0065] The noisy image is fed into a pre-trained diffusion-based shadow removal model. The model will perform an inverse diffusion process to estimate the noise residual, as follows:

[0066] R noise =D -1 I noisy

[0067] Among them, R noise represents the noise residual, D −1 Represents the inverse diffusion operation, which is an iterative process that gradually reduces the noise in an image while preserving the structure and details of the image.

[0068] The denoising step involves subtracting the estimated noise residual from the noisy image, specifically:

[0069] I denoised =I noisy -R noise

[0070] I denoised Represents the denoised image. This allows the pure version of the image to be restored in preparation for the subsequent shadow removal step.

[0071] The denoised image is used as the input for the next iteration and the above steps are repeated. This iterative process will continue until the preset number of denoising iterations N is reached. Each iteration will further reduce the noise in the image and improve the effect of removing shadows. The image after the Nth iteration will be regarded as the de-shadowed image H'.

[0072] This embodiment achieves efficient and accurate shadow removal by constructing and applying a multi-scale cascade diffusion model (MCDM). Specifically, the diffusion model is combined with multi-scale processing, and an adaptive noise adjustment mechanism is introduced. By constructing a diffusion model at multiple scale levels, local details and global structural information in the image can be captured simultaneously. This multi-scale strategy is particularly conducive to processing shadows of different sizes and complexities because it allows the model to learn and remove shadow features at different spatial scales. In the step-by-step processing from coarse to fine, the shadow removal results of the coarser scale provide valuable prior information for the finer scale. This hierarchical processing method can effectively process complex shadow structures while reducing the risk of over-smoothing or introducing artifacts. The introduced conditional diffusion process further enhances the model's ability to transfer and utilize information between different scales by using the shadow removal results of the previous level as conditional input, which is crucial to maintaining the overall consistency and detail accuracy of the image. Another key is the adaptive noise adjustment mechanism. By dynamically adjusting the noise scheduling, the model can adaptively adjust the intensity of the shadow removal according to the shadow intensity of different areas, which is conducive to processing images with large changes in shadow intensity, and can retain the details of weak shadow areas while removing obvious shadows, avoiding image distortion caused by excessive processing. Compared with traditional single-scale or fixed-parameter shadow removal methods, this multi-scale cascade diffusion model shows stronger adaptability and robustness, and can better handle various complex shadow scenes, including soft shadows, hard shadows, and transition areas with changing shadow intensity. In addition, due to the use of a diffusion model, it has the ability to handle uncertainty and can generate more natural and realistic de-shadowing results, effectively avoiding the common problems of over-sharpening or unnatural color transitions in traditional methods. This multi-scale cascade diffusion model method not only improves the quality and efficiency of shadow removal, but also enhances the algorithm's adaptability to different types of shadows, providing a powerful and flexible solution for high-quality image shadow removal.

[0073] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0074] S31. Construct a generator and a discriminator, where the generator uses a densely connected residual attention network and the discriminator uses a multi-scale convolution discriminator; construct a loss function, including content loss, perceptual loss, texture loss, and adversarial loss;

[0075] S32, constructing a self-supervised training pair based on the original image and the image after shadow removal; based on the self-supervised training pair, alternately optimizing the generator and the discriminator, minimizing the loss function, and obtaining a trained generator and discriminator;

[0076] S33. Input the de-shadowed image into the trained generator and adjust the size using Lanczos interpolation to obtain the image with restored resolution.

[0077] In one embodiment of the present application, a generator G is constructed using a densely connected residual attention network (DRAN): a dense residual attention block (DRAB) is defined: DRAB i (x) = F i (x) + A i (x) Θ x, where x represents the input image, F i (x) represents the feature transformation function of the i-th dense residual attention block, A i (x) represents the attention weight of the i-th dense residual attention block, Θ represents element-wise multiplication; construct DRAN: G(x) = x +Σ i w i * DRAB i (x); Initialize adaptive weight w i .

[0078] Construct the discriminator D and use a multi-scale convolution discriminator: define K sub-discriminators {D1, D 2, ..., D K}, each with a different receptive field, for each sub-discriminator, construct a convolution layer, a batch normalization layer and an activation function. Define the loss function: Define the content loss: L content = ||G(O1) - I||1, where G(O1) represents the image generated by the generator and I represents the original high-resolution image; the feature map φ is extracted using the pre-trained VGG network j , calculate the difference between the feature maps of the generated image and the original image at different layers as the perceptual loss: L perceptual = Σ j ||φ j (G(O1)) - φ j (I)||2 2 ; Calculate the Gram matrix of the feature map, and then calculate the difference between the Gram matrix of the generated image and the original image as the texture loss: L texture = ||Gram(G(O1)) - Gram(I)|| F 2 ;Define adversarial loss: L adv = Σ k log(1 - D k (G(O1))), where D k represents the kth sub-discriminator.

[0079] Use the original image I to construct a self-supervised training pair to generate a low-resolution image: Ilr = downsample(I), using the original image as the high-resolution target: I hr = I. Alternately optimize the generator G and the discriminator D: Update the generator G: min G L=λ1L content +λ2L perceptual +λ3L texture +λ4L adv , update the discriminator D: max D L adv , use Adam optimizer for parameter update.

[0080] Perform super-resolution reconstruction. Input the shadow removal result O1 into the trained generator G: sr = G(O1), using Lanczos interpolation to convert I sr Adjust to the same size as the original image I, specifically by calculating the scaling factor: scale = size(I) / size(I sr ), apply Lanczos interpolation: I hdr = lanczos interpolate (I sr , scale), output the final high-resolution de-shadowed image I hdr .

[0081] This embodiment achieves high-quality image super-resolution reconstruction by combining a densely connected residual attention network (DRAN), a multi-scale convolution discriminator, and a self-supervised learning strategy. The core of this step is that it can effectively restore and enhance the details of the image after removing the shadows while maintaining the naturalness and authenticity of the image. The design of DRAN as a generator enhances the network's ability to extract and utilize multi-level features through dense connections and residual learning. This structure is particularly conducive to recovering detail information that may be lost by the shadow removal process because it allows the network to learn and reconstruct image features at different levels of abstraction. The introduction of the residual attention mechanism further enhances the network's ability to focus on key features, allowing the reconstruction process to focus more accurately on those areas that have the greatest impact on image quality. The application of the multi-scale convolution discriminator provides a powerful adversarial learning framework that can evaluate the authenticity of the generated image from multiple receptive fields. This multi-scale strategy is particularly effective because it takes into account both local details and global structures, ensuring that the generated high-resolution images maintain a high degree of authenticity and consistency at all scales. The careful design of the loss function, including the comprehensive use of content loss, perceptual loss, texture loss, and adversarial loss, provides the network with multi-angle optimization goals. In particular, the introduction of perceptual loss and texture loss effectively improves the visual quality of the reconstructed image, making the restored details more natural and realistic. The application of self-supervised learning strategy solves the problem of traditional super-resolution methods' dependence on a large amount of high-quality paired data. By using the original image itself to construct training pairs, it not only reduces the demand for external data, but also better adapts to the characteristics of specific tasks. This is particularly important for image super-resolution after shadow removal, because it allows the model to learn detail reconstruction patterns specific to the shadow removal task. In practical applications, this method has shown significant advantages: it can effectively restore details that may be lost in the shadow removal process, especially at the shadow edge and in areas with complex textures; through adversarial learning and multi-scale discrimination, the generated high-resolution images have higher authenticity and naturalness, effectively avoiding the common over-smoothing or unrealistic texture problems of traditional methods; the self-supervised learning strategy makes this method highly adaptable and can dynamically adjust the reconstruction strategy for different types of images and shadow features. This self-supervised super-resolution reconstruction method based on generative adversarial networks not only improves the resolution and visual quality of images after shadow removal, but also ensures the efficiency and adaptability of the reconstruction process, providing a comprehensive and powerful solution for high-quality image shadow removal and detail restoration.

[0082] According to another aspect of the present application, step S3 may also be:

[0083] S3a, dividing the image after shadow removal into small blocks that do not overlap each other, and randomly performing data enhancement on the small blocks to obtain enhanced small blocks; wherein the data enhancement includes flipping and rotation;

[0084] S3b, building a super-resolution model, and based on the super-resolution model, reconstructing the enhanced small blocks into high-resolution small blocks;

[0085] S3c, perform data enhancement on the high-resolution small blocks to obtain the real high-score small blocks;

[0086] S3d, based on the high-resolution small blocks and the real high-score small blocks, the super-resolution model is trained through adversarial loss and perceptual loss to obtain a trained super-resolution model;

[0087] S3e, use the trained super-resolution model to super-resolve the non-overlapping small blocks to obtain super-resolved small blocks;

[0088] S3f, stitch all the super-resolved small blocks together to form a complete image with restored resolution.

[0089] In this embodiment, the generalization ability of the model is significantly improved by dividing the de-shadowed image into non-overlapping small blocks and performing data enhancement. Data enhancement, such as flipping and rotation, can simulate various transformations, making the model more robust and able to process images of various directions and angles. The construction and application of the super-resolution model not only improves the resolution of the image, but also retains the details after de-shadowing. This super-resolution reconstruction method can generate clearer and more delicate image details, which is particularly important for application scenarios that require high-quality image output. Then the authenticity of the small blocks is further enhanced, and the real high-resolution small blocks obtained by data enhancement can be used as a reference for training the super-resolution model, ensuring the high quality of the super-resolution model output. Through the combination of adversarial loss and perceptual loss, the model can better understand the image content and quality during the learning process. Adversarial loss promotes the realism of the generated image, while perceptual loss ensures the consistency of image quality with human visual perception. Finally, all the super-resolution small blocks are spliced ​​into a complete image, which not only provides a seamless high-resolution image, but also ensures the overall consistency and coherence of the image.

[0090] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0091] S41, divide the original image into grids, define the node features of the grids, and build a graph attention network with an attention mechanism based on the node features;

[0092] S42. Construct a comprehensive loss function including cross entropy loss and Dice coefficient loss. Based on the comprehensive loss function, use stochastic gradient descent with momentum to optimize the graph attention network to obtain a trained graph attention network.

[0093] S43, inputting the original image into the trained graph attention network, generating a segmentation mask of the shadow area, applying the conditional random field to post-process the segmentation mask to obtain an optimized segmentation mask;

[0094] S44. Based on the optimized segmentation mask, extract a set of image slices in the shadow area in the original image.

[0095] In one embodiment of the present application, the original image I is divided into M×N grids, each grid is used as a node of the graph; the node feature v is defined i , including the color, texture and position information of the grid; extract color features and calculate the average RGB value of the grid; extract texture features and calculate the LBP (local binary pattern) features of the grid; encode position information, using the relative coordinates of the grid (x / M, y / N); construct a K-layer graph attention network (GAT), and the attention mechanism of each layer is defined as: Calculate the attention coefficient α ij = softmax j (a T [Wh i || Wh j ]), where h i represents the feature representation of node i, h j Represents the feature representation of node j; Update node representation: h i ' = σ(Σ j α ij W j ), where W is the weight matrix, a is the attention vector, and σ is the activation function.

[0096] Train GAT to segment shadow areas. Calculate the cross entropy loss CE, calculate the Dice coefficient loss Dice, set the weight coefficient λ, and construct the loss function: L seg = CE(GAT(I), Y) + λDice(GAT(I), Y), where Y represents the true label image; use stochastic gradient descent with momentum to optimize GAT: initialize the learning rate η and momentum factor μ, and for each mini-batch data: calculate the gradient g = ▽L seg ; Update momentum v = μv – ηg; Update parameters θ = θ + v.

[0097] Input the original image I into the trained GAT: M shadow = GAT(I); Apply conditional random field (CRF) for post-processing to optimize the segmentation boundary: define unary potential energy and binary potential energy; use mean field inference algorithm to solve CRF; get the optimized segmentation mask: M shadowrefined = CRF(M shadow , I).

[0098] Based on the segmentation mask M shadowrefined , extract the image slice set P of the shadow area in the original image I. For the segmentation mask M shadowrefined Perform connected region analysis; for each connected region i: extract the corresponding image slice: p i = IΘM shadowrefinedi , where Θ represents pixel-by-pixel multiplication; the output image slice set P = {p1, p2, ..., p n}.

[0099] This embodiment achieves high-precision shadow area segmentation by applying a graph attention network (GAT) and a conditional random field (CRF). The core of this step is that it can make full use of the local and global context information of the image to accurately identify and segment the shadow area while taking into account the semantic characteristics of the shadow. The method of dividing the image into grids and constructing GAT allows the model to capture long-range dependencies between pixels, which is crucial for accurately identifying shadow boundaries and large-scale shadow areas. Each grid is a node in the graph, and its features contain color, texture, and position information. This comprehensive representation enables the model to consider multiple visual clues at the same time, improving the accuracy of segmentation. The multi-layer structure and attention mechanism of GAT further enhance the model's ability to learn complex shadow patterns, enabling it to adaptively focus on the most relevant features and areas. Compared with traditional convolutional neural networks, this graph-based representation and processing method shows obvious advantages in processing irregular structures and long-distance relationships, and is particularly suitable for processing complex shadow distributions. The design of the loss function combines cross entropy loss and Dice coefficient loss to provide a more comprehensive optimization target for the model. This combined loss not only focuses on the classification accuracy of each pixel, but also takes into account the quality of the overall segmentation, effectively improving the boundary accuracy and overall consistency of the shadow area. The use of stochastic gradient descent with momentum during training further improves the convergence speed and stability of the model. Post-processing of the conditional random field (CRF) effectively optimizes the boundaries of the segmentation results. CRF can consider the spatial relationship and color similarity between pixels, thereby producing more refined and coherent segmentation boundaries, which is particularly important for accurately locating the edges of shadows. In practical applications, this method has shown significant advantages: it can accurately identify and segment various complex shadow shapes, including soft shadows, hard shadows, and semi-transparent shadows; due to the consideration of global context information, it performs well when processing large-area shadows or multiple scattered shadows; by combining local features and global semantic information, it can effectively distinguish between real shadows and dark areas with similar colors, reducing misclassification; fine boundary processing ensures that the segmentation results are highly consistent with the actual shadow boundaries, providing precise guidance for subsequent shadow removal and image restoration. This semantic-aware shadow area segmentation method based on the graph attention network not only improves the accuracy of shadow recognition and segmentation, but also enhances the algorithm's adaptability to complex scenes, laying a solid foundation for high-quality image shadow removal.

[0100] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0101] S51. Construct a multimodal language vision model, including a visual encoder, a text encoder and a multimodal fusion module;

[0102] S52. Construct contrastive learning loss and use AdamW optimizer to train the multimodal language vision model to obtain a trained multimodal language vision model.

[0103] S53, inputting the image slice set and the predefined category text list of objects to be retained into the trained multimodal language vision model, calculating the similarity score; based on the similarity score, screening out the slice subset of the objects to be retained.

[0104] In one embodiment of the present application, a multimodal language visual model is constructed, specifically: a visual encoder is constructed using a VisionTransformer (ViT) structure, and image slices p i Split into sub-blocks of fixed size, perform linear projection and position encoding on each sub-block, use a multi-layer self-attention mechanism to process the sub-block sequence, and obtain the visual feature sequence f v ; Use the BERT structure to build a text encoder, segment the predefined text list C of the object category to be retained, convert the segmentation results into word embeddings, and use a multi-layer self-attention mechanism to process the word embedding sequence to obtain the text feature sequence f t ; Realize multimodal fusion and use cross attention mechanism to fuse f v and f t :Calculate the attention weights between visual features and text features, update the visual and text features based on the attention weights, and obtain the fusion feature f m .

[0105] Construct contrastive learning loss and calculate the similarity of positive sample pairs: sim(f m , f t +), calculate the similarity of negative sample pairs: sim(f m , f ti ), construct the loss function: L contrast = -log(exp(sim(f m , f t +) / τ) / Σ i exp(sim(f m , f ti ) / τ)), where τ is the temperature parameter; use AdamW optimizer to train the multimodal language vision model: set the learning rate and weight decay parameters, and for each training batch: calculate the loss L contrast , calculate the gradient and update the model parameters.

[0106] Identify and extract key objects. Input the image slice set P and the category text list C into the trained multimodal language vision model. i, extract visual features, extract text features from the category text list C, calculate the similarity matrix S of visual features and text features; filter key object slices according to the similarity score; set the similarity threshold threshold; for each slice p i , if max(S i ) > threshold, then add it to the slice subset P key Output the slice subset P of the object to be retained key .

[0107] This embodiment achieves highly accurate and flexible key object recognition by combining visual and language modalities. The core of this step is that it can use predefined text descriptions to guide the recognition and retention of specific objects in the image, which is crucial for retaining important content in the image while removing shadows. The design of the multimodal language vision model, including a visual encoder, a text encoder, and a multimodal fusion module, enables the model to process and understand image and text information simultaneously. The visual encoder adopts the Vision Transformer structure. This design based on the self-attention mechanism can better capture the global dependencies of the image than the traditional convolutional neural network, and is particularly conducive to identifying target objects in complex backgrounds. The text encoder adopts the BERT structure, which can effectively understand and represent complex language descriptions, so that the model can handle various forms of object descriptions, from simple category names to complex attribute descriptions. The multimodal fusion module realizes the deep fusion of visual and text features through the cross-attention mechanism, allowing the model to refer to the text description when understanding the visual content, and vice versa, thereby achieving more accurate object-text matching. The application of contrastive learning loss further enhances the model's ability to learn visual-language correspondences. By maximizing the similarity of positive pairs while minimizing the similarity of negative pairs, the model is able to learn more robust and discriminative feature representations. This learning strategy is particularly beneficial for fine-grained object recognition tasks, such as distinguishing objects that are similar but have different details. The use of the AdamW optimizer not only improves training efficiency, but also enhances the generalization ability of the model through the weight decay mechanism. In practical applications, it is able to flexibly identify and retain specific objects based on the text description provided by the user. This flexibility allows the method to adapt to a variety of shadow removal scenarios, from retaining important details to identifying specific scene elements. Due to the use of multimodal learning, it is able to handle complex semantic concepts, not limited to simple object categories, but also understand and recognize descriptions based on attributes, relationships, or context. This ability is particularly important when dealing with complex scenes, such as the need to retain specific objects such as "a red car partially obscured by a shadow" or "a person's face under a tree shade". By leveraging pre-trained vision and language models, it is possible to transfer knowledge learned on large-scale datasets, so that high-quality object recognition can be achieved even with limited training data. This is particularly beneficial for dealing with rare or domain-specific objects. This embodiment allows end-to-end training and reasoning, which not only improves the overall efficiency of the system, but also ensures consistency and complementarity between visual and language features. In the context of shadow removal, objects that may be affected by shadows but are critical to the image content can be accurately identified and retained. For example, when processing an image of a street scene containing shadows, key elements such as traffic signs, pedestrians, or vehicles in the shadows can be accurately identified and retained based on text instructions, even if these objects are partially obscured by shadows.This capability ensures that important information in the image is not lost or distorted during the shadow removal process, thereby maintaining the semantic integrity and visual quality of the image. This embodiment not only improves the ability to protect important objects during shadow removal, but also enhances the flexibility and adaptability of the entire system, providing a powerful and intelligent solution for high-quality, semantically aware image shadow removal.

[0108] like Figure 7 As shown, according to one aspect of the present application, step S6 is further:

[0109] S61, using the image after the resolution is restored as the background image, and using the slice subset as the foreground image set;

[0110] S62, defining a target region for each foreground image in the foreground image set, and constructing a multi-level Poisson fusion problem based on the target region and the background image;

[0111] S63. Based on the multi-level Poisson fusion problem, a multi-grid method is used to decompose and obtain problem representations at different levels of different hierarchical structures; Gauss-Seidel iteration method is applied to solve the problem representations at each level of each hierarchical structure to obtain a preliminary solution for each hierarchical structure; based on the preliminary solution, inter-layer interpolation and residual correction method are used to obtain a preliminary fused image;

[0112] S64. Based on the image after preliminary fusion, a local color migration algorithm is applied to perform adjustment to obtain an adjusted image; based on the adjusted image, a guided filter is used to perform smoothing to output a final high-resolution shadow-free image.

[0113] In one embodiment of the present application, the image components required for fusion are prepared: the high-resolution de-shadowed image I hdr As the background image B; the key object slice set P key As the foreground image set F; construct a multi-level Poisson fusion problem, for each foreground image f i ∈F, defines its target area Ω on the background image B i , determine the foreground image f i At the location in the background image B, create a binary mask M i , denoted by Ω i region; construct the Poisson equation, define the Laplace operator Δ, and calculate the guided vector field v = ▽f i - ▽B, in the target area Ω i Construct the equation: Δg = div v, where div represents the divergence operator; in ΨΩ i The boundary condition is defined as: g = B, where ΨΩ represents the boundary of the region Ω.

[0114] Solve the multi-level Poisson equation. Use the multi-grid method to construct a hierarchical structure of the equation system and create grids of different resolutions {G0, G1, ..., G L}, where G0 is the finest grid, and the Poisson equation is discretized on each grid. The Gauss-Seidel iteration method is applied to solve the linear equations at each layer, starting from the coarse grid and solving it layer by layer. i :Initialize solution u i , repeat until convergence or the maximum number of iterations is reached: Update u using Gauss-Seidel iteration i Using interlayer interpolation and residual correction, the solution process is completed from bottom to top, starting from the coarsest grid and moving upward layer by layer. i :solve u i Interpolate to a finer grid G i-1 , in the grid G i-1 The residual correction is performed on the image to obtain the final fusion result g.

[0115] Perform post-processing and color consistency adjustment. Apply the local color migration algorithm to adjust the color consistency of the fusion area: for each fusion area Ω i , calculate the fusion area Ω i The color statistical features μ of the corresponding area in the background B B , σ B ; Calculate the foreground f i The color statistics μ f , σ f ; for Ω i Each pixel x in is fused: final (x) = α(x)((g(x) -μ f ) * (σ B / σ f ) + μ B ) + (1-α(x))B(x), where α(x) is the weight function of spatial variation. Use guided filtering to smooth the fusion boundary: Construct a guided filter, use the background image B as the guide image, and smooth the fusion result I final Apply guided filtering to obtain the result after boundary smoothing and output the final high-quality shadow-free image I final .

[0116] This embodiment achieves high-quality image fusion and color consistency adjustment by applying multi-level Poisson equations and local color migration algorithms. The core of this step is that it can seamlessly merge the background image after shadow removal with the retained key object slices while maintaining the naturalness and visual coherence of the overall image. By converting the fusion task into a series of Poisson equations, it is possible to achieve image fusion across different scales while maintaining the continuity of the gradient field. It is particularly suitable for processing complex fusion scenes, such as combining the background after shadow removal with the foreground object that retains the original details. The use of Poisson equations ensures a smooth transition of the fusion result at the boundary, effectively avoiding the common seams and unnatural boundary problems of traditional fusion methods. The application of multi-grid methods improves the efficiency of solving Poisson equations. By constructing and solving equations on grids of different resolutions, it is possible to quickly converge to high-quality solutions, even when processing large-size images, and maintain high efficiency. This multi-scale strategy not only speeds up the calculation process, but also helps to capture image details at different scales, ensuring that the fusion results remain high quality at all scales. The use of Gauss-Seidel iteration further optimizes the solution process at each grid level, improving numerical stability and convergence speed. The introduced local color migration algorithm solves the problem of color inconsistency that may occur after fusion. By analyzing and adjusting the color statistical characteristics of the local area, the fusion area is ensured to be in harmony with the surrounding background. The use of spatially varying weight functions allows for smooth transitions in color adjustments, avoiding abrupt color changes. This is particularly important for shadow removal under complex lighting conditions, as it can compensate for local color distortion that may be caused by shadow removal. Post-processing with guided filtering optimizes the smoothness of the fusion boundary. By using the structural information of the background image to guide the filtering process, it is possible to reduce the noise and artifacts that may exist in the fusion area while maintaining edge sharpness. In practical applications, it can achieve high-quality fusion of the de-shadowed background and the retained key objects, achieving a natural transition effect even in complex scenes. For example, when processing a partially shadowed portrait, the de-shadowed background can be seamlessly combined with the detailed face area while maintaining the consistency of skin color and lighting. The multi-level processing strategy can simultaneously handle large-scale structural changes and tiny texture details, ensuring the visual quality of the fusion result at all scales. The application of local color migration effectively solves the color inconsistency problem that may occur after shadow removal, making the final image more visually natural and harmonious. The whole process is computationally efficient and can be processed in a reasonable time even for high-resolution images. In general, this seamless fusion method based on Poisson image editing not only improves the overall quality of the image after shadow removal, but also ensures the detail preservation of key objects and the naturalness of the overall visual effect, providing a comprehensive and powerful solution for high-quality image shadow removal and post-processing.

[0117] According to another aspect of the present application, cascade diffusion model shadow removal may also be performed, specifically:

[0118] S21, sending the complete down-resolution image to the first-level diffusion model IR-SDE-1, and outputting a rough shadow-removed image;

[0119] S22, introduce the second-level diffusion model IR-SDE-2, take the rough shadow removal image as input, and output the fine shadow removal image. The second-level diffusion model IR-SDE-2 adopts the conditional diffusion model, takes the complete down-resolution image as condition, and corrects the artifacts and other defects output by IR-SDE-1;

[0120] S23. Introduce the third-level diffusion model IR-SDE-3. Based on the fine shadow removal image, it focuses on restoring the texture details of the shadow area and outputs a high-fidelity shadow removal image.

[0121] The denoising process of the three-level diffusion model adopts an adaptive step-size sampling method, dynamically adjusting the intensity of each denoising according to the noise level to achieve a balance between denoising speed and effect.

[0122] This embodiment provides a rough de-shadowed image by initially removing shadows; then further refines the de-shadowing effect to correct artifacts and defects that may be generated by the first-level model; then focuses on restoring the texture details of the shadow area and outputs a high-fidelity de-shadowed image. The high-fidelity de-shadowed image finally output not only removes shadows, but also retains the details and texture of the image, improving the overall quality of the image.

[0123] According to another aspect of the present application, step S4 further includes performing superpixel segmentation and fusion on the original image, specifically:

[0124] S41. Use the SLIC (Simple Linear Iterative Clustering) algorithm to perform superpixel segmentation on the original image to obtain a superpixel set;

[0125] S42, for each superpixel, extract its multi-scale features and determine whether it contains a shadow area;

[0126] S421, using multiple convolution kernels of fixed scales (such as 33, 55, 7*7) to extract texture features from superpixels;

[0127] S422, sending the texture features to a pre-trained shadow area detection model (such as ResNet), determining whether the superpixel contains a shadow, and outputting the probability of containing a shadow;

[0128] S43, filtering and fusing superpixels based on shadow probability;

[0129] S431, eliminating superpixels with shadow probability less than a threshold, and obtaining a candidate shadow superpixel set;

[0130] S432, performing region growing fusion on adjacent superpixels in the candidate shadow superpixel set, and the fusion criterion is:

[0131] If the shadow probability difference between adjacent superpixels is less than the threshold t1 and the experimental color space distance is less than the threshold t2, fusion is performed;

[0132] S433, obtaining a fused superpixel set of shadow areas;

[0133] S44. Map the fused shadow area superpixel set back to the original image to obtain a detailed shadow area mask image.

[0134] This embodiment can segment the image into superpixels with similar features through the SLIC algorithm. These superpixels are more uniform in shape and size, which helps to improve the efficiency and accuracy of subsequent processing. Through superpixel segmentation, the number of pixels to be processed can be reduced, thereby reducing the computational complexity; using convolution kernels of different scales to extract the texture features of superpixels can capture image details at different scales. This multi-scale feature extraction method can more comprehensively describe the texture information of superpixels and improve the accuracy of shadow detection. The extracted texture features are sent to the pre-trained shadow area detection model (such as ResNet), and the powerful feature extraction and classification capabilities of the deep learning model can be used to accurately determine whether the superpixel contains a shadow. By outputting the shadow probability, the shadow degree of each superpixel can be quantified, providing a basis for subsequent processing. By setting a threshold, superpixels with a low shadow probability are eliminated, and a set of candidate shadow superpixels is retained. Then, the adjacent superpixels are regionally grown and fused to further improve the coherence and accuracy of the shadow area. This method can effectively reduce false detections and missed detections and improve the accuracy of shadow detection. The fused shadow area superpixel set is mapped back to the original image to obtain a detailed shadow area mask map. This mask map can accurately mark the shadow area in the image, providing a reliable basis for subsequent shadow removal and image enhancement.

[0135] According to another aspect of the present application, step S5 may also perform key object extraction guided by knowledge distillation, specifically:

[0136] S51, build and train a multimodal language vision-teacher model;

[0137] S511. Use large-scale, high-quality image-text pair datasets (such as COCO, Visual Genome, etc.) to train multimodal language vision-teacher models;

[0138] S512, Multimodal Language Vision-Teacher Model learns multimodal joint representation of image and text, which can accurately match image content and natural language description;

[0139] S52, extracting the high-resolution feature map learned by the super-resolution model;

[0140] S53, feed the high-resolution feature map into the multimodal language vision-teacher model, output the classification result, and predict all key object categories present in the image;

[0141] S54, based on all key object categories, using the knowledge distillation of the multimodal language vision-teacher model to transfer to the multimodal language vision-student model;

[0142] S541, fixing the parameters of the multimodal language vision-teacher model, and using its prediction results to guide the training of the multimodal language vision-student model;

[0143] S542, Multimodal Language Vision-Student Model has the same structure as Multimodal Language Vision-Teacher Model but is smaller in scale, extracting multi-scale feature maps on high-fidelity shadow-removed images;

[0144] S543, taking all key object categories as soft labels, applying KL divergence loss to the output of the multimodal language vision-student model, forcing it to learn the knowledge of the multimodal language vision-teacher model;

[0145] S544. Simultaneously, all key object categories are used as hard labels, and cross entropy loss is applied to the output of the multimodal language vision-student model to improve its classification accuracy;

[0146] S55. Use the trained multimodal language vision-student model to reason about the high-resolution shadow-free image and output the key object categories contained therein;

[0147] S56: Based on the key object category, crop the corresponding key object image block from the original image.

[0148] According to another aspect of the present application, step S6 may also be based on Poisson fusion images, specifically:

[0149] S61, using a shadow mask, replacing the shadow area of ​​the high-resolution shadow-free image with the original image block of the corresponding area in the original image, to obtain a first fused image;

[0150] S62, replacing the corresponding area in the first fused image with the key object image block to obtain a second fused image;

[0151] S63, in order to eliminate the color difference and illumination inconsistency that may exist in the second fused image, adjust it using the Poisson image fusion technology based on the Poisson equation to obtain a final shadow-free high-quality image;

[0152] S631, extracting the Laplacian pyramid representation of the second fused image, separating the high-frequency details from the low-frequency basis;

[0153] S632. For each layer of the Laplace pyramid, use the Poisson equation to adjust the areas with inconsistent color difference and illumination to the average level of the surrounding areas, and then use the Gaussian pyramid to fuse them;

[0154] S633, at the top layer of the Laplacian pyramid, use global color mapping to adjust the overall tone to the average tone of the original image;

[0155] S634, reconstruct layer by layer from the top layer to the bottom layer to obtain the final output with consistent color and lighting and preserved details.

[0156] This application introduces attention mechanism and deformable convolution to achieve adaptive and detail-preserving resolution reduction; uses a cascaded conditional diffusion model to remove shadows, correct defects, and restore textures in stages; combines self-supervised learning to train super-resolution models without additional high-resolution samples; extracts fine shadow area masks based on superpixel segmentation and region fusion; uses teacher-student knowledge distillation to achieve accurate extraction of key objects with a lighter model; and adopts Poisson image fusion technology based on the Poisson equation to eliminate chromatic aberration and lighting inconsistency in image stitching.

[0157] According to one aspect of the present application, a shadow removal system based on diffusion, segmentation and super-resolution models includes: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used by the processor to execute the shadow removal method based on diffusion, segmentation and super-resolution models described in any of the above embodiments.

[0158] This application achieves high-quality and efficient image shadow removal and detail retention by combining adaptive resolution adjustment, multi-scale cascade diffusion model, generative adversarial network super-resolution reconstruction, graph attention network segmentation, multimodal Transformer object recognition and Poisson image editing. Comprehensive consideration of all key links in the shadow removal process, from image preprocessing to final detail recovery and image fusion, each step introduces targeted technical innovations to address the challenges faced by traditional methods. The adaptive resolution adjustment and feature enhancement steps realize intelligent analysis and processing of image content by combining edge detection, quadtree segmentation and attention-based deformable convolutional network. Not only does it improve the efficiency of subsequent processing, but it also effectively retains edge and texture information that is critical to shadow removal. The application of the multi-scale cascade diffusion model effectively handles shadows of different sizes and complexities by applying the diffusion process step by step at multiple scales. The introduction of the adaptive noise adjustment mechanism further enhances the model's adaptability to different shadow intensities, and is able to remove obvious shadows while retaining details in weak shadow areas. The super-resolution reconstruction step of the generative adversarial network combines a densely connected residual attention network and a multi-scale convolutional discriminator, which not only restores the high-resolution details of the image, but also ensures the authenticity and naturalness of the reconstruction results through adversarial learning. This is particularly important for repairing details that may be lost during shadow removal. The application of the graph attention network performs well in the task of shadow area segmentation. By capturing the long-range dependencies between pixels, it can more accurately identify complex shadow boundaries and large-scale shadow areas. The post-processing of the conditional random field further optimizes the segmentation results and improves the accuracy of the boundaries. The application of the multimodal Transformer enables the system to accurately identify and retain key objects based on text descriptions. This feature plays a key role in protecting the semantic integrity of the image and ensures that important image content is not lost while removing shadows. The image fusion technology based on the Poisson equation achieves seamless fusion of the de-shadowed background and the retained object through multi-level solution and local color migration, while maintaining the consistency and natural transition of the overall color. Compared with traditional shadow removal methods, this application can more comprehensively and finely handle various complex shadow scenes, including soft and hard shadows, large-area shadows, and subtle shadow details. Through multi-scale and adaptive processing strategies, while maintaining overall efficiency, the processing strategy can be flexibly adjusted for different areas and features, avoiding the problem of over-removal or under-processing. Through the innovative combination of deep learning and traditional image processing technology, while improving the quality of shadow removal, it also improves the authenticity and naturalness of the results. The modular design and end-to-end processing flow of this application not only improves the overall efficiency, but also enhances the scalability and adaptability of the system, enabling it to easily cope with various complex practical application scenarios.

[0159] In general, this application not only significantly improves the quality and efficiency of shadow removal, but also provides a powerful and flexible technical framework for the field of image processing, opening up new possibilities for high-quality image restoration and enhancement.

[0160] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A shadow removal method based on diffusion, segmentation and super-resolution model, characterized in that: The steps include: S1, read the original image, perform adaptive resolution adjustment and feature enhancement on the original image, and obtain an image with reduced resolution; S2, inputting the image with reduced resolution into the pre-trained multi-scale cascade diffusion model to remove shadows and obtain a de-shadowed image; S3, based on the pre-trained super-resolution model, restoring the resolution of the image after the shadow removal to the resolution of the original image, to obtain an image with restored resolution; S4, inputting the original image into a pre-trained image segmentation model to obtain a segmentation mask of the shadow, and based on the segmentation mask, segmenting a corresponding set of image slices in the original image; S5, inputting the image slice set and the predefined category text list of objects to be retained into the pre-trained multimodal language vision model to obtain a slice subset of the objects to be retained; S6, fusing the image after the resolution is restored and the slice subset to obtain a final high-resolution shadow-free image; Step S2 is further as follows: S21, construct a multi-scale cascade diffusion model, define predetermined scale levels and corresponding diffusion models, and set noise scheduling for each diffusion model; S22, performing multi-scale decomposition on the image after reducing the resolution, generating a multi-scale image set, and calculating the residual of each scale; S23, based on the multi-scale image set and the residual of each scale, applying the multi-scale cascade diffusion model step by step to perform diffusion processing to obtain a shadow removal result; S24. Based on the de-shadowing result, an adaptive noise adjustment mechanism is introduced to calculate the shadow intensity index of each scale, and the noise scheduling is dynamically adjusted according to the shadow intensity index to obtain the adjusted noise scheduling; based on the adjusted noise scheduling, steps S22 to S23 are re-executed to obtain the final de-shadowed image.

2. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 1, characterized in that: Step S1 is further as follows: S11, reading the original image and current GPU memory information, using an adaptive block strategy to divide the original image into a predetermined number of sub-blocks of unequal size, and calculating the significance score of each sub-block; S12, adaptively assigning a resolution reduction coefficient to each sub-block based on the significance score of each sub-block and current GPU memory information; S13, based on the resolution reduction coefficient corresponding to each sub-block, using a deformable convolutional network based on an attention mechanism to reduce the resolution of each sub-block to obtain a sub-block with reduced resolution; S14, rejoining the sub-blocks with reduced resolution to obtain a complete image with reduced resolution.

3. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 2, characterized in that: Step S3 is further as follows: S31. Construct a generator and a discriminator, where the generator uses a densely connected residual attention network and the discriminator uses a multi-scale convolution discriminator; construct a loss function, including content loss, perceptual loss, texture loss, and adversarial loss; S32, constructing a self-supervised training pair based on the original image and the image after shadow removal; based on the self-supervised training pair, alternately optimizing the generator and the discriminator, minimizing the loss function, and obtaining a trained generator and discriminator; S33. Input the de-shadowed image into the trained generator and adjust the size using Lanczos interpolation to obtain an image with restored resolution.

4. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 3, characterized in that: Step S4 is further as follows: S41, divide the original image into grids, define the node features of the grids, and build a graph attention network with an attention mechanism based on the node features; S42. Construct a comprehensive loss function including cross entropy loss and Dice coefficient loss. Based on the comprehensive loss function, use stochastic gradient descent with momentum to optimize the graph attention network to obtain a trained graph attention network. S43, inputting the original image into the trained graph attention network, generating a segmentation mask of the shadow area, applying the conditional random field to post-process the segmentation mask to obtain an optimized segmentation mask; S44. Based on the optimized segmentation mask, extract a set of image slices in the shadow area in the original image.

5. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 4, characterized in that: Step S5 is further as follows: S51. Construct a multimodal language vision model, including a visual encoder, a text encoder and a multimodal fusion module; S52. Construct contrastive learning loss and use AdamW optimizer to train the multimodal language vision model to obtain a trained multimodal language vision model. S53, inputting the image slice set and the predefined category text list of objects to be retained into the trained multimodal language vision model, and calculating a similarity score; Based on the similarity score, a subset of slices of the object to be retained is selected.

6. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 5, characterized in that: Step S6 is further as follows: S61, using the image after the resolution is restored as the background image, and using the slice subset as the foreground image set; S62, defining a target region for each foreground image in the foreground image set, and constructing a multi-level Poisson fusion problem based on the target region and the background image; S63. Based on the multi-level Poisson fusion problem, the multi-grid method is used to decompose and obtain the problem representations of each level in different hierarchical structures; the Gauss-Seidel iteration method is applied to solve the problem representations of each level in each hierarchical structure to obtain the preliminary solution of each hierarchical structure; Based on the preliminary solution, the interlayer interpolation and residual correction method are used to obtain the preliminary fused image; S64, applying a local color migration algorithm to perform adjustment based on the initially fused image to obtain an adjusted image; Based on the adjusted image, guided filtering is used for smoothing to output the final high-resolution shadow-free image.

7. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 6, characterized in that: Step S3 may also be: S3a, dividing the image after shadow removal into small blocks that do not overlap each other, and randomly performing data enhancement on the small blocks to obtain enhanced small blocks; Data enhancement includes flipping and rotation; S3b, building a super-resolution model, and based on the super-resolution model, reconstructing the enhanced small blocks into high-resolution small blocks; S3c, perform data enhancement on the high-resolution small blocks to obtain the real high-score small blocks; S3d, based on the high-resolution small blocks and the real high-score small blocks, the super-resolution model is trained through adversarial loss and perceptual loss to obtain a trained super-resolution model; S3e, use the trained super-resolution model to super-resolve the non-overlapping small blocks to obtain super-resolved small blocks; S3f, stitch all the super-resolved small blocks together to form a complete image with restored resolution.

8. The shadow removal method based on diffusion, segmentation and super-resolution model according to claim 6, characterized in that: Step S11 is further as follows: S111, reading the original image and current GPU memory information, and applying Gaussian blur to the original image to obtain a smooth image of a predetermined scale; S112, based on each smoothed image, applying the Canny edge detection algorithm to obtain edge maps of different scales; performing weighted fusion on the edge maps of all scales to obtain a final edge map; S113, based on the final edge map, using a quadtree segmentation algorithm to divide the original image into a predetermined number of sub-blocks; S114: Calculate the square sum of the gradient values ​​and the entropy value of each sub-block, and calculate the significance score of each sub-block based on the square sum of the gradient values ​​and the entropy value.

9. A shadow removal system based on diffusion, segmentation and super-resolution models, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the shadow removal method based on diffusion, segmentation and super-resolution model as described in any one of claims 1 to 8.

Images