Plug-and-play image shadow edge removal method, device, equipment and medium

Abstract

The present application relates to a plug-and-play image shadow edge removal method, apparatus, device, and medium. A brightness change coordinate system is constructed based on the brightness changes of pixels in a shadow image and a shadow-free image, and a Gaussian mixture model is used for modeling. The maximum a posteriori estimation problem for the position and belonging region of each point data in the brightness change coordinate system is used as a problem model. An iterative algorithm is used to solve the problem. During each iteration, the belonging region division is updated, the distribution parameters representing each updated belonging region are updated, and the position of each point data in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image. The edge region in the image is then optimized and updated to obtain the current iteration result, until a preset iteration termination condition is met to obtain a shadow-free image. This method can enhance the performance of shadow edge region recovery and obtain an image with better shadow removal effect.

Description

Technical Field

[0001] The present application relates to the field of optical image intelligent enhancement and computer intelligence technology, and in particular to a plug-and-play image shadow edge removal method, device, equipment and medium. Background Art

[0002] Image shadow edge removal is an open problem, whose main goal is to enhance the visibility of the image in the shadow edge area and accurately estimate the brightness change of each pixel in the shadow edge area to pursue consistent illumination distribution between the shadow area, shadow edge and non-shadow area.

[0003] In recent years, with the rapid development of artificial intelligence, many basic frameworks such as Generative Adversarial Network (GAN), Transformer, and Diffusion have emerged, and a series of image deshadowing algorithms have been born based on these frameworks. These algorithms are committed to restoring the image content in shadow areas and shadow edge areas, and have achieved many impressive results. However, the shadow-free images they restore have very obvious artifacts in the shadow edge areas. These residual artifacts will have a negative impact on downstream high-level visual tasks and cannot meet the image quality requirements of mobile phone photography, map surveying and mapping, autonomous driving and other fields. Based on the above reasons, it is necessary to design a shadow edge optimizer for the existing image deshadowing algorithm to enhance the restoration performance in the shadow edge area. Summary of the Invention

[0004] Based on this, it is necessary to provide a plug-and-play image shadow edge removal method, device, equipment and medium that can enhance the shadow edge area restoration performance to address the above technical problems.

[0005] A plug-and-play method for removing shadow edges of an image, the method comprising:

[0006] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0007] A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed.

[0008] After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0009] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

[0010] In one embodiment, based on a degradation model of the image shadow removal problem, the brightness change coordinate system is constructed according to a mapping relationship between pixels at the same position between the shadow image and the shadow-free image;

[0011] In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration;

[0012] The coordinates of each point data in the brightness change coordinate system are pixel values ​​at the same position in the shadow image and the shadow-free image.

[0013] In one embodiment, the belonging regions include an inner belonging region representing the interior of the image shadow, an outer belonging region representing the exterior of the image shadow, and an edge belonging region representing the edge region of the image shadow.

[0014] In one embodiment, the problem model is expressed as:

[0015]

[0016] In the above formula, represents the coordinate set of the point data in the brightness change coordinate system, The latent variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

[0017] In one embodiment, during each iteration:

[0018] The shadow-free image with optimized edge regions obtained in the previous iteration is used as input;

[0019] When updating the division of the belonging region, updating the point data coordinates in the brightness change coordinate system according to the optimized shadow-free image and the shadow image of the edge region, calculating the intermediate posterior probability of each updated point data coordinate, and updating the division of the belonging region;

[0020] When updating the distribution parameters representing each subordinate region, the updated distribution parameters of each subordinate region are calculated according to the intermediate posterior probability;

[0021] When updating the position of each point data in the brightness change coordinate system, Markov chain Monte Carlo sampling is used to estimate the position of the point data in the edge membership region so that it satisfies the distribution parameters of the corresponding updated membership region, thereby obtaining a reference image with distribution matching;

[0022] When optimizing and updating the edge area, a diffusion model and a lightweight network are used to estimate the shadow-free image and weight matrix of the edge area after optimization obtained in the previous iteration under the guidance of the reference image. The estimated shadow-free image and shadow image of the edge area after optimization obtained in the previous iteration are fused by using the weight matrix to obtain the result of the current iteration, i.e., the shadow-free image of the edge area after optimization.

[0023] In one embodiment, an expectation maximization algorithm is used when updating the division of the membership regions and updating the distribution parameters characterizing each membership region.

[0024] In one embodiment, the diffusion model is a pre-trained model.

[0025] The present application also provides a plug-and-play image shadow edge removal device, the device comprising:

[0026] An image acquisition module, configured to acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0027] a problem model construction module, configured to construct a brightness change coordinate system based on brightness changes at the same position in the shadow image and the corresponding shadow-free image, and to model the point data in the brightness change coordinate system using a Gaussian mixture model, thereby converting the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging region of each point data in the brightness change coordinate system, and constructing a corresponding problem model;

[0028] a problem model solving module, configured to introduce the shadow image and the shadow-free image into the problem model and solve the problem using an iterative algorithm, wherein during each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain a current iteration result, i.e., a shadow-free image after optimizing the edge region, and the shadow-free image after optimizing the edge region is used as input for the next iteration;

[0029] The iteration termination judgment module is used to meet the preset iteration termination condition, and the shadow-free image after the edge area optimization currently obtained is the final output result to achieve image shadow edge removal.

[0030] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0032] A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed.

[0033] After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0034] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0036] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0037] A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed.

[0038] After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0039] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

[0040] The plug-and-play image shadow edge removal method, device, equipment, and medium described above construct a brightness change coordinate system based on brightness changes at the same position in the shadow image and the corresponding shadow-free image, and model it using a Gaussian mixture model. The image shadow removal problem is then transformed into a maximum a posteriori estimation problem for the position and belonging region of each point data in the brightness change coordinate system, and a problem model is constructed. The shadow image and the corresponding shadow-free image are introduced into the problem model and solved using an iterative algorithm. In each iteration, the division of the belonging regions is updated, the distribution parameters representing each belonging region are updated, and the position of each point data in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image. The edge region in the reference image is then optimized and updated to obtain the current iteration result, i.e., a shadow-free image with optimized edge regions. The shadow-free image with optimized edge regions is then used as the input for the next iteration until a preset iteration termination condition is met. The currently obtained shadow-free image with optimized edge regions is then the shadow-free image with shadow edge removal. This method can enhance the performance of shadow edge region restoration, thereby obtaining an image with better shadow removal effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a flow chart of a plug-and-play method for removing shadow edges of an image in one embodiment;

[0042] Figure 2A schematic diagram of region division and GMM modeling based on the LV coordinate system in one embodiment;

[0043] Figure 3 Schematic diagram of the overall framework of a plug-and-play shadow edge removal method in one embodiment;

[0044] Figure 4 This is an example diagram of the visualization results of this method and multiple existing methods on the shadow removal dataset in an experiment;

[0045] Figure 5 This is an example diagram of the visualization results of the intermediate iterations in an experiment when using this method;

[0046] Figure 6 Schematic diagram of the distribution of more shadow / no-shadow image pairs in the LV coordinate system and their respective one-dimensional distributions in an experiment;

[0047] Figure 7 A structural block diagram of a plug-and-play image shadow edge removal device in one embodiment;

[0048] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] Shadow edge removal currently faces two major difficulties. First, the edge area is difficult to accurately define. Simply using graphics erosion and dilation operations may result in incorrect area division. Second, the brightness changes in the edge area are very complex. The restoration result is prone to produce serious edge artifacts. To address these two problems, in this application, Figure 1 As shown in the figure, a plug-and-play image shadow removal method is proposed, which aims to optimize the existing image shadow removal technology. The specific steps include:

[0051] Step S100 , obtaining a shadow image and a shadow-free image after preliminary restoration of the shadow image.

[0052] Step S110, constructing a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and using a Gaussian mixture model to model the point data in the brightness change coordinate system, converting the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and constructing a corresponding problem model.

[0053] In step S120, after the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated in turn, the distribution parameters representing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, that is, the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0054] Step S130 , until a preset iteration termination condition is met, the shadow-free image obtained after the edge area is optimized is the final output result, so as to achieve image shadow edge removal.

[0055] The above method is based on the existing image deshadowing technology and combines the physical model of pixel brightness changes before and after image deshadowing. It focuses on the research of plug-and-play real-time intelligent image shadow edge removal technology to achieve high-quality and efficient shadow edge removal of images, providing more high-quality samples for downstream advanced computer vision tasks such as image segmentation, target detection and recognition, and target tracking.

[0056] In step S100, the initially restored shadow-free image is obtained based on the shadow image using existing shadow removal methods, including but not limited to methods based on illumination models, image filtering, threshold segmentation, and deep learning. In this case, the resulting shadow-free image often exhibits very obvious artifacts in the shadow edge region.

[0057] In step S110, based on the degradation model of the image shadow removal problem, a brightness change coordinate system is constructed according to the mapping relationship between the pixels at the same position between the shadow image and the shadow-free image. The degradation model of the image shadow removal problem is usually modeled as follows:

[0058] I sf =α⊙I r +(1-α)⊙I s (1)

[0059] In formula (1), I s represents the shadow image, I sf represents the restored shadow-free image, I r represents the image after brightness adjustment, α represents the weight matrix, and ⊙ represents the element-by-element multiplication operation.

[0060] From formula (1), we can see that for I s A pixel value at any position on There is a corresponding mapping function to map it to Isf The pixel value at the same position on Based on this, a brightness variation (LV) coordinate system is established in this method to describe I s and I sf Mapping relationship at the same position. In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration. That is, the coordinates of each point data in the brightness change coordinate system are the pixel values ​​at the same position in the shadow image and the shadow-free image. Therefore, the pixel values ​​at the same position before and after the image is deshadowed can be represented as a point in the LV coordinate system. Noted as s n (x n ,y n ).

[0061] like Figure 2 As shown in (a)-(c), assuming that the shadow image contains N pixels, the mapping relationship between it and its paired shadow-free image can be expressed as a set of N coordinate points in the LV coordinate system.

[0062] Furthermore, the distribution of N coordinate points of the paired shadow and shadow-free images in the LV coordinate system is Use Gaussian mixture model (GMM) for modeling, such as Figure 2 (d) is shown in the figure, and its mathematical expression is as follows:

[0063]

[0064] in,

[0065] In formula (2), K represents the number of Gaussian distributions, ω k 、μ k and ∑ k represent the weight, mean, and variance of the k-th Gaussian distribution respectively.

[0066] like Figure 2 As shown in , if the points belonging to different areas in the LV coordinate system are represented by different colors, it can be found that the distribution of points in different areas can be approximated by a single or multiple Gaussian distributions (ellipses). The distribution of coordinate points in the shadow inner area and the shadow outer area can be fitted by only one Gaussian distribution. The distribution of coordinate points in the shadow edge area requires multiple Gaussian distributions to fit due to the complex brightness changes. Figure 2As shown in Figure (e), the coordinate points of the outer and edge regions are marked in red, blue, and yellow, respectively. It can be seen that the coordinate points of the inner and outer shadow regions can be fitted with a single Gaussian distribution (corresponding to the red and blue circles), while the points of the shadow edge region can be fitted with three Gaussian distributions (corresponding to the yellow circle). Therefore, by adjusting the inner, outer, and edge regions of the shadow in the LV coordinate system to make their distribution more reasonable, an image with better shadow removal can be obtained.

[0067] Furthermore, based on the above phenomenon, the coordinate point The corresponding area category can be equivalently represented by the Gaussian distribution to which it belongs. In order to facilitate the formulation, in this method, the latent variable To describe The region to which it belongs, express belongs to the kth Gaussian distribution.

[0068] In this embodiment, the belonging regions include an inner belonging region indicating the interior of the image shadow, an outer belonging region indicating the exterior of the image shadow, and an edge belonging region indicating the edge region of the image shadow.

[0069] In the actual shadow removal work, since the shadow-free image I sf The pixel values ​​and region divisions are usually unknown, which leads to the coordinate point set and the set of latent variables According to the definition of LV coordinate system, the image shadow removal problem can be transformed into the LV coordinate system. The maximum a posteriori estimation (MAP) problem is the problem model to be solved, and the mathematical expression is as follows:

[0070]

[0071] In formula (3), represents the coordinate set of the point data in the brightness change coordinate system, The latent variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

[0072] Furthermore, according to the GMM model, Multiple Gaussian distributions can be used for fitting. The specific expressions are as follows:

[0073]

[0074] Therefore, in the above MAP problem, we need to The distribution parameters of And its corresponding regional category latent variable Make an estimate.

[0075] In step S120, an iterative solution method is proposed to solve the problem model represented by formula (3). In each iteration process: the shadow-free image of the edge area after optimization obtained in the previous iteration is used as input. When the division of the subordinate area is updated, the point data coordinates in the brightness change coordinate system are updated based on the shadow-free image after optimization of the edge area and the shadow-free image, the intermediate posterior probability of each updated point data coordinate is calculated, and the division of the subordinate area is updated. When updating the distribution parameters characterizing each subordinate region, the updated distribution parameters of each subordinate region are calculated based on the intermediate posterior probability. Then, when updating the position of each point data in the brightness change coordinate system, Markov chain Monte Carlo sampling is used to estimate the position of the middle point data in the edge subordinate region so that it satisfies the distribution parameters of the corresponding updated subordinate region, and a reference image with distribution matching is obtained. When optimizing and updating the edge region, the diffusion model is used to estimate the optimized shadow-free image and weight matrix of the edge region obtained in the previous iteration under the guidance of the reference image. The estimated shadow-free image of the edge region obtained in the previous iteration and the shadow image are fused by using the weight matrix to obtain the result of the current iteration, i.e., the optimized shadow-free image of the edge region.

[0076] In this embodiment, an iterative algorithm solution method for EMSD is proposed, in which each iteration process is divided into four steps: E, M, S, and D, which respectively represent updating the division of the subordinate areas, namely step E, updating the distribution parameters characterizing each subordinate area, namely step M, updating the data position of each point in the brightness change coordinate system, namely step S, and optimizing and updating the edge area, namely step D.

[0077] In this embodiment, when updating the division of the membership regions and updating the distribution parameters characterizing each membership region, an Expectation Maximization (EM) algorithm is used.

[0078] Specifically, in step E, the latent variable set The posterior probability is used as the indicator of regional division. In the tth round of iteration, when the coordinate point set at this time is given The distribution parameters of When, for A point in Its corresponding hidden variable The posterior probability can be expressed as follows:

[0079]

[0080] In formula (5), Indicates the coordinate point at the tth iteration The probability of belonging to the kth Gaussian distribution.

[0081] Furthermore, after calculating the posterior probability of the latent variable corresponding to each coordinate point, it can be used as the basis for updating the region division result. At the same time, in this method, the Gaussian distribution describing the outer area of ​​the shadow is Gaussian distribution No. 1, the Gaussian distribution describing the inner area of ​​the shadow is Gaussian distribution No. 2, and the remaining Gaussian distributions are used to describe the edge area of ​​the shadow. By calculating The maximum value in determines the coordinate point To which Gaussian distribution it belongs, we can determine which region it belongs to. The specific expression is as follows:

[0082]

[0083] In formula (6), M out 、M in and M edge They represent the outer area of ​​the shadow, the inner area of ​​the shadow, and the edge area of ​​the shadow respectively.

[0084] Specifically, in the M step, at the tth iteration, after the region division is updated using the E step, the LV coordinate points belonging to each region will change, so the parameters representing the distribution of each region need to be recalculated. In this method, the posterior probability calculated in the E step is used to update the distribution parameters. The specific expression is as follows:

[0085]

[0086] Updated distribution parameters It will be used to correct the distribution of the shadow edge area, so as to obtain the edge area pixel value with better restoration effect.

[0087] Then, in the S and D steps, the S step uses Markov Chain Monte Carlo (MCMC) sampling to adjust the positions of the pixels in the shadow edge area in the LV coordinate system so that they satisfy the distribution represented by the GMM model, while the D step uses a Diffusion model pre-trained on other datasets to refine the points sampled in the S step.

[0088] In this embodiment, after the update of step M, the coordinate point set Position in the LV coordinate system and updated distribution parameters So it is necessary to Adjust to ensure that its prior and The priors described are consistent.

[0089] In this embodiment, MCMC sampling is used to obtain a new set of coordinate points. Satisfy The corresponding distribution.

[0090] Furthermore, for A point in As far as the shadow image is concerned, since the shadow image has been determined, it is only compared with the nth pixel value of the shadow-free image. It can be expressed as therefore The distribution of can be expressed using the following formula:

[0091]

[0092] In formula (8), preferably, The value range of is [0,255]. However, it is difficult to obtain all possible To calculate the above integral, in this method, Monte Carlo sampling is used to approximate the integral in formula (8) as a summation form, and the expression is as follows:

[0093]

[0094] In formula (9), l = 1, 2, ..., L represents the Monte Carlo sampling index. In order to ensure that the obey The corresponding distribution needs to be As a condition to control the sampling process, the specific expression is as follows:

[0095]

[0096] In formula (10), Indicates the distribution parameters based on the current region Monte Carlo sampling The weight applied. Rearrange according to the coordinate position to get The reference image corresponding to the distribution represented The mathematical representation is as follows:

[0097]

[0098] Preferably, only The points in the shadow edge area are adjusted because the degradation of the non-shadow area and the inner shadow area is relatively simple, and the existing shadow removal method can achieve a good solution effect, so no further adjustment is needed. However, the brightness of the shadow edge area changes complexly, and the pixel prediction error is large. The coordinate point in the LV coordinate system is different from the updated distribution parameters. The mismatch is large, so MCMC sampling is needed for prior correction. At the same time, after this step, a preliminary reference image can actually be obtained. The image can be obtained by using the coordinate points in the LV coordinate system obtained in the current step.

[0099] Furthermore, in the tth iteration, the distribution parameters updated in step S are It is not necessarily the optimal parameter to describe the distribution of coordinate points in the edge area, so the current reference image needs to be Further refinement is performed on the edge areas.

[0100] In this embodiment, step D utilizes the powerful data distribution fitting capability of the Diffusion model to solve the shadow edge removal problem, and uses the Diffusion model pre-trained on other datasets as a priori generator to improve the quality of the restored image in the shadow edge area.

[0101] Diffusion defines a T-step forward diffusion process and a T-step backward diffusion process. The forward diffusion process slowly adds Gaussian noise to the training data. After T steps, the training data will tend to be Gaussian distributed. The backward diffusion process needs to iteratively extract training data samples from the Gaussian noise in the opposite direction of the forward diffusion process, which is equivalent to a denoising process. The forward diffusion process can be described mathematically as:

[0102]

[0103] In formula (12), x t is the training data, is a hyperparameter.

[0104] The reverse diffusion process requires Gaussian noise x T Start and iterate step by step to solve x T-1 ,x T-2 ,…,x0. According to Bayes’ formula, if we want to t Start and solve for x t-1 , you can first use x t Estimate the original image x at this time 0|t , and then according to x t and x 0|t Sampling to get xt-1 The specific mathematical formula is as follows:

[0105]

[0106] In formula (13), ∈ θ (·) is the denoising network, is standard Gaussian noise.

[0107] Furthermore, according to the principle of Diffusion, if the shadow-free image As the end point of reverse diffusion, the t-th step diffusion process can be expressed as follows:

[0108]

[0109] In this embodiment, the That is, the intermediate variable of the reference image to the Diffusion reverse diffusion process The shadow edge area is corrected to obtain a finer shadow-free image. The mathematical expression is as follows:

[0110]

[0111] In formula (15), is the gradient operator, is the shadow edge mask operator at the tth iteration. Compared with existing diffusion-based deshadowing methods, this method has two main advantages: (1) the adopted diffusion model is ready-made, thus avoiding the time-consuming data collection and training process. (2) A reasonable edge prior is used to guide the diffusion inference process, ensuring effective shadow edge recovery.

[0112] In this embodiment, inspired by the image deshadowing problem degradation model, the shadow-free image obtained by the tth round of iteration can be With shadow image I s Perform weighted fusion to obtain a shadow-free image with better performance. The mathematical expression is as follows:

[0113]

[0114] In formula (16), is the shadow-free image obtained in the tth iteration, α t is the weight matrix of the tth round.

[0115] Furthermore, in this method, a lightweight neural network G is used α To estimate the weight matrix α t ,therefore The expression can be rewritten as follows:

[0116]

[0117] In formula (17), The lightweight network G at the tth iteration α Parameters.

[0118] To ensure The distribution of the shadow edge area and the parameters after the S step update The distribution described is consistent, and the following loss function is used to adjust the parameters To update:

[0119]

[0120] The update process is implemented using gradient descent:

[0121]

[0122] In formula (19), is the update step size. Through the optimization of the above loss function, the distribution prior of the shadow edge area can be embedded into the parameters of the lightweight network, thereby ensuring that the estimated weight matrix α t effectiveness.

[0123] After the Diffusion model and the lightweight network G α The shadow-free image output by the tth round of iteration can be expressed as follows:

[0124]

[0125] It is used as the input for the next iteration. Repeat the above EMSD iteration process and finally restore the shadow-free image.

[0126] like Figure 3 The figure shows the overall block diagram of the shadow edge removal technology in this method. Let t = T, T-1, ..., 0, which represents the number of iterations. For the tth iteration, step E, the shadow-free image obtained by the previous iteration is As input, calculate the posterior probability of the LV coordinate point And update the region division; M step according to the posterior probability To calculate the distribution parameters of each region The S step obtains the distribution parameters calculated in the M step through MCMC sampling. Matching reference images The consistency of the edge prior is guaranteed. The D step uses the existing diffusion model and lightweight network. Estimating shadow-free images under the guidance of and the weight matrix α t-1 Finally, using α t-1 Fusion and shadow image I s , get the shadow-free image output in round t By repeating the above iterative process, the shadow edge area is finally enhanced and repaired.

[0127] In this paper, the effectiveness of this method is also demonstrated through experimental results.

[0128] like Figure 4 As shown in the figure, compared with other shadow removal algorithms, the shadow-free image reconstructed by the shadow removal algorithm proposed in this paper not only has no obvious chromaticity distortion in the overall shadow area, but also the generated content in the shadow boundary area is very harmonious, and no shadow artifacts can be observed at the visual level. All the most advanced shadow removal methods (corresponding to Figure 4 (b)-(e)) all produce serious shadow edge artifacts and lose some texture structure information. On the contrary, the algorithm EMSD proposed in this method can help the existing shadow removal method to better remove shadow edge artifacts while retaining the texture details of the original image (corresponding to Figure 4 (f)-(g)).

[0129] As shown in Table 1, the EMSD algorithm proposed in this paper helps the existing deshadowing algorithms DC-ShadowNet (unsupervised) and Homoformer (supervised) achieve the best deshadowing performance in the shadow area and the entire image, proving the effectiveness of the algorithm proposed in this paper at the data level.

[0130] Table 1 Quantitative experimental results of this method and various existing methods on the shadow removal dataset

[0131]

[0132] like Figure 5 As shown, as the iterative process proceeds, the PSNR value of the restored shadow-free image gradually increases and the LPIPS value gradually decreases, which proves the effectiveness of the algorithm of the present invention at the data level. (c): As the iterative process proceeds, the visualization result of the restored shadow-free image is displayed (the shadow area is magnified). It can be seen that it gradually removes the shadow edge artifacts while retaining the texture details of the original image. (d): As the iterative process proceeds, the distribution of the image pair composed of the restored shadow-free image and the original shadow image in the LV coordinate system is displayed. It can be seen that in the process of shadow edge recovery, the coordinate points of different regions in the LV coordinate system (marked with different colors) are gradually separated, indicating that the region division is gradually becoming reasonable.

[0133] As shown in Table 2, the performance of the proposed algorithm degrades to varying degrees when any of the four steps, E, M, S, and D, are omitted. The performance degradation is most severe when the D step is omitted, indicating that although the Diffusion model used in this method is readily available, its rich data prior information still plays an important role in the shadow edge recovery process.

[0134] Table 2 Quantitative experimental results of performance evaluation of each step of this method

[0135]

[0136] like Figure 6 As shown, the second and fourth columns are paired shadow / no-shadow image pairs, and the first column shows the distribution of coordinate points formed by these image pairs in the LV coordinate system. The distribution results shown in the first column of images further confirm the phenomenon discovered by this method in step S110: the distribution of points in different regions can be approximated by single or multiple Gaussian distributions (ellipses) (inside the shadow: red ellipse, outside the shadow: blue ellipse, shadow edge: yellow ellipse). The third column shows the distribution of pixel values ​​in the shadow image, and the fifth column shows the distribution of pixel values ​​in the no-shadow image. These are the results of projecting the distribution of coordinate points in the LV coordinate system along the two coordinate axes.

[0137] In this plug-and-play image shadow removal method, a luminance variation (LV) coordinate system is first established. This system characterizes the distribution of luminance changes for all pixels before and after shadow removal, thereby enabling the simultaneous description of region delimitation and pixel brightness adjustment within a single coordinate system. A Gaussian mixture model (GMM) is then designed to redefine the image shadow removal problem as a maximum a posteriori (MAP) estimation problem for coordinate points in the LV coordinate system. To address this MAP problem, an iterative algorithm, called EMSD, is proposed. This algorithm divides each iteration into four steps: E, M, S, and D. Steps E and M follow the traditional Expectation Maximization (EM) algorithm to update the region segmentation results and the parameters of the GMM model. Step S uses Markov Chain Monte Carlo (MCMC) sampling to adjust the positions of pixels in the shadow edge region in the LV coordinate system so that they conform to the distribution represented by the GMM model. Step D uses a diffusion model pre-trained on another dataset to refine the points sampled in step S. These four steps sequentially and iteratively optimize the MAP problem, achieving accurate edge segmentation and adaptive brightness correction, ultimately achieving edge enhancement and restoration. The proposed EMSD algorithm can be easily and fully applied to various existing shadow removal frameworks, providing a plug-and-play approach for fine-tuning shadow edge restoration, significantly improving the performance of shadow edge restoration. Furthermore, it requires no training data and enables unsupervised real-time optimization. Extensive experimental results confirm that when integrated with the current state-of-the-art methods, the proposed EMSD algorithm significantly improves the recovery performance of shadow edge regions, especially on the methods DC-ShadowNet (unsupervised) and Homoformer (supervised).

[0138] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0139] In one embodiment, Figure 7 As shown, a plug-and-play image shadow edge removal device is provided, comprising: an image acquisition module 200, a problem model construction module 210, a problem model solution module 220, and an iteration termination judgment module 230, wherein:

[0140] The image acquisition module 200 is used to acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image.

[0141] The problem model construction module 210 is used to construct a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and use a Gaussian mixture model to model the point data in the brightness change coordinate system, thereby converting the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and constructing a corresponding problem model.

[0142] The problem model solving module 220 is used to bring the shadow image and the shadow-free image into the problem model and solve them using an iterative algorithm. In each iteration, the division of the subordinate areas is updated in turn, the distribution parameters representing each subordinate area are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge area in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge area is optimized, and the shadow-free image after the edge area is optimized is used as the input for the next iteration.

[0143] The iteration termination judgment module 230 is used to perform the iteration until a preset termination condition is met, and the shadow-free image obtained after the edge area optimization is obtained is the final output result, so as to achieve image shadow edge removal.

[0144] The specific definition of the plug-and-play image edge removal device can be found in the definition of the plug-and-play image edge removal method described above and will not be further elaborated here. Each module in the aforementioned plug-and-play image edge removal device can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0145] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a plug-and-play image shadow edge removal method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0146] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0148] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0149] A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed.

[0150] After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0151] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0153] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0154] A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed.

[0155] After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0156] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

[0157] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A plug-and-play image shadow edge removal method, characterized in that: The method comprises: Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image; A brightness change coordinate system is constructed based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model. The image shadow removal problem is transformed into a maximum a posteriori estimation problem for the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed. After the shadow image and the shadow-free image are introduced into the problem model, an iterative algorithm is used to solve the problem. In each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image. The edge region in the reference image is then optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized. The shadow-free image after the edge region is optimized is used as the input for the next iteration. Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the removal of the shadow edge of the image.

2. The plug-and-play image shadow edge removal method according to claim 1, characterized in that: Based on a degradation model of the image shadow removal problem, the brightness change coordinate system is constructed according to a mapping relationship between pixels at the same position between the shadow image and the shadow-free image; In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration; The coordinates of each point data in the brightness change coordinate system are pixel values ​​at the same position in the shadow image and the shadow-free image.

3. The plug-and-play image shadow edge removal method according to claim 2, characterized in that: The belonging regions include an inner belonging region representing the interior of the image shadow, an outer belonging region representing the exterior of the image shadow, and an edge belonging region representing an edge region of the image shadow.

4. The plug-and-play image shadow edge removal method according to claim 3, characterized in that: The problem model is expressed as: In the above formula, represents the coordinate set of the point data in the brightness change coordinate system, The latent variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, and They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

5. The plug-and-play image shadow edge removal method according to claim 4, characterized in that: During each iteration: The shadow-free image with optimized edge regions obtained in the previous iteration is used as input; When updating the division of the belonging region, updating the point data coordinates in the brightness change coordinate system according to the optimized shadow-free image of the edge region and the shadow image, calculating the intermediate posterior probability of each updated point data coordinate, and updating the division of the belonging region; When updating the distribution parameters representing each subordinate region, the updated distribution parameters of each subordinate region are calculated according to the intermediate posterior probability; When updating the position of each point data in the brightness change coordinate system, Markov chain Monte Carlo sampling is used to estimate the position of the point data in the edge membership region so that it satisfies the distribution parameters of the corresponding updated membership region, thereby obtaining a reference image with distribution matching; When optimizing and updating the edge area, a diffusion model and a lightweight network are used to estimate the shadow-free image and weight matrix of the edge area after optimization obtained in the previous iteration under the guidance of the reference image. The estimated shadow-free image and shadow image of the edge area after optimization obtained in the previous iteration are fused by using the weight matrix to obtain the result of the current iteration, i.e., the shadow-free image of the edge area after optimization.

6. The plug-and-play image shadow edge removal method according to claim 5, characterized in that: When updating the division of the subordinate regions and updating the distribution parameters characterizing each subordinate region, an expectation maximization algorithm is used.

7. The plug-and-play image shadow edge removal method according to claim 6, characterized in that: The diffusion model is a pre-trained model.

8. A plug-and-play image shadow edge removal device, characterized in that: The device comprises: An image acquisition module, configured to acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image; a problem model construction module, configured to construct a brightness change coordinate system based on brightness changes at the same position in the shadow image and the corresponding shadow-free image, and to model the point data in the brightness change coordinate system using a Gaussian mixture model, thereby converting the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging region of each point data in the brightness change coordinate system, and constructing a corresponding problem model; a problem model solving module, configured to introduce the shadow image and the shadow-free image into the problem model and solve the problem using an iterative algorithm, wherein during each iteration, the division of the subordinate regions is updated, the distribution parameters representing each subordinate region are updated, the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution-matched reference image, and the edge region in the reference image is optimized and updated to obtain a current iteration result, i.e., a shadow-free image after optimizing the edge region, and the shadow-free image after optimizing the edge region is used as input for the next iteration; The iteration termination judgment module is used to meet the preset iteration termination condition, and the shadow-free image after the edge area optimization currently obtained is the final output result to achieve image shadow edge removal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images