Plug-and-play image shadow edge removing method and device, equipment and medium

Abstract

The invention relates to a plug-and-play image shadow edge removing method and device, equipment and a medium, and the method comprises the steps: constructing a brightness change coordinate system based on the brightness change of pixels of a shadow image and a non-shadow image, and carrying out the Gaussian mixture model modeling of the brightness change coordinate system; taking the maximum posteriori estimation problem of the position and the subordinate area of each point data in the brightness change coordinate system as a problem model, and solving the problem model by adopting an iterative algorithm; in each iteration process, the division of the membership areas is updated in sequence, the distribution parameters representing the updated membership areas are updated, the data positions of all points in the brightness change coordinate system are updated according to the updated distribution parameters to obtain a reference image with matched distribution, and then the edge areas in the image are optimized and updated. And obtaining a current iteration result until a preset iteration termination condition is met to obtain a shadow-free image. By adopting the method, the recovery performance of the shadow edge region can be enhanced, and an image with a better shadow removal effect can be obtained.

Description

Technical Field

[0001] The present application relates to the field of optical image intelligent enhancement and computer intelligence technology, and in particular to a plug-and-play image shadow edge removal method, device, equipment and medium. Background Art

[0002] Image shadow edge removal is an open problem, whose main goal is to enhance the visibility of the image in the shadow edge area and accurately estimate the brightness change of each pixel in the shadow edge area to pursue consistent illumination distribution between the shadow area, shadow edge and non-shadow area.

[0003] In recent years, with the rapid development of artificial intelligence, many basic frameworks such as Generative Adversarial Network (GAN), Transformer, Diffusion model, etc. have emerged, and a series of image de-shadowing algorithms have been born based on these frameworks. These algorithms are committed to restoring the image content in shadow areas and shadow edge areas, and have achieved many impressive results, but the shadow-free images they restore have very obvious artifacts in the shadow edge areas. These residual artifacts will have a negative impact on downstream high-level visual tasks and cannot meet the image quality requirements in fields such as mobile photography, mapping, and autonomous driving. For the above reasons, it is necessary to design a shadow edge optimizer for the existing image de-shadowing algorithm to enhance the restoration performance in the shadow edge area. Summary of the invention

[0004] Based on this, it is necessary to provide a plug-and-play image shadow edge removal method, device, equipment and medium that can enhance the shadow edge area restoration performance in order to solve the above technical problems.

[0005] A plug-and-play method for removing shadow edges of an image, the method comprising:

[0006] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0007] A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed;

[0008] After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration;

[0009] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

[0010] In one embodiment, based on a degradation model of the image shadow removal problem, the brightness change coordinate system is constructed according to a mapping relationship between pixels at the same position between the shadow image and the shadow-free image;

[0011] In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration;

[0012] The coordinates of each point data in the brightness change coordinate system are pixel values ​​at the same position in the shadow image and the shadow-free image.

[0013] In one embodiment, the belonging regions include an inner belonging region representing the inside of the image shadow, an outer belonging region representing the outside of the image shadow, and an edge belonging region representing the edge region of the image shadow.

[0014] In one embodiment, the problem model is expressed as:

[0015]

[0016] In the above formula, represents the coordinate set of point data in the brightness change coordinate system, The hidden variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

[0017] In one embodiment, during each iteration:

[0018] The shadow-free image with optimized edge area obtained in the previous iteration is used as input;

[0019] When the division of the subordinate region is updated, the point data coordinates in the brightness change coordinate system are updated according to the optimized shadow-free image of the edge region and the shadow image, the intermediate posterior probability of each updated point data coordinate is calculated, and the division of the subordinate region is updated;

[0020] When updating the distribution parameters characterizing each subordinate region, the updated distribution parameters of each subordinate region are calculated according to the intermediate posterior probability;

[0021] When updating the position of each point data in the brightness change coordinate system, Markov chain Monte Carlo sampling is used to estimate the position of the point data in the edge membership region so that it satisfies the distribution parameters of the corresponding updated membership region, thereby obtaining a reference image with distribution matching;

[0022] When optimizing and updating the edge area, a diffusion model and a lightweight network are used to estimate the optimized shadow-free image and weight matrix of the edge area obtained in the previous iteration under the guidance of the reference image. The estimated optimized shadow-free image of the edge area obtained in the previous iteration and the shadow image are fused by using the weight matrix to obtain the result of the current iteration, i.e., the optimized shadow-free image of the edge area.

[0023] In one embodiment, an expectation maximization algorithm is used when updating the division of the belonging regions and updating the distribution parameters characterizing each belonging region.

[0024] In one embodiment, the diffusion model is a pre-trained model.

[0025] The present application also provides a plug-and-play image shadow edge removal device, the device comprising:

[0026] An image acquisition module, used for acquiring a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0027] A problem model building module, used to build a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and use a Gaussian mixture model to model the point data in the brightness change coordinate system, convert the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and build a corresponding problem model;

[0028] A problem model solving module, used for bringing the shadow image and the shadow-free image into the problem model, solving the problem using an iterative algorithm, updating the division of the subordinate regions in each iteration, updating the distribution parameters characterizing each subordinate region, updating the data position of each point in the brightness change coordinate system according to the updated distribution parameters to obtain a reference image with distribution matching, and optimizing and updating the edge region in the reference image to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and using the shadow-free image after the edge region is optimized as the input for the next iteration;

[0029] The iteration termination judgment module is used to meet the preset iteration termination condition, and the currently obtained edge area optimized shadow-free image is the final output result to achieve image shadow edge removal.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0032] A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed;

[0033] After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration;

[0034] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0036] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0037] A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed;

[0038] After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration;

[0039] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

[0040] The above-mentioned plug-and-play image shadow edge removal method, device, equipment and medium, by constructing a brightness change coordinate system based on the brightness change of the same position in the shadow image and the corresponding shadow-free image, and modeling it using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system and a problem model is constructed, and the shadow image and the corresponding shadow-free image are brought into the problem model, and an iterative algorithm is used to solve it, and in each iteration process, the division of the belonging area is updated in turn, the distribution parameters representing each belonging area are updated, and the position of each point data in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a distribution matching reference image, and the edge area in the reference image is optimized and updated to obtain the current iteration result, that is, the shadow-free image after the edge area is optimized, and the shadow-free image after the edge area is optimized is used as the input of the next iteration until the preset iteration termination condition is met, and the shadow-free image after the edge area is optimized is the shadow-free image after the shadow edge is removed. The method can enhance the shadow edge area recovery performance, thereby obtaining an image with better shadow removal effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a flow chart of a plug-and-play method for removing shadow edges of an image in one embodiment;

[0042] Figure 2A schematic diagram of region division and GMM modeling based on the LV coordinate system in one embodiment;

[0043] Figure 3 A schematic diagram of the overall framework of a plug-and-play shadow edge removal method in one embodiment;

[0044] Figure 4 This is an example of the visualization results of this method and multiple existing methods on a shadow removal dataset in an experiment;

[0045] Figure 5 This is an example of an intermediate iteration visualization result in an experiment when using this method;

[0046] Figure 6 The distribution of more shadow / no-shadow image pairs in the LV coordinate system in an experiment and the schematic diagram of their respective one-dimensional distributions;

[0047] Figure 7 A structural block diagram of a plug-and-play image shadow edge removal device in one embodiment;

[0048] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] There are currently two major difficulties in removing shadow edges. First, the edge area is difficult to accurately define. Simply using graphics erosion and dilation operations may result in incorrect area division. Second, the brightness changes in the edge area are very complex, and the restoration result is prone to produce serious edge artifacts. In response to these two problems, in this application, Figure 1 As shown, a plug-and-play image shadow removal edge method is proposed, which aims to optimize the existing image shadow removal technology, and specifically includes the following steps:

[0051] Step S100, obtaining a shadow image and a shadow-free image after preliminary restoration of the shadow image.

[0052] Step S110, constructing a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and using a Gaussian mixture model to model the point data in the brightness change coordinate system, converting the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and constructing a corresponding problem model.

[0053] Step S120, after the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve it. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0054] Step S130, until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to achieve image shadow edge removal.

[0055] The above method is based on the existing image deshadowing technology, combined with the physical model of pixel brightness changes before and after image deshadowing, and focuses on the research of plug-and-play real-time intelligent image shadow edge removal technology to achieve high-quality and efficient image shadow edge removal, providing more high-quality samples for downstream advanced computer vision tasks such as image segmentation, target detection and recognition, and target tracking.

[0056] In step S100, the shadow-free image after preliminary restoration is obtained based on the shadow image using an existing shadow removal method, including but not limited to a shadow removal method based on an illumination model, a shadow removal method based on image filtering, a shadow removal method based on threshold segmentation, and a shadow removal method based on deep learning. At this time, the obtained shadow-free image has very obvious artifacts in the shadow edge area.

[0057] In step S110, based on the degradation model of the image shadow removal problem, a brightness change coordinate system is constructed according to the mapping relationship between the pixels at the same position between the shadow image and the shadow-free image. The degradation model of the image shadow removal problem is usually modeled as follows:

[0058] I sf =α⊙I r +(1-α)⊙I s (1)

[0059] In formula (1), I s represents the shadow image, I sf represents the restored shadow-free image, I r represents the image after brightness adjustment, α represents the weight matrix, and ⊙ represents the element-by-element multiplication operation.

[0060] From formula (1), we can see that for I s A pixel value at any position on There is a corresponding mapping function that maps it to Isf The pixel value at the same position on Based on this, a brightness variation (LV) coordinate system is established in this method to describe I s and I sf In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration. That is, the coordinates of each point data in the brightness change coordinate system are the pixel values ​​at the same position in the shadow image and the shadow-free image. Therefore, the pixel values ​​at the same position before and after the image is deshadowed can be represented as a point in the LV coordinate system. Note: s n (x n ,y n ).

[0061] like Figure 2 As shown in (a)-(c), assuming that the shadow image contains N pixels, the mapping relationship between it and its paired shadow-free image can be expressed as a set of N coordinate points in the LV coordinate system.

[0062] Furthermore, the distribution of the N coordinate points of the paired shadow and shadow-free images in the LV coordinate system is Use Gaussian mixture model (GMM) for modeling, such as Figure 2 (d) is shown, and its mathematical expression is as follows:

[0063]

[0064] in,

[0065] In formula (2), K represents the number of Gaussian distributions, ω k , μ k and∑ k They represent the weight, mean and variance of the kth Gaussian distribution respectively.

[0066] like Figure 2 As shown in , if the points belonging to different areas in the LV coordinate system are represented by different colors, it can be found that the distribution of points in different areas can be approximated by a single or multiple Gaussian distributions (ellipses). The distribution of coordinate points in the shadow inner area and the shadow outer area can be fitted by only one Gaussian distribution. Due to the complex brightness changes in the shadow edge area, the distribution of its coordinate points requires multiple Gaussian distributions to fit. Figure 2As shown in (e), the coordinate points of the outer and edge areas are marked in red, blue, and yellow, respectively. It can be seen that the coordinate points of the inner and outer areas of the shadow can be fitted with one Gaussian distribution (corresponding to the red circle and the blue circle), and the points of the edge area of ​​the shadow can be fitted with three Gaussian distributions (corresponding to the yellow circle). Therefore, by adjusting the inner area of ​​the shadow, the outer area of ​​the shadow, and the edge area of ​​the shadow in the LV coordinate system to make their distribution more reasonable, an image with better shadow removal effect can be obtained.

[0067] Furthermore, based on the above phenomenon, the coordinate point The corresponding area category can be equivalently represented by the Gaussian distribution to which it belongs. In order to facilitate the formal expression, in this method, the latent variable To describe The region to which it belongs, express belongs to the kth Gaussian distribution.

[0068] In this embodiment, the belonging regions include an inner belonging region representing the inside of the image shadow, an outer belonging region representing the outside of the image shadow, and an edge belonging region representing the edge region of the image shadow.

[0069] In the actual shadow removal work, since the shadow-free image I sf The pixel values ​​and region divisions are usually unknown, which leads to the coordinate point set and the latent variable set According to the definition of LV coordinate system, the image shadow removal problem can be transformed into the LV coordinate system. The maximum a posteriori estimation (MAP) problem is the problem model to be solved, and the mathematical expression is as follows:

[0070]

[0071] In formula (3), represents the coordinate set of point data in the brightness change coordinate system, The hidden variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

[0072] Furthermore, according to the GMM model, Multiple Gaussian distributions can be used for fitting. The specific expressions are as follows:

[0073]

[0074] Therefore, in the above MAP problem, we need to The distribution parameters of And its corresponding regional category latent variable Make an estimate.

[0075] In step S120, an iterative solution method is proposed to solve the problem model represented by formula (3). In each iteration process: the optimized shadow-free image of the edge area obtained in the previous iteration is used as input. When the division of the belonging area is updated, the point data coordinates in the brightness change coordinate system are updated according to the optimized shadow-free image of the edge area and the shadow-free image, the intermediate posterior probability of each updated point data coordinate is calculated, and the division of the belonging area is updated. When the distribution parameters characterizing each subordinate region are updated, the distribution parameters of each subordinate region after the update are calculated according to the intermediate posterior probability. Then, when the data position of each point in the brightness change coordinate system is updated, the Markov chain Monte Carlo sampling is used to estimate the position of the middle point data in the edge subordinate region so that it satisfies the distribution parameters of the corresponding updated subordinate region, and a reference image with distribution matching is obtained. When the edge region is optimized and updated, the diffusion model is used to estimate the optimized shadow-free image and weight matrix of the edge region obtained in the previous iteration under the guidance of the reference image. The estimated shadow-free image of the edge region after the optimization of the previous iteration and the shadow image are fused by using the weight matrix to obtain the result of the current iteration, i.e., the optimized shadow-free image of the edge region.

[0076] In this embodiment, an iterative algorithm solution method for EMSD is proposed, in which each iteration process is divided into four steps: E, M, S, and D, which respectively represent updating the division of the subordinate areas, i.e., step E, updating the distribution parameters characterizing each subordinate area, i.e., step M, updating the data position of each point in the brightness change coordinate system, i.e., step S, and optimizing and updating the edge area, i.e., step D.

[0077] In this embodiment, when updating the division of the subordinate regions and updating the distribution parameters characterizing each subordinate region, an expectation maximization (EM) algorithm is used.

[0078] Specifically, in step E, the latent variable set The posterior probability is used as the indicator of regional division. In the tth iteration, when the coordinate point set at this time is given The distribution parameters of When, for A point in Its corresponding hidden variable The posterior probability can be expressed as follows:

[0079]

[0080] In formula (5), Indicates the coordinate point at the tth iteration The probability of belonging to the kth Gaussian distribution.

[0081] Furthermore, after calculating the posterior probability of the latent variable corresponding to each coordinate point, it can be used as the basis for updating the region division result. At the same time, in this method, the Gaussian distribution describing the outer area of ​​the shadow is Gaussian distribution No. 1, the Gaussian distribution describing the inner area of ​​the shadow is Gaussian distribution No. 2, and the remaining Gaussian distributions are used to describe the edge area of ​​the shadow. The maximum value in determines the coordinate point To which Gaussian distribution it belongs, we can determine which region it belongs to. The specific expression is as follows:

[0082]

[0083] In formula (6), M out 、M in and M edge They represent the outer area of ​​the shadow, the inner area of ​​the shadow, and the edge area of ​​the shadow respectively.

[0084] Specifically, in the M step, in the tth iteration, when the region division is updated using the E step, the LV coordinate points belonging to each region will change, so the parameters representing the distribution of each region need to be recalculated. In this method, the posterior probability calculated in the E step is used to update the distribution parameters, and the specific expression is as follows:

[0085]

[0086] Updated distribution parameters It will be used to correct the distribution of the shadow edge area, so as to obtain the edge area pixel value with better restoration effect.

[0087] Then, in the S and D steps, the S step uses Markov Chain Monte Carlo (MCMC) sampling to adjust the positions of the pixels in the shadow edge area in the LV coordinate system so that they satisfy the distribution represented by the GMM model, while the D step uses a Diffusion model pre-trained on other data sets to refine the points sampled in the S step.

[0088] In this embodiment, after the update of step M, the coordinate point set Position in the LV coordinate system and updated distribution parameters So it is necessary to Adjust to ensure that its prior The priors described are consistent.

[0089] In this embodiment, MCMC sampling is used to obtain a new set of coordinate points. Satisfy The corresponding distribution.

[0090] Furthermore, for A point in As far as the shadow image is concerned, since it has been determined, it is only compared with the nth pixel value of the shadow-free image. It can be expressed as therefore The distribution of can be expressed using the following formula:

[0091]

[0092] In formula (8), preferably, The value range of is [0,255]. However, it is difficult to obtain all possible To calculate the above integral, in this method, Monte Carlo sampling is used to approximate the integral in formula (8) as a summation, and the expression is as follows:

[0093]

[0094] In formula (9), l = 1, 2, ..., L represents the Monte Carlo sampling index. obey The corresponding distribution needs to be As a condition to control the sampling process, the specific expression is as follows:

[0095]

[0096] In formula (10), Indicates the distribution parameters based on the current region Monte Carlo sampling The weight applied. By rearranging the coordinates, we can get The reference image corresponding to the distribution represented by The mathematical representation is as follows:

[0097]

[0098] Preferably, only The points in the shadow edge area are adjusted because the degradation of the non-shadow area and the inner shadow area is relatively simple, and the existing shadow removal method can achieve a good solution effect, so no further adjustment is needed. However, the brightness of the shadow edge area changes complexly, and the pixel prediction error is large. The coordinate point in the LV coordinate system is different from the updated distribution parameter. The mismatch is large, so MCMC sampling is needed for prior correction. At the same time, after this step, a preliminary reference image can actually be obtained. The image can be obtained by the coordinate points in the LV coordinate system obtained in the current step.

[0099] Furthermore, in the tth iteration, the distribution parameters updated in step S are It is not necessarily the optimal parameter to describe the distribution of coordinate points in the edge area, so it is also necessary to Further refinement is performed on the edge areas.

[0100] In this embodiment, step D utilizes the powerful data distribution fitting ability of the Diffusion model to the shadow edge removal problem, and uses the Diffusion model pre-trained on other data sets as a priori generator to improve the quality of the restored image in the shadow edge area.

[0101] Diffusion defines a T-step forward diffusion process and a T-step reverse diffusion process, in which the forward diffusion process slowly adds Gaussian noise to the training data. After T steps, the training data will tend to Gaussian distribution; while the reverse diffusion process needs to iterate the training data samples from the Gaussian noise step by step in the opposite direction of the forward diffusion process, which is equivalent to a denoising process. The forward diffusion process can be described by the mathematical formula:

[0102]

[0103] In formula (12), x t is the training data, is a hyperparameter.

[0104] The reverse diffusion process requires Gaussian noise x T Start from the beginning and gradually iterate to solve x T-1 ,x T-2 ,…,x0. According to the Bayesian formula, if we want to t Let’s start by solving for x t-1 , you can first use x t Estimate the original image x at this time 0|t , and then according to x t and x 0|t Sampling to get xt-1 The specific mathematical formula is as follows:

[0105]

[0106] In formula (13), ∈ θ (·) is the denoising network, is standard Gaussian noise.

[0107] Furthermore, according to the principle of Diffusion, if the shadow-free image As the end point of reverse diffusion, the t-th step diffusion process can be expressed as follows:

[0108]

[0109] In this embodiment, use That is, the reference image is the intermediate variable of the Diffusion reverse diffusion process The shadow edge area is corrected to obtain a more refined shadow-free image. The mathematical expression is as follows:

[0110]

[0111] In formula (15), is the gradient operator, is the shadow edge mask operator at the tth iteration. Compared with the existing diffusion-based shadow removal methods, this method has two main advantages: (1) The adopted diffusion model is ready-made, thus avoiding the time-consuming data collection and training process. (2) A reasonable edge prior is used to guide the diffusion reasoning process, ensuring effective shadow edge recovery.

[0112] In this embodiment, inspired by the image deshadowing problem degradation model, the shadow-free image obtained by the tth round of iteration can be With shadow image I s Weighted fusion is performed to obtain a shadow-free image with better performance. The mathematical expression is as follows:

[0113]

[0114] In formula (16), is the shadow-free image obtained in the tth iteration, α t is the weight matrix of the tth round.

[0115] Furthermore, in this method, a lightweight neural network G is used α To estimate the weight matrix α t ,therefore The expression can be rewritten as follows:

[0116]

[0117] In formula (17), is the lightweight network G at the tth iteration α Parameters.

[0118] To ensure The distribution of the shadow edge area and the parameters after S step update The distribution described is consistent with the following loss function. To update:

[0119]

[0120] The update process is implemented using gradient descent:

[0121]

[0122] In formula (19), is the update step size. Through the optimization of the above loss function, the distribution prior of the shadow edge area can be embedded into the parameters of the lightweight network, thereby ensuring that the estimated weight matrix α t effectiveness.

[0123] After the Diffusion model and the lightweight network G α The optimization of the shadow-free image finally output by the tth iteration can be expressed as follows:

[0124]

[0125] It is used as the input for the next iteration. Repeat the above EMSD iteration process and finally restore the shadow-free image.

[0126] like Figure 3 As shown in the figure, it is the overall block diagram of the shadow edge removal technology in this method. Let t = T, T-1, ..., 0, which represents the number of iterations. For the tth iteration, step E, the shadow-free image obtained by the previous iteration As input, calculate the posterior probability of the LV coordinate point And update the region division; M step according to the posterior probability To calculate the distribution parameters of each region The S step obtains the distribution parameters calculated in the M step through MCMC sampling Matching reference images The consistency of edge priors is guaranteed. The D step uses the existing diffusion model and lightweight network. Estimation of shadow-free image under the guidance of and the weight matrix α t-1 Finally, using α t-1 Fusion and shadow image I s , get the shadow-free image output in round t By repeating the above iterative process, the shadow edge area is finally enhanced and repaired.

[0127] In this paper, the effectiveness of this method is also demonstrated through experimental results.

[0128] like Figure 4 As shown in Figure 2, compared with other shadow removal algorithms, the shadow-free image reconstructed by the shadow removal algorithm proposed in this paper not only has no obvious chromaticity distortion in the overall shadow area, but also the generated content in the shadow boundary area is very harmonious, and no shadow artifacts can be observed at the visual level. All the most advanced shadow removal methods (corresponding to Figure 4 (b)-(e)) all produce serious shadow edge artifacts and lose some texture structure information. On the contrary, the algorithm EMSD proposed in this method can help the existing shadow removal method to better remove shadow edge artifacts while retaining the texture details of the original image (corresponding to Figure 4 (f)-(g)).

[0129] As shown in Table 1, the EMSD algorithm proposed in this paper helps the existing shadow removal algorithms DC-ShadowNet (unsupervised) and Homoformer (supervised) achieve the best shadow removal performance in the shadow area and the entire image, proving the effectiveness of the algorithm proposed in this paper at the data level.

[0130] Table 1 Quantitative experimental results of this method and various existing methods on the shadow removal dataset

[0131]

[0132] like Figure 5 As shown, as the iterative process proceeds, the PSNR value of the restored shadow-free image gradually increases, and the LPIPS value gradually decreases, which proves the effectiveness of the algorithm of the present invention at the data level. (c): As the iterative process proceeds, the visualization result of the restored shadow-free image is displayed (the shadow area is enlarged). It can be seen that it gradually removes the shadow edge artifacts while retaining the texture details of the original image. (d): As the iterative process proceeds, the distribution of the image pair composed of the restored shadow-free image and the original shadow image in the LV coordinate system is displayed. It can be seen that in the process of shadow edge recovery, the coordinate points of different regions in the LV coordinate system (marked with different colors) are gradually separated, indicating that the regional division is gradually becoming reasonable.

[0133] As shown in Table 2, when any of the four steps E, M, S, and D is missing, the performance of the algorithm proposed in this method will decrease to varying degrees. When there is no D step, the performance degradation is the most serious, which shows that although the Diffusion model used in this method is ready-made, its rich data prior information still plays an important role in the process of shadow edge recovery.

[0134] Table 2 Quantitative experimental results of performance evaluation of each step of this method

[0135]

[0136] like Figure 6 As shown, the second and fourth columns are paired shadow / no-shadow image pairs, and the first column is the distribution of coordinate points formed by these image pairs in the LV coordinate system. The distribution results shown in the first column of images further confirm the phenomenon found in step S110 of this method: the distribution of points in different regions can be approximated by a single or multiple Gaussian distributions (ellipses) (inside the shadow: red ellipse, outside the shadow: blue ellipse, edge of the shadow: yellow ellipse). The third column shows the distribution of pixel values ​​of the shadow image, and the fifth column shows the distribution of pixel values ​​of the no-shadow image, which are the results of projecting the distribution of coordinate points in the LV coordinate system along two coordinate axes.

[0137] In the above plug-and-play image shadow edge removal method, a luminance variation (LV) coordinate system is first established, which characterizes the distribution of luminance variation of all pixels before and after shadow removal, thereby achieving the simultaneous description of region delimitation and pixel brightness adjustment in one coordinate system. Then a Gaussian mixture model (GMM) is designed to redefine the image shadow removal problem and model it as a maximum a posteriori estimation (MAP) problem of coordinate points on the LV coordinate system. In order to solve the MAP problem, an iterative algorithm named EMSD is proposed. The algorithm divides each iterative process into four steps: E, M, S, and D. The E and M steps follow the traditional expectation maximization (EM) algorithm to update the region segmentation results and the parameters of the GMM model. The S step uses Markov Chain Monte Carlo (MCMC) sampling to adjust the position of the pixel points in the shadow edge area in the LV coordinate system so that it satisfies the distribution represented by the GMM model. The D step uses a diffusion model pre-trained on other data sets to fine-tune the points sampled in the S step. These four steps optimize and solve the MAP problem in sequence and iteratively, achieving accurate segmentation and adaptive brightness correction of the edge area, thereby ultimately achieving enhancement and restoration of the edge area. The EMSD algorithm proposed in this method can be easily and completely applied to various existing shadow removal frameworks, and fine-tune the shadow edge in a plug-and-play manner, significantly improving the restoration performance of the shadow edge area. In addition, it does not require any training data and can achieve unsupervised real-time optimization. Extensive experimental results confirm that when integrated with the current state-of-the-art methods, the proposed EMSD algorithm significantly improves the restoration performance of shadow edge regions, especially on the methods DC-ShadowNet (unsupervised) and Homoformer (supervised).

[0138] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0139] In one embodiment, Figure 7 As shown, a plug-and-play image shadow edge removal device is provided, comprising: an image acquisition module 200, a problem model construction module 210, a problem model solution module 220, and an iteration termination judgment module 230, wherein:

[0140] The image acquisition module 200 is used to acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image.

[0141] The problem model construction module 210 is used to construct a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and use a Gaussian mixture model to model the point data in the brightness change coordinate system, so as to convert the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and construct a corresponding problem model.

[0142] The problem model solving module 220 is used to bring the shadow image and the shadow-free image into the problem model and solve it by an iterative algorithm. In each iteration, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input for the next iteration.

[0143] The iteration termination judgment module 230 is used to meet the preset iteration termination condition, and the currently obtained edge area optimized shadow-free image is the final output result to achieve image shadow edge removal.

[0144] The specific definition of the plug-and-play image shadow edge removal device can be found in the definition of the plug-and-play image shadow edge removal method above, which will not be repeated here. Each module in the above-mentioned plug-and-play image shadow edge removal device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0145] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a plug-and-play image shadow edge removal method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0146] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0148] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0149] A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed;

[0150] After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration;

[0151] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

[0152] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0153] Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image;

[0154] A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed;

[0155] After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and the edge region in the reference image is optimized and updated to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration;

[0156] Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A plug-and-play method for removing shadow edges of an image, characterized in that: The method comprises: Acquire a shadow image and a shadow-free image after preliminary restoration of the shadow image; A brightness change coordinate system is constructed based on the brightness change at the same position in the shadow image and the corresponding shadow-free image, and the point data in the brightness change coordinate system is modeled using a Gaussian mixture model, the image shadow removal problem is converted into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and a corresponding problem model is constructed; After the shadow image and the shadow-free image are brought into the problem model, an iterative algorithm is used to solve the problem. In each iteration process, the division of the subordinate regions is updated in turn, the distribution parameters characterizing each subordinate region are updated, and the data position of each point in the brightness change coordinate system is updated according to the updated distribution parameters to obtain a reference image with distribution matching, and then the edge region in the reference image is optimized and updated to obtain the current iteration result, that is, the shadow-free image after the edge region is optimized, and the shadow-free image after the edge region is optimized is used as the input of the next iteration; Until the preset iteration termination condition is met, the shadow-free image after the edge area optimization currently obtained is the final output result, so as to realize the image shadow edge removal.

2. The plug-and-play image shadow edge removal method according to claim 1, characterized in that: Based on the degradation model of the image shadow removal problem, the brightness change coordinate system is constructed according to the mapping relationship between the pixel points at the same position between the shadow image and the shadow-free image; In the brightness change coordinate system, the horizontal axis represents the pixel value of the shadow image, and the vertical axis represents the pixel value of the shadow-free image after preliminary restoration; The coordinates of each point data in the brightness change coordinate system are pixel values ​​at the same position in the shadow image and the shadow-free image.

3. The plug-and-play image shadow edge removal method according to claim 2, characterized in that: The belonging regions include an inner belonging region representing the inside of the image shadow, an outer belonging region representing the outside of the image shadow, and an edge belonging region representing the edge region of the image shadow.

4. The plug-and-play image shadow edge removal method according to claim 3, characterized in that: The problem model is expressed as: In the above formula, represents the coordinate set of point data in the brightness change coordinate system, The hidden variable set representing the region set to which each point data in the brightness change coordinate system belongs, I s represents the shadow image, represents the data likelihood term, and They represent the data priors of the coordinate point set and the latent variable set respectively, and p(·) represents the probability density.

5. The plug-and-play image shadow edge removal method according to claim 4, characterized in that: During each iteration: The shadow-free image with optimized edge area obtained in the previous iteration is used as input; When the division of the subordinate region is updated, the point data coordinates in the brightness change coordinate system are updated according to the optimized shadow-free image of the edge region and the shadow image, the intermediate posterior probability of each updated point data coordinate is calculated, and the division of the subordinate region is updated; When updating the distribution parameters characterizing each subordinate region, the updated distribution parameters of each subordinate region are calculated according to the intermediate posterior probability; When updating the position of each point data in the brightness change coordinate system, Markov chain Monte Carlo sampling is used to estimate the position of the point data in the edge membership region so that it satisfies the distribution parameters of the corresponding updated membership region, thereby obtaining a reference image with distribution matching; When optimizing and updating the edge area, a diffusion model and a lightweight network are used to estimate the optimized shadow-free image and weight matrix of the edge area obtained in the previous iteration under the guidance of the reference image. The estimated optimized shadow-free image of the edge area obtained in the previous iteration and the shadow image are fused by using the weight matrix to obtain the result of the current iteration, i.e., the optimized shadow-free image of the edge area.

6. The plug-and-play image shadow edge removal method according to claim 5, characterized in that: When updating the division of the subordinate regions and updating the distribution parameters characterizing each subordinate region, an expectation maximization algorithm is used.

7. The plug-and-play image shadow edge removal method according to claim 6, characterized in that: The diffusion model is a pre-trained model.

8. A plug-and-play image shadow edge removal device, characterized in that: The device comprises: An image acquisition module, used for acquiring a shadow image and a shadow-free image after preliminary restoration of the shadow image; A problem model building module, used to build a brightness change coordinate system based on the brightness changes at the same position in the shadow image and the corresponding shadow-free image, and use a Gaussian mixture model to model the point data in the brightness change coordinate system, convert the image shadow removal problem into a maximum a posteriori estimation problem of the position and belonging area of ​​each point data in the brightness change coordinate system, and build a corresponding problem model; A problem model solving module, used for bringing the shadow image and the shadow-free image into the problem model, solving the problem using an iterative algorithm, updating the division of the subordinate regions in each iteration, updating the distribution parameters characterizing each subordinate region, updating the data position of each point in the brightness change coordinate system according to the updated distribution parameters to obtain a reference image with distribution matching, and optimizing and updating the edge region in the reference image to obtain the current iteration result, i.e., the shadow-free image after the edge region is optimized, and using the shadow-free image after the edge region is optimized as the input for the next iteration; The iteration termination judgment module is used to meet the preset iteration termination condition, and the currently obtained edge area optimized shadow-free image is the final output result to achieve image shadow edge removal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images