Low-light defect image inpainting method and related apparatus

Abstract

This invention belongs to the field of image processing and discloses a method and related apparatus for repairing low-light defect images. The method includes acquiring a low-light defect image to be repaired and constructing a defect region masking image of the low-light defect image; performing illumination enhancement on the low-light defect image according to a pre-trained low-light image enhancement model to obtain an illumination-enhanced defect image; and performing defect content repair on the illumination-enhanced defect image according to a pre-trained defect content repair model and the defect region masking image to obtain a repaired image. The low-light image enhancement model is constructed based on Retinex theory; the defect content repair model is constructed based on a method applying receptive fields of different sizes. Images acquired in low-light environments first undergo illumination enhancement processing, and then the structure and texture of image defects are repaired by applying methods with different receptive field sizes to obtain a complete imaging effect. By applying a two-stage repair strategy of illumination enhancement and defect content repair, better low-light defect image repair results are obtained.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a method and related apparatus for repairing low-light defect images. Background Technology

[0002] Ancient murals are invaluable cultural heritage with immense archaeological value. Scholars can glean information about ancient religions, ceremonies, and customs from their content. However, due to long-term oxidation and a lack of effective protection, ancient murals continuously suffer from damage such as flaking and mold. Furthermore, because ancient murals were typically painted in indoor locations such as temples and tombs, images captured using digital devices often suffer from low lighting, resulting in poor observation of textures and structures. The structural defects and low-light conditions of ancient murals prevent the effective collection of valuable information. However, restoring ancient murals using manual painting methods is extremely slow, requires advanced expertise, and the process is irreversible; a poor restoration is tantamount to secondary damage. Therefore, research on the protection and display of ancient murals based on digital technology has become a current research hotspot. Currently, most research in image restoration focuses on repairing defective areas in the acquired images.

[0003] In summary, there are currently two main types of digital technologies for the restoration of damaged murals: (1) using traditional diffusion or patching techniques to propagate semantic information from the neighborhood to the missing area around the boundary to reconstruct the missing holes in the mural in order to achieve the restoration purpose. (2) using deep learning methods, employing progressive, structured information-guided, attention-based, convolution-based, and diversified strategies to restore mural images.

[0004] However, the field of ancient mural image restoration currently faces several unresolved issues. First, most ancient murals were painted indoors, and the use of strong lighting equipment such as flashlights is restricted to protect them, resulting in images typically acquired under low-light conditions. Current ancient mural restoration algorithms often focus on repairing damaged areas, neglecting the crucial factor of enhancing low-light conditions. This means that even if defective areas are repaired, the most valuable textures and structures remain difficult to observe with the naked eye. Second, the size of ancient murals depends on the size of the wall on which they were painted, often exhibiting large and diverse dimensions. Traditional deep learning-based image restoration models have limited ability to extract structural and textural information from large images and generally only process images of specific sizes in batches, failing to effectively restore damaged mural information. Furthermore, murals possess unique and rich colors and textures, with valuable information often hidden within these unique textures and colors. However, the availability of complete mural data suitable for training neural networks is limited, and the types of damage to murals are diverse, making it difficult for current mural restoration models to be trained robustly under such limited dataset conditions, resulting in poor restoration performance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for repairing low-light defect images.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] In a first aspect, the present invention provides a method for repairing low-light defect images, comprising:

[0008] Obtain the low-light defect image to be repaired, and construct a defect area masking image of the low-light defect image;

[0009] The low-light defect image is enhanced by illumination based on the pre-trained low-light image enhancement model to obtain an illumination-enhanced defect image.

[0010] Based on the pre-trained defect content restoration model and the defect region occlusion image, the illumination-enhanced defect image is restored to obtain the restored image.

[0011] Among them, the low-light image enhancement model is built based on Retinex theory; the defect content repair model is built based on the application of receptive fields of different sizes.

[0012] Optionally, the defect region masking image used to construct the low-light defect image includes:

[0013] Outlier threshold G based on pixel gradient thFilter defect edge pixels in low-light defect images; where the outlier threshold G of the pixel gradient is... th for:

[0014] G th =G avg +λ G σ G

[0015] Among them, G avg It is the mean pixel gradient of the low-light defect image, σ G λ is the standard deviation of the pixel gradient in a low-light defect image. G It is the preset threshold for the gradient;

[0016] The pixel outlier threshold P of the low-light defect image is obtained from the defect edge pixels of the low-light defect image. th :

[0017] P th =P avg +λ P σ P

[0018] Among them, P avg σ is the mean pixel value of the defect edge in a low-light defect image. P λ is the standard deviation of the pixel values ​​at the defect edge of the low-light defect image. P It is a preset threshold for the pixel values ​​of the defective area;

[0019] Based on the pixel outlier threshold P of the low-light defect image th The low-light defect image is calculated channel by channel, and pixels with values ​​higher than the outlier threshold P of the low-light defect image are identified. th The pixel region is used as the defect region of the low-light defect image, and a defect region masking image of the low-light defect image is generated based on the defect region of the low-light defect image.

[0020] Optionally, the low-light image enhancement model includes an enhancement module E(v t ) and self-calibration module S(x k );

[0021] Enhancement module E(v) t Using an illumination estimation network The construction, specifically:

[0022]

[0023] Self-calibration module S(x) k A self-calibration network is used. The construction, specifically:

[0024]

[0025] Among them, v t It is an enhancement module E(v) t The transition input in round t during the asymptotic process, u t It is an enhancement module E(v) t The residual x in the t-th round of the asymptotic process. t It is an enhancement module E(v) t The illumination in the t-th round during the progressive process; y is the low-light defect image, z k x is the reflection component of the kth round; k Let be the lighting component for the k-th round.

[0026] Optionally, the loss function of the low-light image enhancement model adopts an unsupervised loss function.

[0027]

[0028]

[0029] Where α and β are preset weights, T is the total number of rounds in the progressive process, N is the total number of pixels in the low-light defect image, i is the i-th pixel in the low-light defect image, c is the image channel in the YUV color space of the low-light defect image, and σ is the standard deviation of the Gaussian kernel; s t-1 This is the calibration amount for the (t-1)th round; Let be the illumination component of the i-th pixel in the low-light defect image during the t-th round; Let y be the illumination component of the j-th pixel in the low-light defect image during round t; i，c The weak light observation component of the YUV color space channel of the i-th pixel in the low light defect image; The calibration value for the YUV color space channel of the i-th pixel in the low-light defect image during round t-1; y j，c Let J be the weak light observation component of the YUV color space channel of the j-th pixel in the low light defect image. The calibration amount is the YUV color space channel of the j-th pixel in the low-light defect image during round t-1.

[0030] Optionally, the defect content repair model includes a coarse repair network model, a discriminator, a local repair network model, and a global refinement network model connected in sequence; wherein, the coarse repair network model consists of 8 downsampling and upsampling operations, and information is passed through long skip connections; the discriminator consists of an adversarial generative network; the local repair network model consists of two downsampling operations, four residual blocks, and two upsampling operations, and processes local regions of the input image in a sliding window manner; the global refinement network model is obtained by embedding three attention mechanism modules between the upsampling operations in the local repair network model.

[0031] Optionally, the loss function of the defective content repair model for:

[0032]

[0033] in, The pixel-level loss function for the coarse-grained repair network model:

[0034]

[0035] in, The output image of the coarse-repair network model; I gt M is a defect-free image under normal lighting; M is an image occluded by grayscale defects, where 0 represents valid pixels and 1 represents missing pixels; M r The inverted grayscale image obtained by inverting the pixel value corresponding to M;

[0036] The adversarial generative loss function for the coarse-grained repair network model:

[0037]

[0038]

[0039] in, Let I be the mathematical expectation, where I mer Follows the probability distribution PI mer (I mer ); D is the stitched image of the output image of the coarse-repair network model and the defect region of the illumination-enhanced defect image; D() is the discriminator output of the generative adversarial network; I in To enhance the image of defects through illumination;

[0040] The loss function for the discriminator:

[0041]

[0042] in, Let be the mathematical expectation, where I follows the probability distribution Pdata(I);

[0043] The loss function for the local repair network model is:

[0044]

[0045]

[0046]

[0047]

[0048] in, For the pixel-level loss function of the local repair network model, by... In Replace with Obtain; among them, The output image of the local repair network model; λ tv , λ per and λ sty To preset weights, This is the total change loss function for smoothing the changes in neighboring pixels in the hole region in the local repair network model. The smoothing loss function, defined in the feature space, is used to construct the local repair network model on the VGG-16 network. The style loss function defined on the VGG-16 network of the local insulation network model is computed in the feature space. The i-th layer feature map represents the VGG-16 network; It is a Gram matrix; This is a stitched image of the defect region between the output image of the local repair model and the illumination-enhanced defect image; for The pixel at position (i, j+1);

[0049] To globally refine the loss function of the network model, By In Replace with Obtain; among them, This is the output image of the globally refined network model.

[0050] Optionally, the training sets for the pre-training process of the pre-trained low-light image enhancement model and the pre-trained defect content restoration model are obtained in the following manner: several original mural images are acquired and subjected to low-light processing at different scales to obtain several low-light original mural images, and the several low-light original mural images are segmented to obtain several segmented images; several defects are generated on each segmented image through one or more of random walk, edge erosion and random scatter point generation methods to obtain the training set.

[0051] The process of obtaining the low-light defect image to be repaired includes: acquiring the target image and segmenting it according to a preset size to obtain the low-light defect image to be repaired.

[0052] In a second aspect, the present invention provides a low-light defect image restoration system, comprising:

[0053] The image acquisition module is used to acquire the low-light defect image to be repaired and to construct a defect area masking image of the low-light defect image.

[0054] The illumination enhancement module is used to enhance the illumination of low-light defect images based on a pre-trained low-light image enhancement model, resulting in an illumination-enhanced defect image.

[0055] The defect repair module is used to repair the defect content of the illumination-enhanced defect image based on the pre-trained defect content repair model and the defect region occlusion image, and obtain the repaired image.

[0056] Among them, the low-light image enhancement model is built based on Retinex theory; the defect content repair model is built based on the application of receptive fields of different sizes.

[0057] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described low-light defect image restoration method.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described low-light defect image restoration method.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] This invention discloses a low-light defect image restoration method. First, it enhances the low-light defect image using a pre-trained low-light image enhancement model to obtain an enhanced image. Images acquired in low-light environments undergo this enhancement stage, resulting in better observability of the image's structure and texture, and preparing for subsequent feature extraction for defect restoration. Then, the output of the low-light image enhancement stage is used as input for the defect content restoration stage. Based on a pre-trained defect content restoration model and a defect region occlusion image, the enhanced image undergoes defect content restoration. Furthermore, by applying receptive fields of different sizes, the structure and texture of the image defects are restored, achieving a complete imaging effect. By employing a two-stage restoration strategy of illumination enhancement and defect content restoration, a better low-light defect image restoration effect is obtained. Attached Figure Description

[0061] Figure 1 This is a flowchart of the low-light defect image restoration method according to an embodiment of the present invention.

[0062] Figure 2 This is an example diagram of defect area masking image extraction according to an embodiment of the present invention.

[0063] Figure 3 This is a schematic diagram of the low-light image enhancement model and defect content repair model according to an embodiment of the present invention.

[0064] Figure 4 These are example images showing the results of low-light defect image restoration under various defect types according to embodiments of the present invention.

[0065] Figure 5 This is an example image showing the restoration result of a large low-light defect image according to an embodiment of the present invention.

[0066] Figure 6 This is a schematic diagram illustrating the evaluation results of the low-light defect image restoration method according to an embodiment of the present invention and existing methods.

[0067] Figure 7 This is a structural block diagram of a low-light defect image restoration system according to an embodiment of the present invention. Detailed Implementation

[0068] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0069] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0070] The present invention will now be described in further detail with reference to the accompanying drawings:

[0071] See Figure 1 In one embodiment of the present invention, a method for restoring low-light defect images is provided, which can be used to restore damaged ancient mural images under low light conditions, and achieve a better mural restoration effect.

[0072] Specifically, the low-light defect image restoration method includes the following steps:

[0073] S1: Obtain the low-light defect image to be repaired and construct a defect area masking image of the low-light defect image.

[0074] S2: Apply illumination enhancement to the low-light defect image based on the pre-trained low-light image enhancement model to obtain an illumination-enhanced defect image.

[0075] S3: Based on the pre-trained defect content repair model and the defect region occlusion image, perform defect content repair on the illumination-enhanced defect image to obtain the repaired image.

[0076] Among them, the low-light image enhancement model is built based on Retinex theory; the defect content repair model is built based on the application of receptive fields of different sizes.

[0077] This invention discloses a low-light defect image restoration method. First, it enhances the low-light defect image using a pre-trained low-light image enhancement model to obtain an enhanced image. Images acquired in low-light environments undergo this enhancement stage, resulting in better observability of the image's structure and texture, and preparing for subsequent feature extraction for defect restoration. Then, the output of the low-light image enhancement stage is used as input for the defect content restoration stage. Based on a pre-trained defect content restoration model and a defect region occlusion image, the enhanced image undergoes defect content restoration. Furthermore, by applying receptive fields of different sizes, the structure and texture of the image defects are restored, achieving a complete imaging effect. By employing a two-stage restoration strategy of illumination enhancement and defect content restoration, a better low-light defect image restoration effect is obtained.

[0078] In one possible implementation, the defect region masking image used to construct the low-light defect image includes:

[0079] Outlier threshold G based on pixel gradient th Filter defect edge pixels in low-light defect images; where the outlier threshold G of the pixel gradient is... th for:

[0080] G th =G avg +λ G σ G

[0081] Among them, G avg It is the mean pixel gradient of the low-light defect image, σ G λ is the standard deviation of the pixel gradient in a low-light defect image. G It is the preset threshold of the gradient.

[0082] The pixel outlier threshold P of the low-light defect image is obtained from the defect edge pixels of the low-light defect image. th :

[0083] P th =P avg +λ P σ P

[0084] Among them, P avg σ is the mean pixel value of the defect edge in a low-light defect image. P λ is the standard deviation of the pixel values ​​at the defect edge of the low-light defect image. P It is a preset threshold for the pixel values ​​of the defective area.

[0085] Based on the pixel outlier threshold P of the low-light defect image th The low-light defect image is calculated channel by channel, and pixels with values ​​higher than the outlier threshold P of the low-light defect image are identified. th The pixel region is used as the defect region of the low-light defect image, and a defect region masking image of the low-light defect image is generated based on the defect region of the low-light defect image.

[0086] Specifically, defect regions in low-light defect images can be manually labeled for the repair network to identify. However, considering that manual labeling of defect regions is slow in real-world applications, and the accuracy of the labeled regions directly affects the model's repair performance, a gradient method can be used to automatically identify defect regions in low-light defect images. Since the colors of defect regions do not belong to the original image, the color range of defect regions will be large. Therefore, the boundaries of defect regions can be obtained by filtering out outliers in pixel gradients.

[0087] After obtaining the boundary of the defect region, the pixel value of the boundary is calculated. It can be assumed that the pixel values ​​of the defect region are similar and outliers. Finally, the low-light defect image is calculated by channel, and the region with pixel value higher than the pixel value threshold is defined as the defect region, thereby generating a defect region occlusion image describing the defect region.

[0088] By using the gradient method to calculate the pixels of the image, and by finding outliers that are different from the colors of the original image, a masking image of the defect area is obtained to prepare for defect repair.

[0089] See Figure 2 The results of image extraction for defect-covered areas are shown. Figures (a1), (a2), and (a3) ​​are three low-light mural images containing defects, and Figures (b1), (b2), and (b3) are the three mural images at λ. G =4, λ P The defect region extraction results when λ=3 are shown in Figures (c1), (c2), and (c3), which are the three mural images at λ=3 respectively. G =4, λ PThe defect region extraction results when λ = 2 are shown in Figures (d1), (d2), and (d3), which are the three mural images at λ = 2. G =5, λ P The defect region extraction results are shown when λ = 1.5. The results differ for the three sets of parameters, adapting to different application scenarios. The appropriate parameter λ should be determined based on the actual repair requirements. G With λ P And Figures (d1), (d2), and (d3) are three mural images at λ. G =5, λ P When the value is 1.5, the result of extracting the defect area is used as the repair result when masking the defect description.

[0090] This invention employs a two-stage restoration strategy: illumination enhancement and defect content restoration. Low-light defect images acquired in low-light environments first undergo illumination enhancement, resulting in better observability of the image's structure and texture, and preparing for subsequent feature extraction for defect restoration. Then, the output of the low-light image enhancement stage—the illuminated defect image—is used as input for the defect content restoration stage. By applying receptive fields of different sizes, the structure and texture of the image defects are restored, achieving a complete imaging effect.

[0091] See Figure 3 In one possible implementation, the low-light image enhancement model includes an enhancement module E(v t ) and self-calibration module S(x k Enhancement module E(v) t Using an illumination estimation network The construction, specifically:

[0092]

[0093] Self-calibration module S(x) k A self-calibration network is used. The construction, specifically:

[0094]

[0095] Among them, v t It is an enhancement module E(v) t The transition input in round t during the asymptotic process, u t It is an enhancement module E(v) t The residual x in the t-th round of the asymptotic process. t It is an enhancement module E(v) t The illumination in the t-th round during the progressive process; y is the low-light defect image, z k x is the reflection component of the kth round; k Let be the lighting component for the k-th round.

[0096] Specifically, the low-light image enhancement model was constructed with reference to the classic Re-tinex theory. This implies a correlation between the weak-light observation y and the desired sharp image z. Here, x represents the illumination component, which is the core optimization objective in the illumination enhancement stage. According to Retinex theory, a good estimation of illumination leads to better visual observation imaging results. This implementation employs an illumination estimation network that shares the structure and learnable parameter θ. The enhancement module E is constructed, and to ensure that the outputs of different rounds in the incremental optimization process converge to the same state, a learnable parameter is used. network The self-calibration module S gradually corrects the input of each round of enhancement module.

[0097] Optionally, an unsupervised loss function was used to better train the low-light image enhancement model.

[0098]

[0099]

[0100] Where α and β are preset weights, T is the total number of rounds in the progressive process, N is the total number of pixels in the low-light defect image, i is the i-th pixel in the low-light defect image, c is the image channel in the YUV color space of the low-light defect image, and σ is the standard deviation of the Gaussian kernel; s t-1 This is the calibration amount for the (t-1)th round; Let be the illumination component of the i-th pixel in the low-light defect image during the t-th round; Let y be the illumination component of the j-th pixel in the low-light defect image during round t; i，c The weak light observation component of the YUV color space channel of the i-th pixel in the low light defect image; The calibration value for the YUV color space channel of the i-th pixel in the low-light defect image during round t-1; y j，c Let J be the weak light observation component of the YUV color space channel of the j-th pixel in the low light defect image. This is the calibration value for the YUV color space channel of the j-th pixel in the low-light defect image during round t-1. In this embodiment, the total number of optimization enhancement rounds is set to 6, the standard deviation of the Gaussian kernel σ = 0.1, and the weights α and β are set to 1.5 and 1, respectively.

[0101] The defect content restoration model was constructed using a strategy of applying different receptive fields and multiple loss functions. This is because networks with larger receptive fields are better at restoring the overall structure and larger color blocks, while networks with smaller receptive fields are better at restoring local, detailed textures.

[0102] See you again Figure 3 In one possible implementation, the defect content repair model includes a coarse repair network model, a discriminator, a local repair network model, and a global refinement network model connected in sequence. The coarse repair network model consists of eight downsampling and upsampling operations, with information passed through long skip connections. The discriminator is composed of a generative adversarial network. The local repair network model consists of two downsampling operations, four residual blocks, and two upsampling operations, processing local regions of the input image using a sliding window approach. The global refinement network model is obtained by embedding three attention mechanism modules between the upsampling operations in the local repair network model.

[0103] Optionally, the loss function of the defective content repair model for:

[0104]

[0105] in, The pixel-level loss function for the coarse-grained repair network model:

[0106]

[0107] in, The output image of the coarse-repair network model; I gt M is a defect-free image under normal lighting; M is an image occluded by grayscale defects, where 0 represents valid pixels and 1 represents missing pixels; M r The inverted grayscale image is obtained by inverting the pixel value corresponding to M.

[0108] The adversarial generative loss function for the coarse-grained repair network model:

[0109]

[0110]

[0111] in, Let I be the mathematical expectation, where I mer Follows the probability distribution PI mer (I mer ); D is the stitched image of the output image of the coarse-repair network model and the defect region of the illumination-enhanced defect image; D() is the discriminator output of the generative adversarial network; I in To enhance the image of defects by illumination.

[0112] The loss function for the discriminator:

[0113]

[0114] in, Let be the mathematical expectation, where I follows the probability distribution Pdata(I).

[0115] The loss function for the local repair network model is:

[0116]

[0117]

[0118]

[0119]

[0120] in, For the pixel-level loss function of the local repair network model, by... In Replace with Obtain; among them, The output image of the local repair network model; λ tv , λ per and λ sty To preset weights, This is the total change loss function for smoothing the changes in neighboring pixels in the hole region in the local repair network model. The smoothing loss function, defined in the feature space, is used to construct the local repair network model on the VGG-16 network. The style loss function defined on the VGG-16 network of the local insulation network model is computed in the feature space. The i-th layer feature map represents the VGG-16 network; It is a Gram matrix; This is a stitched image of the defect region between the output image of the local repair model and the illumination-enhanced defect image; for The pixel at position (i, j+1).

[0121] To globally refine the loss function of the network model, By In Replace with Obtain; among them, This is the output image of the globally refined network model.

[0122] Specifically, in order to roughly restore the overall structure of the damaged image, the output of the illumination enhancement stage, i.e., the illumination enhancement defect image, is first used as the input image I to be repaired by the network in this stage. in It, along with the binary mask image describing the missing region, i.e., the grayscale defect region masking image M, is input into a coarse-grained repair network model Net with a large receptive field. C In the middle, the grayscale defect region occlusion image M is randomly generated during training, but in practical applications, it can be obtained through gradient calculation or manual annotation.

[0123] Net C It consists of 8 downsampling and upsampling operations, which pass information between the encoder and decoder via long skip connections to recover information lost during downsampling. C The output is Furthermore, a discriminator was added to enhance the realism of the restoration. This discriminator takes the real image and the restored image as input and produces a value of size [missing information]. The output is a two-dimensional feature map. In this feature map, each element is used to distinguish whether the content it represents is real or fake.

[0124] After coarse image restoration, further detailed restoration of the image's local structure and texture is required. The coarsely restored image is then input into a local restoration network model Net with a smaller receptive field. L Net L Composed of two downsampling operations, four residual blocks, and two upsampling operations, this local inpainting network processes local regions of the input image using a sliding window approach, effectively restoring local information in murals. The pixel-level loss of this local inpainting network model is... Its and The calculation process is the same, only the formula is changed. Replace with NetL output The merging method is also similar to that in the coarse repair step. The merging method is the same, except that in the formula... Replace with Net L Output Furthermore, attention modules are added to the local repair network model, with each module having a resolution of 16×16, 32×32, and 64×64. Additionally, a total variation loss is used to smooth the changes in neighboring pixels in the hole region. And the smoothing loss defined on VGG-16, computed in the feature space. and style loss They and Combined target loss This will achieve better recovery results.

[0125] In this embodiment, the preset weight λ tv , λ per and λ sty They were set to 0.1, 0.05, and 120 respectively.

[0126] After refining and repairing local textures and structures, the image exhibits good texture restoration results. To achieve a better overall visual effect, the output image of the local restoration network model is input into the global refinement network model Net. G Net G In Net L Three attention modules have been added to .NET, and the attention mechanism can bring better results. G Training loss function Calculation and Similarly, you only need to... Replace with Finally, the training loss for the entire image inpainting stage can be obtained. It is the sum of the losses of the three repair subnetworks and the discriminator loss:

[0127] Based on the application scenario of mural restoration, this invention focuses on the imaging observation effect after mural image restoration, taking into account both the enhancement of mural image illumination and the repair of defects. It uses a two-stage restoration model and employs multiple receptive field processing strategies in the restoration stage. At the same time, it incorporates a discriminator and an attention module, which has a more efficient processing speed and restoration imaging effect compared to traditional mural image restoration models.

[0128] See Figure 4 The images illustrate the results of low-light defect image restoration using the method of the present invention for various defect types. Figures (a1), (a2), and (a3) ​​show the reference image, low-light defect image, and restored image for a large area of ​​dirt (dusk defect type), respectively; Figures (b1), (b2), and (b3) show the reference image, low-light defect image, and restored image for a jelly defect type simulating bacterial erosion, respectively; Figures (c1), (c2), and (c3) show the reference image, low-light defect image, and restored image for a droplet defect type simulating splash-type defects, respectively; Figures (d1), (d2), and (d3) show the reference image, low-light defect image, and restored image for a block defect type simulating large-area peeling of a mural, respectively; and Figures (e1), (e2), and (e3) show the reference image, low-light defect image, and restored image for a line defect type simulating scratch defects, respectively. It is evident that the low-light defect image restoration method of the present invention can achieve good image restoration results for various defect types.

[0129] In one possible implementation, the training set in the pre-training process of the pre-trained low-light image enhancement model and the pre-trained defect content repair model is obtained by: acquiring several original mural images and performing low-light processing at different scales to obtain several low-light original mural images, and segmenting the several low-light original mural images to obtain several segmented images; generating several defects on each segmented image through one or more of random walk, edge erosion and random scatter point generation methods to obtain the training set.

[0130] Specifically, in this embodiment, 39, 42, and 48 complete ancient mural image samples were selected from publicly available datasets based on the murals of Guangsheng Temple in Hongtong, the Water and Land Murals of Princess Temple in Fanshi, and the Yongle Palace Murals in Ruicheng, respectively. The original mural image samples underwent low-light processing, reducing the image brightness to 55%, 37%, and 12% of the original brightness, respectively, to simulate images collected in real-world application scenarios. Simultaneously, the low-light processed mural images were segmented into 256×256 pixel .png images to expand the dataset, ultimately resulting in 123,299 low-light mural image data. Furthermore, various artificial occlusion methods, such as random walk, edge erosion, and randomized scattering, were used to generate various types of artificial occlusion to simulate different types of mural damage in real-world application scenarios, thereby improving the model's generalization ability and robustness. The low-light mural images and the various types of occluded images were used together as the training dataset for the model.

[0131] During training, the two models are coupled, with the optimizer adjusting network parameters based on the loss to minimize the difference between the generated output and the target. However, if both networks update their parameters simultaneously, the input to the defect content restoration stage network becomes unstable due to the continuous changes in the illumination enhancement stage network parameters, increasing the learning difficulty of the image restoration stage. Furthermore, considering the different complexities of the two networks, the number of training samples required for optimization differs. Therefore, the model uses an alternating parameter update strategy. In each training epoch, the dataset is divided into several groups of 60 data points each. The first 6 data points in each group are used for network optimization in the illumination enhancement stage, while simultaneously freezing the parameters of the defect content restoration stage network and only updating the parameters of the illumination enhancement stage network. The remaining 54 data points in the group are used for network optimization in the defect content restoration stage, updating the parameters of the defect content restoration stage network while freezing the parameters of the illumination enhancement stage network.

[0132] Furthermore, the loss of the entire two-stage repair model can be expressed as follows:

[0133]

[0134] Where, λ RSet to 1 during the defect content repair phase and to 0 during the lighting enhancement phase; λ E Set to 0 in the defect content repair phase and to 1 in the lighting enhancement phase.

[0135] In one possible implementation, obtaining the low-light defect image to be repaired includes: acquiring a target image and segmenting it according to a preset size to obtain the low-light defect image to be repaired.

[0136] Specifically, in this embodiment, to improve identification and restoration, a strategy is adopted to segment the large target image into 256×256 pixel sub-images for repair, and then stitch the repaired sub-images together. Compared to directly processing the large target image, this strategy can segment and restore the damaged areas separately, resulting in better defect identification and feature extraction, and ultimately, better restoration performance.

[0137] See Figure 5 Figure (a) shows an image of a large mural with low light defects, Figure (b) shows the corresponding image of the extracted defect area covered, and Figure (c) shows an image of the large mural after restoration. It can be seen that a good restoration effect has been achieved.

[0138] This invention presents a low-light defect image restoration method that achieves superior performance compared to existing methods under various illumination intensities and defect types, demonstrating significant advantages in LPIPS, PSNR, and SSIM evaluation metrics. Applied to the field of mural restoration, the improved restoration imaging results effectively help researchers extract information about ancient religions, ceremonies, and folk customs from the structure and texture of murals, promoting multifaceted cultural research. Furthermore, this invention allows for structural modifications to the prediction head of the final model to achieve richer and more diverse tasks, such as low-light image enhancement and damaged image restoration.

[0139] In one possible implementation, the invention is further illustrated through experimental verification:

[0140] 1. Experimental conditions: The experiment was conducted on a single TITAN GPU server with Ubuntu 18.04 operating system. The time to repair a single 256×256 pixel mural with low light defects was approximately 0.03 seconds.

[0141] 2. Experimental Results Analysis: The evaluation indicators used in the experiment were LPIPS, PSNR, and SSIM. The experiment was validated using 4581 256×256 images of defective low-light murals. See [link to relevant documentation]. Figure 6 The results show a comparison of the final evaluation index data with those obtained using other methods. Figure 6Figure (a1) shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 55% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the LIPIS evaluation metric. Figure (a2) shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 55% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the PSNR evaluation metric. Figure (a3) ​​shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 55% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the SSIM evaluation metric. Figure (b1) shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 12% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the LIPIS evaluation index. Figure (b2) shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 12% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the PSNR evaluation index. Figure (b3) shows the comparison of restoration results with other restoration models after processing the low light of the test dataset to 12% of the original brightness and applying random areas of 5%-20%, 20%-35%, and 35%-50% of the original image under the SSIM evaluation index. In this context, LG represents the restoration model from the paper “W.Quan,R.Zhang,Y.Zhang,Z.Li,J.Wang,and D.-M.Yan,Image Inpainting With Local and GlobalRefinement,” in IEEE Transactions on Image Processing, vol.31, pp.2405-2420, 2022.DOI:10.1109 / TIP.2022.3152624.”, MADF represents the restoration model from the paper “M.Zhu,J.Huang,andX.Guo,Image Inpainting by End-to-End Cascaded Refinement With MaskAwareness,” in IEEE Transactions on Image Processing, vol.30, 2021, pp.4855-4866.DOI:10.1109 / TIP.2021.3076310.”, and PEN represents the restoration model from the paper “Zeng,Y.,Fu,J.,Chao,H.The inpainting model is from the paper "J.Yu,Z.Lin,J.Yang,X.Shen,X.Lu,andTSHuang,Generative Image Inpainting with Contextual Attention," in Proceedings of the 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp.5505-5514, doi:10.1109 / CVPR.2018.00577," and the SCI is from the paper "Ma, et al.,Toward Fast, Flexible,andRobust Low-Light Image Enhancement," in [reference needed]. The model is described in the 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5627-5636. It is evident that this invention achieved good results across all three evaluation metrics.

[0142] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0143] See Figure 7 In another embodiment of the present invention, a low-light defect image restoration system is provided, which can be used to implement the above-mentioned low-light defect image restoration method. Specifically, the low-light defect image restoration system includes an image acquisition module, an illumination enhancement module, and a defect restoration module.

[0144] The system comprises the following modules: an image acquisition module for acquiring the low-light defect image to be repaired and constructing a defect region occlusion image; an illumination enhancement module for enhancing the low-light defect image using a pre-trained low-light image enhancement model to obtain an illuminated defect image; and a defect repair module for repairing the defect content of the illuminated defect image using a pre-trained defect content repair model and the defect region occlusion image to obtain a repaired image. The low-light image enhancement model is based on Retinex theory, and the defect content repair model is constructed using a method that applies receptive fields of different sizes.

[0145] All relevant content of each step involved in the aforementioned embodiments of the low-light defect image restoration method can be referenced to the functional description of the corresponding functional module of the low-light defect image restoration system in the embodiments of the present invention, and will not be repeated here.

[0146] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0147] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a low-light defect image restoration method.

[0148] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the low-light defect image restoration method in the above embodiments.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A low-light-defect image inpainting method, characterized by, include: Obtain the low-light defect image to be repaired, and construct a defect area masking image of the low-light defect image; The low-light defect image is enhanced by illumination based on the pre-trained low-light image enhancement model to obtain an illumination-enhanced defect image. Based on the pre-trained defect content restoration model and the defect region occlusion image, the illumination-enhanced defect image is restored to obtain the restored image. Among them, the low-light image enhancement model is built based on Retinex theory; the defect content repair model is built based on the application of receptive fields of different sizes. The defect region masking image used to construct the low-light defect image includes: Outlier threshold of pixel gradient , screening the defect edge pixels of the low light defect image; wherein the outlier threshold of pixel gradient is: wherein is the mean of the pixel gradients of the low light defect image, is the standard deviation of the pixel gradients of the low light defect image, is a preset threshold of the gradient. The pixel outlier threshold of the low-light defect image is obtained based on the defect edge pixels. : in, It is the mean pixel value of the defect edge in the low-light defect image. It is the standard deviation of the pixel values ​​at the defect edge in the low-light defect image. It is a preset threshold for the pixel values ​​of the defective area; Based on the pixel outlier threshold of low-light defect images The low-light defect image is calculated channel by channel, and pixels with values ​​higher than the outlier threshold of the low-light defect image are identified. The pixel region is used as the defect region of the low-light defect image, and a defect region masking image of the low-light defect image is generated based on the defect region of the low-light defect image. The low-light image enhancement model includes an enhancement module. and self-calibration module ; Enhancement Module Using an illumination estimation network The construction, specifically: Self-calibration module Employing a self-calibration network The construction, specifically: in, It is an enhancement module The first step in the gradual process t Wheel conversion input, It is an enhancement module The first step in the gradual process t The residual of the wheel, It is an enhancement module The first step in the gradual process t The light of the wheel; It is a low-light defect image. For the first k The reflected component of the wheel; For the first k The lighting components of the wheel; The defect content repair model includes a coarse repair network model, a discriminator, a local repair network model, and a global refinement network model connected in sequence. The coarse repair network model consists of eight downsampling and upsampling operations, and information is passed through long skip connections. The discriminator is composed of a generative adversarial network. The local repair network model consists of two downsampling operations, four residual blocks, and two upsampling operations, and processes local regions of the input image in a sliding window manner. The global refinement network model is obtained by embedding three attention mechanism modules between the upsampling operations in the local repair network model.

2. The low-light defect image restoration method according to claim 1, characterized in that, The loss function of the low-light image enhancement model adopts an unsupervised loss function. : in, and It is a preset weight. T For the total number of rounds in the gradual process, N The total number of pixels in the low-light defect image. i The first low-light defect image i 1 pixel, c For low-light defect images, the image channels are located in the YUV color space. It is the standard deviation of the Gaussian kernel; For the first The calibration amount of the wheel; For the first t Low-light defect image in wheel i The illumination component of each pixel; For the first t Low-light defect image in wheel j The illumination component of each pixel; For low-light defect images i The low-light observation component of each pixel's YUV color space channel; For the first Low-light defect image in wheel i The calibration amount of each pixel's YUV color space channel; For low-light defect images j The low-light observation component of each pixel's YUV color space channel; For the first Image of low-light defects in wheel j The calibration amount of each pixel's YUV color space channel.

3. The low-light defect image restoration method according to claim 1, characterized in that, The loss function of the defective content repair model for: + + + + in, The pixel-level loss function for the coarse-grained repair network model: in, The output image of the coarse-repair network model; An image with no defects and normal illumination; To mask the grayscale defect area of ​​the image, where 0 represents a valid pixel and 1 represents a missing pixel; The inverted grayscale image obtained by inverting the pixel value corresponding to M; The adversarial generative loss function for the coarse-grained repair network model: in, Let be the mathematical expectation, where, Follows probability distribution ; This is a stitched image of the defect region between the output image of the coarse repair network model and the illumination-enhanced defect image; To generate the discriminator output of the adversarial network; To enhance the image of defects through illumination; The loss function for the discriminator: in, Let be the mathematical expectation, where, Follows probability distribution ; The loss function for the local repair network model is: in, For the pixel-level loss function of the local repair network model, by... In Replace with Obtain; among them, The output image of the local repair network model; , and To preset weights, This is the total change loss function for smoothing the changes in neighboring pixels in the hole region in the local repair network model. The smoothing loss function, defined in the feature space, is used to construct the local repair network model on the VGG-16 network. The style loss function defined on the VGG-16 network of the local insulation network model is computed in the feature space. The first representing the VGG-16 network Layer feature map; It is a Gram matrix; This is a stitched image of the defect region between the output image of the local repair model and the illumination-enhanced defect image; No. The position in pixels; To globally refine the loss function of the network model, By In Replace with Obtain; among them, This is the output image of the globally refined network model.

4. The low-light defect image restoration method according to claim 1, characterized in that, The training sets for the pre-training process of the pre-trained low-light image enhancement model and the pre-trained defect content repair model are obtained in the following manner: several original mural images are acquired and subjected to low-light processing at different scales to obtain several low-light original mural images; the several low-light original mural images are segmented to obtain several segmented images; several defects are generated on each segmented image through one or more of the following methods: random walk, edge erosion, and random scatter point generation, to obtain the training set. The process of obtaining the low-light defect image to be repaired includes: acquiring the target image and segmenting it according to a preset size to obtain the low-light defect image to be repaired.

5. A low-light defect image restoration system based on the low-light defect image restoration method of claim 1, characterized in that, include: The image acquisition module is used to acquire the low-light defect image to be repaired and to construct a defect area masking image of the low-light defect image. The illumination enhancement module is used to enhance the illumination of low-light defect images based on a pre-trained low-light image enhancement model, resulting in an illumination-enhanced defect image. The defect repair module is used to repair the defect content of the illumination-enhanced defect image based on the pre-trained defect content repair model and the defect region occlusion image, and obtain the repaired image. Among them, the low-light image enhancement model is built based on Retinex theory; The defect content repair model is constructed based on the application of receptive fields of different sizes.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the low-light defect image restoration method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the low-light defect image restoration method as described in any one of claims 1 to 4.

Images