Image stain repair method and device based on deep learning, medium and equipment

Abstract

The application provides a kind of based on deep learning's image dirty repair method, device, medium and equipment, method includes: constructing glass surface particle stain image dataset;Lightweight deep learning network and combination loss function are constructed;Iterative training is carried out to lightweight deep learning network using dataset and combination loss function, until iterative convergence, output target image dirty repair model;Current to be repaired image of camera module is repaired using target image dirty repair model, and the repair image of removing dirt is obtained;In this way, manual marking and repair of image stain area are not required, errors caused by manual operation are avoided, and the repair efficiency of glass surface particle stain is improved;The network can reduce the computing cost through lightweight modification, adapt to the real-time processing needs of mobile camera equipment, and the combination loss function can constrain the repair effect from the aspects of pixels, gradients and multidimensional structure, to obtain a repair image with better imaging quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, medium and device for image dirt repair based on deep learning. Background Technology

[0002] During the production, assembly, and actual use of mobile device camera modules, the glass cover of the lens module is susceptible to environmental influences, resulting in particle-on-glass (POG) contamination. These particles adhere to the glass cover surface and create speckle artifacts in the image during imaging, thus affecting the final image quality.

[0003] Currently, blemishes introduced by POGs in images are typically repaired using dedicated image processing software. For example, the blemish repair algorithms used in Adobe Photoshop mainly include content-based fill, texture-based inpainting, and approximate matching inpainting algorithms. Content-based fill automatically fills in the blemish-damped areas based on the surrounding image content; texture-based inpainting reconstructs and repairs blemish areas using local image texture information; and approximate matching inpainting replaces and fills in blemish areas by searching for similar image patches within the image.

[0004] However, the aforementioned restoration algorithms in existing technologies require professional operators to manually mark the blemish areas to achieve the desired restoration effect, which is time-consuming and makes it difficult to achieve automated blemish identification and real-time restoration. In mobile device shooting scenarios, users have a high demand for real-time image restoration during the viewfinder preview stage, and traditional restoration solutions cannot be adapted to this application scenario. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method, apparatus, medium, and device for image dirt repair based on deep learning, in order to solve or partially solve the technical problem that image stains caused by POG contamination of lens cover plates require manual marking and processing, which is time-consuming and cannot meet the needs of automated, real-time preview repair in mobile device shooting scenarios.

[0006] A first aspect of the present invention provides a deep learning-based method for image dirt restoration, the method comprising: A glass surface particle stain image dataset is constructed; the glass surface particle stain image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; A lightweight deep learning network and a combined loss function are constructed; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term. The lightweight deep learning network is iteratively trained using the glass surface particle stain image dataset and the combined loss function until iterative convergence, and the target image stain repair model is output. The target image dirt repair model is used to repair the current image to be repaired from the camera module, resulting in a repaired image with dirt removed.

[0007] In the above scheme, constructing the image dataset of glass surface particle stains includes: Multiple camera modules with glass surface particle stains were used to capture images of the background panel, resulting in multiple initial images of the glass surface particle stains. For each initial image of a glass surface particle stain, glass surface particle stain features are extracted to obtain a glass surface particle stain template image. Multiple uncontaminated image samples are obtained, and multiple template images of glass surface particle stains and multiple uncontaminated image samples are randomly fused to obtain the glass surface particle stain image dataset.

[0008] In the above scheme, the step of extracting glass surface particle stain features from each initial image of the glass surface particle stain to obtain a glass surface particle stain template image includes: For each initial image of the glass surface particle stain, the initial image of the glass surface particle stain is binarized to obtain the corresponding binarized image; The binarized image is traversed using pixel adjacency rules to obtain multiple foreground connected components; By filtering through the multiple foreground connected regions, the glass surface particle stain connected regions are obtained; Determine the smallest closed circle of the connected domain of the particulate contaminants on the glass surface; The neighborhood of the glass surface particle stain connected region is determined based on the smallest closed circle of the glass surface particle stain connected region; the neighborhood of the glass surface particle stain connected region is an annular area obtained by extending outward from the smallest closed circle by a preset distance, the preset distance being the radius of the smallest closed circle; The mean gray value of each pixel in the neighborhood is determined. For each pixel in the connected region of the glass surface particle stain, the mean gray value is subtracted from the gray value of each pixel in the connected region of the glass surface particle stain to obtain the corresponding gray value difference. The negative value of each gray value is then calculated to obtain the template image of the glass surface particle stain.

[0009] In the above scheme, multiple template images of glass surface particle stains and multiple uncontaminated image samples are randomly fused to obtain the glass surface particle stain image dataset, including: Each of the glass surface particle stain template image samples is randomly superimposed onto any uncontaminated image to obtain a contaminated image sample with particle stains. The contaminated image sample is paired with the corresponding uncontaminated image sample to obtain the image sample; multiple image samples are determined as the glass surface particle stain image dataset.

[0010] In the above scheme, the lightweight deep learning network includes: an input layer, an encoding layer, a decoding layer, and an output layer; During training, the input layer is used to receive the image samples and transmit the image samples to the encoding layer; The encoding layer includes multiple sequentially connected first convolutional modules and mean pooling layers. The first convolutional module includes a first convolutional layer, a first batch of normalization layers, and a first activation layer. Each time an image sample passes through a mean pooling layer, its size is reduced to half of its original size. The number of output channels of each level of the first convolutional module increases progressively to extract multi-scale deep features of blemishes and output multi-scale blemish feature maps. The decoding layer includes a three-level decoding fusion branch, and each level of the decoding fusion branch includes a second convolutional module. The second convolutional module includes a second convolutional layer, a second batch normalization layer, and a second activation layer. The decoding layer is used to perform progressive upsampling resolution restoration on the multi-scale taint deep feature map output by the encoding layer to obtain a taint fusion feature map. The output layer includes a third convolutional layer and a third activation layer. The output layer is used to perform feature aggregation mapping on the taint fusion feature map and output a pixel residual map. The pixel residual map is used to repair the tainted image samples in the image samples to obtain the repaired image samples corresponding to the tainted image samples.

[0011] In the above scheme, the combined loss function is determined in the following way: According to the formula Determine the loss function L ;in, The For robust pixel loss terms, The weight coefficients corresponding to the robust pixel loss term, the For the gradient consistency loss term, The weight coefficients corresponding to the gradient consistency loss term, the For the structural similarity loss term, These are the weighting coefficients corresponding to the structural similarity loss term.

[0012] In the above scheme, the training set also includes a test set; the step of iteratively training the lightweight deep learning network using the training set and the combined loss function until iterative convergence, and outputting a target image dirt repair model, includes: Data augmentation is performed on contaminated and uncontaminated image samples in the training set to align the pixels in the augmented contaminated and uncontaminated image samples. The enhanced contaminated image sample is input into the lightweight deep learning network to obtain a residual feature map. The residual feature map and the contaminated image sample are then fused to obtain a reconstructed and repaired image. The loss value between the reconstructed and restored image and the uncontaminated image sample is calculated using the combined loss function; The network weights are optimized by backpropagation based on the loss value until the iteration converges and training stops, and the initial image dirt restoration model is output. The generalization performance of the image dirt restoration model is tested using the test set. If the performance test results meet the requirements, the initial image dirt restoration model is determined as the target image dirt restoration model.

[0013] A second aspect of the present invention provides an image dirt repair device based on deep learning, the device comprising: The first construction unit is used to construct a glass surface particle stain image dataset; the glass surface particle stain image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; The second building unit is used to build a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term. The training unit is used to iteratively train the lightweight deep learning network using the glass surface particle stain image dataset and the combined loss function until the iteration converges, and outputs the target image stain repair model. The repair unit is used to repair the current image to be repaired of the camera module using the target image dirt repair model, so as to obtain a repaired image with dirt removed.

[0014] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0015] A fourth aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method described in any of the first aspects.

[0016] This invention provides a method, apparatus, medium, and device for image contamination repair based on deep learning. The method includes: constructing a glass surface particle contamination image dataset; the glass surface particle contamination image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; constructing a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term; iteratively training the lightweight deep learning network using the glass surface particle contamination image dataset and the combined loss function until iterative convergence, outputting a target image contamination repair model; using the target image contamination repair model to repair the current image to be repaired by a camera module, obtaining a repaired image with the contamination removed; thus, manual marking and repair of contamination areas in the image is unnecessary, avoiding errors caused by manual operation and improving the repair efficiency of glass surface particle contamination; the lightweight modified network can reduce computational overhead, adapting to the real-time processing requirements of mobile camera devices, while the combined loss function constrains the repair effect from the pixel, gradient, and multi-dimensional structural levels, preserving the original texture details of the image while removing POG particle contamination, resulting in a repaired image with better imaging quality. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A schematic flowchart of an image dirt restoration method based on deep learning according to an embodiment of the present invention is shown; Figure 2 A schematic diagram of an initial image of a qualified glass surface with particulate stains obtained by photographing a blue background, according to an embodiment of the present invention, is shown. Figure 3 A schematic diagram of an initial image of a qualified glass surface with particulate stains taken against a white background, according to an embodiment of the present invention, is shown. Figure 4 A schematic diagram of an initial image of a qualified glass surface with particulate stains taken against an outdoor blue sky background, according to an embodiment of the present invention, is shown. Figures 5 to 20 A schematic diagram of contamination image samples under different backgrounds, different particle morphologies, or different superposition intensities is shown according to an embodiment of the present invention. Figure 21 A schematic diagram of an image dirt repair device based on deep learning according to an embodiment of the present invention is shown. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0019] This invention provides a deep learning-based method for image contamination repair, such as... Figure 1 As shown, the method mainly includes the following steps: S110, construct a glass surface particle stain image dataset; the glass surface particle stain image dataset contains multiple image samples, each image sample includes a contaminated image sample and a corresponding uncontaminated image sample.

[0020] This invention requires the construction of a glass surface particle stain image dataset, which will then be used to train a lightweight deep learning network. The glass surface particle stain image dataset contains multiple image samples, each including a contaminated image sample and a corresponding uncontaminated image sample.

[0021] In one implementation, constructing a dataset of particle stain images on a glass surface includes: Multiple camera modules with glass surface particle stains were used to capture images of the background panel, resulting in multiple initial images of the glass surface particle stains. For each initial image of a glass surface particle stain, the features of the glass surface particle stain are extracted to obtain a glass surface particle stain template image sample. Multiple uncontaminated image samples are obtained, and multiple glass surface particle stain template image samples and multiple uncontaminated image samples are randomly fused to obtain a glass surface particle stain image dataset.

[0022] In one embodiment, the initial image of each glass surface particle stain is used to extract glass surface particle stain features to obtain a glass surface particle stain template image sample, including: For each initial image of a particle stain on the glass surface, the initial image of the particle stain on the glass surface is binarized to obtain the corresponding binarized image; The binarized image is traversed using pixel adjacency rules to obtain multiple foreground connected components; By filtering through multiple foreground connected regions, the connected regions of glass surface particle stains are obtained; Determine the minimum closed circle of the connected domain of particulate contaminants on the glass surface; The neighborhood of the connected region of the particle stain on the glass surface is determined based on the smallest closed circle of the connected region of the particle stain on the glass surface; the neighborhood of the connected region of the particle stain on the glass surface is an annular area obtained by extending outward by a preset distance based on the smallest closed circle, and the preset distance is the radius of the smallest closed circle. The mean gray value of each pixel in the neighborhood is determined. For each pixel in the connected region of the glass surface particle stain, the mean gray value is subtracted from the gray value of each pixel in the connected region of the glass surface particle stain to obtain the corresponding gray value difference. The negative value of each gray value is then calculated to obtain the template image sample of the glass surface particle stain.

[0023] It should be noted that after performing a negative operation on each of the grayscale differences, a non-negative compensation residual image is obtained, which is the glass surface stain template image sample. The glass surface particle stain template image sample is stored in a grayscale range of 0 to 255. Before being fed into the model for training, each grayscale value of the glass surface particle stain template image sample needs to be normalized so that the grayscale values ​​of the glass surface particle stain template image sample are normalized to between 0 and 1.

[0024] In one embodiment, multiple glass surface particle stain template images and multiple uncontaminated image samples are randomly fused to obtain a glass surface particle stain image dataset, including: Randomly overlay the template image samples of each glass surface particle stain onto any uncontaminated image to obtain a contaminated image sample with particle stains. The contaminated image sample is paired with the corresponding uncontaminated image sample to obtain an image sample; the glass surface particle stain image dataset contains multiple such image samples.

[0025] Specifically, during the camera module production stage, multiple camera modules with surface-particle contaminants on the glass can be used to photograph a background under uniform lighting conditions, or under natural outdoor light, to obtain an initial image of the contaminants. The initial image of the contaminants can be as follows: Figures 2 to 4 As shown; where, Figure 2 The image was taken under uniform lighting conditions, showing a blue background. Figure 3 The image was taken under uniform lighting on a white background. Figure 4 It was taken against an outdoor blue sky background.

[0026] Then, for each initial image of glass surface particle stains, glass surface particle stain features are extracted to obtain glass surface particle stain template image samples, as follows: First, the initial image of each glass surface particle stain is binarized to obtain a binarized image. The binarized image contains the foreground and background regions of the stain pixels. A label matrix with the same size as the binarized image can be pre-generated, initially all of which are unlabeled. The binarized image is traversed row by row using the 8-pixel adjacency rule. When an unlabeled stain foreground pixel is encountered, the pixel is used as a seed point and labeled as a connected component. All connected stain pixels are found according to the 8-pixel adjacency rule and assigned the same label. After traversing the entire image, stain pixels with the same label are grouped together to obtain the corresponding foreground connected components.

[0027] The 8-pixel adjacency rule is as follows: For any current pixel, determine the adjacent pixels in the top, bottom, left, right and four diagonal directions, a total of 8 directions; if any of the adjacent pixels in any of these 8 directions is a foreground pixel, then the current pixel and its adjacent pixels are determined to belong to the same connected component.

[0028] Since the foreground connected regions may contain not only real glass surface particle blemishes, but also pseudo connected regions caused by imaging noise and sensor noise, the number of such pseudo connected regions is extremely small and they are not real POG particle blemishes. Therefore, it is necessary to filter the foreground connected regions based on a preset filtering strategy to remove pseudo connected regions caused by noise and retain the real glass surface particle blemish connected regions.

[0029] The filtering strategy is as follows: Obtain the pixel area of ​​each foreground connected component. If the pixel area of ​​a connected component is less than a preset area threshold, it is determined to be a noise pseudo-connected component and directly discarded. Connected components with pixel areas greater than or equal to the preset area threshold are considered as glass surface particle stain connected components. The preset pixel area threshold can be set to the area corresponding to 3 to 8 pixels.

[0030] Then, determine the minimum circumcircle of the connected domain of the glass surface particle stains. The minimum circumcircle is the outer circle of the connected domain of the glass surface particle stains, and obtain the radius R of the minimum circumcircle. Extend the minimum circumcircle outward by a distance R from its boundary. Using the center of the minimum circumcircle as the origin, and the sum of the radius of the minimum circumcircle and the distance R (which equals 2R) as the radius, draw a new reference circle. The annular region between the minimum circumcircle and the reference circle is the neighborhood of the connected domain of the glass surface particle stains.

[0031] Determine the average gray value of each pixel in the neighborhood. For each pixel in the connected domain of the glass surface particle stain, subtract the gray value of each pixel in the connected domain of the glass surface particle stain from the average gray value to obtain the corresponding gray value difference. Perform a negative operation on each gray value difference to obtain the non-negative repair compensation residual of each pixel. The non-negative compensation residual of each pixel is the template image sample of the glass surface particle stain.

[0032] Each glass surface particle stain template image sample is randomly superimposed onto any uncontaminated image to simulate the imaging effect of a real camera module's glass cover plate being contaminated with particle stains, resulting in a contaminated image sample with particle stains. The number of templates, scaling ratio, superposition intensity, and superposition position of the glass surface particle stain template image samples can be randomly selected within a preset range; each glass surface particle stain template image sample is completely located within the boundary of the uncontaminated image, and different glass surface particle stain template image samples do not overlap.

[0033] The superimposed pixel values ​​are calculated according to the formula. The truncation process is performed, where, To compensate for the non-negative repair residual, L min and L max These are the lower and upper limits of the effective dynamic range of the image, respectively.

[0034] For uncontaminated image pixels, Represents non-negative compensated residuals. a For dimensionless coefficients, through The corresponding pixels in the image are dimmed by attenuation to simulate the localized brightness reduction effect caused by grainy imperfections. a This is used to control the intensity of stain appearance; the larger the value, the more pronounced the stain effect. Then, a truncation function is used to... The corresponding calculation results are constrained within the effective grayscale dynamic range of the image to prevent pixel values ​​from exceeding the limits. L min It is usually set to 0. L max The value is usually 255.

[0035] It should be noted that some POG particle stains have small local grayscale differences and may not be visually noticeable in a single static image; however, in dynamic images captured in continuous shooting or preview, as the background content changes, the relative position of the glass surface particle stains in the image remains stable, making these stains easier to observe.

[0036] An example of a contaminated image sample obtained by randomly overlaying a template image of glass surface particulate contaminants onto a clean image is shown below. Figures 5 to 20 As shown in the figure. Each contaminated image sample has a corresponding uncontaminated image sample as a real clean label image during training. Each contaminated image sample and its corresponding uncontaminated image sample constitute a glass surface particle stain image dataset; the uncontaminated image samples do not need to be shown one by one in the attached figure.

[0037] After determining the dataset of glass surface particle stain images, the dataset is divided into three independent sets: a training set, a validation set, and a test set; for example, a ratio of 7:2:1. In practical applications, the above ratio can be adjusted according to the amount of data, sample distribution, and training convergence; no restrictions are imposed here.

[0038] It should be noted that after determining the image dataset of glass surface particulate stains, when using the dataset for training, the gray values ​​of each contaminated image sample need to be normalized to between 0 and 1 before training.

[0039] S111, Construct a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term.

[0040] To improve computational efficiency, reduce the number of network parameters and inference overhead, and address the issues of complex structure, high computational cost, and limited deployment of traditional deep learning networks, this invention constructs a lightweight deep learning network. By reducing the number of network channels, employing low-resolution residual output, and multi-scale feature fusion, this lightweight deep learning network reduces the number of model parameters and inference computation while maintaining image restoration effectiveness. This improves image restoration inference efficiency and facilitates deployment on resource-constrained devices such as mobile and embedded terminals, thus balancing restoration effectiveness with practical deployment.

[0041] The lightweight deep learning network is preferably the lightweight UNet3 network. Without changing the multi-scale feature extraction, feature fusion and pixel residual map output mechanism, other convolutional neural networks with encoder-decoder structure can also be used, such as MobileNet, EfficientNet or ResNet as the encoder backbone, and the feature maps output at different downsampling scales are connected to the decoder fusion branch after channel adaptation, and the pixel residual map is output.

[0042] Taking the lightweight deep learning network UNet3 as an example, the lightweight deep learning network includes: an input layer, an encoding layer, a decoding layer, and an output layer. During training, the input layer receives image samples and transmits them to the encoding layer. The encoding layer includes multiple concatenated first convolutional modules and average pooling layers. The first convolutional module includes a first convolutional layer, a first batch of normalization layers, and a first activation layer. Each time an image sample passes through an average pooling layer, its size is reduced to half of its original size. The number of output channels of each first convolutional module increases progressively to extract multi-scale taint deep features and output multi-scale taint feature maps. The decoding layer includes three levels of decoding fusion branches. Each level of decoding fusion branch contains a second convolutional module, which includes a second convolutional layer, a second batch normalization layer, and a second activation layer. The decoding layer is used to perform progressive upsampling resolution restoration on the multi-scale taint deep feature map output by the encoding layer to obtain the taint fusion feature map. The output layer includes a third convolutional layer and a third activation layer. The output layer is used to perform channel dimension compression and feature aggregation mapping on the taint fusion feature map and output a pixel residual map. The pixel residual map is used to repair tainted image samples in the image sample.

[0043] Specifically, taking the lightweight deep learning network UNet3 as an example, the structure of the lightweight deep learning network is shown in Table 1: Table 1

[0044] In lightweight deep learning networks, network tensors are used to store images and feature information. The network tensor in this invention has the dimension BCHW, where dimension B represents the batch size. For example, if a batch of inputs consists of 8 images, then B=8; dimension C represents the number of channels; dimension H represents the image height; and dimension W represents the image width.

[0045] The encoding layer consists of four concatenated first convolutional modules and average pooling layers. The first convolutional module includes a first convolutional layer Conv2d1, a first batch normalization layer BatchNorm2d1, and a first activation layer ReLU1. Each first convolutional module is followed by an average pooling layer. After each average pooling, the height and width of the image are halved. The number of channels in the four concatenated first convolutional modules are C1, C2, C3, and C4, respectively. The number of encoded output channels satisfies C1 < C2 < C3 < C4.

[0046] In a preferred embodiment, C1, C2, C3 and C4 correspond to 16, 32, 64 and 128 respectively; each level of the first convolutional module includes a convolutional layer with a kernel of 3×3, a stride of 1 and padding of 1, a batch normalization layer and a ReLU activation layer, and the first three levels of the first convolutional modules are respectively connected to mean pooling layers with a kernel of 2×2 and a stride of 2.

[0047] K2, K4, and K8 represent the number of input channels after spatial alignment and concat stitching at the H / 2, H / 4, and H / 8 scales, respectively; in this embodiment, K2=K4=K8=C1+C2+C3+C4=240.

[0048] D represents the number of output channels in each decoding fusion branch; in this embodiment, D=32. Each second convolutional module includes a convolutional layer with a 3×3 kernel, a stride of 1, and padding of 1, a batch normalization layer, and a ReLU activation layer. The output layer uses a convolutional layer with a 3×3 kernel, a stride of 1, and padding of 1 to map the 32-channel features into a single-channel residual map, and employs the Sigmoid activation function.

[0049] After being processed sequentially by the first convolutional module and the mean pooling layer, the encoding layer outputs four sets of multi-scale taint feature maps with different spatial resolutions.

[0050] In this system, the first convolutional module of the first-level encoding outputs C1 channels, corresponding to the feature map at the original image resolution; the first convolutional module of the second-level encoding outputs C2 channels, corresponding to feature maps at resolutions of H / 2 and W / 2; the first convolutional module of the third-level encoding outputs C3 channels, corresponding to feature maps at resolutions of H / 4 and W / 4; and the first convolutional module of the fourth-level encoding outputs C4 channels, corresponding to feature maps at resolutions of H / 8 and W / 8. These four sets of multi-scale feature maps obtained from the encoding layer are then input into the decoding branches of the decoding layer, allowing the decoding layer to perform spatial alignment and channel concatenation operations on each feature map.

[0051] The decoding layer includes three levels of decoding fusion branches. Each level of decoding fusion branch contains a second convolutional module. The second convolutional module includes a second convolutional layer Conv2d2, a second batch normalization layer BatchNorm2d2, and a second activation layer ReLU2. The second convolutional module of the decoding layer has the same structure as the first convolutional module of the encoding layer.

[0052] The decoding layer receives four sets of multi-scale taint feature maps output by the encoding layer. It upsamples the taint feature maps of different scales to complete spatial alignment. The spatially aligned multi-path features are then concatenated in the channel dimension and fused through the corresponding second convolution module.

[0053] The first-level decoding fusion branch outputs taint fusion feature maps with spatial resolutions of H / 8 and W / 8, the second-level decoding fusion branch outputs taint fusion feature maps with spatial resolutions of H / 4 and W / 4, and the third-level decoding fusion branch outputs taint fusion feature maps with spatial resolutions of H / 2 and W / 2.

[0054] The output layer includes a third convolutional layer (Conv2d3) and a third activation layer. The third convolutional layer is used to aggregate and map the multi-channel features of the taint fusion feature map into a single-channel feature output. The third activation layer uses the Sigmoid activation function to constrain the output pixel values ​​to the range of 0 to 1, thereby obtaining the pixel residual map.

[0055] The pixel residual map has a spatial resolution of H / 2 and W / 2. Since the pixel residual map is inconsistent with the original resolution (H, W) of the contaminated image sample, bilinear interpolation is required to align the pixel residual map with the size of the contaminated image sample. The pixel residual map after size alignment is then added to the contaminated image sample pixel by pixel to remove dirt from the contaminated image sample and obtain the repaired image.

[0056] In one implementation, aligning the pixel residual map with the size of the contaminated image sample using bilinear interpolation includes: Obtain the original resolutions H and W of the contaminated image samples; using the original resolutions H and W as target sizes, perform upsampling processing on the pixel residual maps with resolutions of H / 2 and W / 2; A bilinear interpolation algorithm is used to calculate the gray level of the newly added pixel position under the target size based on the gray level values ​​of adjacent pixels in the residual map. This results in an aligned residual map with resolutions H and W after scaling, ensuring that the aligned residual map corresponds one-to-one with the pixel grid of the contaminated image sample.

[0057] Specifically, since several new pixels will be generated after the pixel residual map with resolution of H / 2 and W / 2 is magnified by one time, for any new pixel, the average gray value of the four original pixels surrounding the new pixel is selected as the gray value of the new pixel position.

[0058] Normalize the compensation residual at the location of the newly added pixel. Mapped to the actual grayscale compensation amount in the range of 0 to 255 G The mapping relationship is ),in, These are the grayscale mapping coefficients. This refers to the normalized compensation residual obtained by normalizing the aforementioned non-negative compensation residual. In a preferred embodiment... =255; The value range is 0 to 1, therefore G The value range is 0 to 255.

[0059] Based on the actual grayscale compensation amount, bilinear interpolation is performed on the grayscale values ​​of adjacent pixels in the residual map to obtain a pixel residual map that is aligned with the size of the original image.

[0060] Based on the above analysis, the aligned pixel residual map can be obtained, and the gray value of each pixel in the aligned pixel residual map is between 0 and 255.

[0061] Then, the pixel grayscale values ​​of the contaminated image sample are added one by one to the pixel grayscale values ​​at the corresponding positions in the pixel residual map to complete the repair of the contaminated image sample and obtain the repaired image sample corresponding to the contaminated image sample.

[0062] To balance pixel brightness restoration, edge transition continuity, and image texture fidelity, a combined loss function is constructed. This function jointly constrains the network output from multiple dimensions, including pixels, gradients, and structure, avoiding the imperfections caused by using only a single loss function. This improves the image restoration accuracy of POG blemish areas, resulting in images with reasonable brightness, smooth edge transitions, and preservation of original image texture details. In this embodiment, the loss function is a combined loss function, determined as follows: According to the formula Determine the loss function L ;in, For robust pixel loss terms, These are the weighting coefficients corresponding to the robust pixel loss term. For gradient consistency loss term, These are the weight coefficients corresponding to the gradient consistency loss term. For structural similarity loss terms, These are the weighting coefficients corresponding to the structural similarity loss term.

[0063] The robust pixel loss term is used to constrain the pixel differences between the repaired image sample and the uncontaminated image sample, the gradient consistency loss term is used to constrain edge and texture details, and the structural similarity loss term is used to constrain local structural similarity.

[0064] The robust pixel loss term is determined using a Smooth L1 type piecewise function. Let... To repair the image sample and compare it with the uncontaminated image sample at the 1st i Error at each pixel ,in To preset a positive scaling factor, For the scaled repaired image sample and the uncontaminated image sample, at the 1st... i Error at 1 pixel.

[0065] when hour, ,when hour, ,in, To preset a positive threshold, For the first i The value of the robust pixel loss term for each pixel.

[0066] The robust pixel loss term is: ,in For optional blemish area masking, The combined loss function is: ,in , and To preset positive weights, specifically, These are the weighting coefficients corresponding to the robust pixel loss term. These are the weight coefficients corresponding to the gradient consistency loss term. These are the weighting coefficients corresponding to the structural similarity loss term.

[0067] Specifically, it refers to the gradient difference loss in the horizontal and vertical directions. The loss is based on structural similarity.

[0068] The gradient consistency loss term is based on the formula Sure, i To repair the first in the image sample i 1 pixel, N To restore the total number of pixels in the image sample, To repair the first image sample i grayscale value of each pixel. The first in the uncontaminated image sample i grayscale value of each pixel. To repair image samples in the first i The gradient at each pixel position. For contaminated image samples in the first i The gradient at each pixel position. This is the gradient operator.

[0069] Methods for determining the structural similarity loss term include: When calculating the similarity between repaired image samples normalized to 0–1 and uncontaminated image samples, an 11×11 local Gaussian weighted window is preferred, with a Gaussian kernel (Gaussian standard deviation) of 1.5; image dynamic range L=1, and empirical constants. empirical constants ,in, K 1 = 0.01 K 2 = 0.03, that is C 1 = 0.0001 C 2 = 0.0009.

[0070] Within a weighted window, the local mean, variance, and covariance are calculated based on Gaussian weights to obtain the single-window similarity. Then according to the formula Calculate structural similarity loss , This represents the weighted number of windows.

[0071] S112, the lightweight deep learning network is iteratively trained using the glass surface particle stain image dataset and the combined loss function until the iteration converges, and the target image stain repair model is output.

[0072] After constructing the glass surface particle stain image dataset and the combined loss function, the lightweight deep learning network can be iteratively trained using the glass surface particle stain image dataset and the combined loss function until the iteration converges, outputting the target image stain repair model.

[0073] In one implementation, a lightweight deep learning network is iteratively trained using a training set and a combined loss function until iterative convergence, outputting a target image dirt restoration model, including: Data augmentation is performed on contaminated and uncontaminated image samples in the training set to align the pixels in the augmented contaminated and uncontaminated image samples. The enhanced contaminated image samples are input into a lightweight deep learning network to obtain residual feature maps. The residual feature maps and contaminated image samples are then fused to obtain the reconstructed and repaired images. The loss value between the reconstructed and restored image and the uncontaminated image sample is calculated using the combined loss function; The network weights are optimized by backpropagation based on the loss value until the iteration converges and training stops, outputting the initial image dirt restoration model; The generalization performance of the image dirt restoration model is tested using a test set. If the performance test results meet the requirements, the initial image dirt restoration model is determined as the target image dirt restoration model.

[0074] Specifically, the same random data augmentation operation is performed on the paired contaminated and uncontaminated image samples in the training set to ensure that the pixel positions of the augmented contaminated and uncontaminated image samples are strictly aligned. The data augmentation includes random horizontal flipping, random vertical flipping, and random angle rotation. The same transformation parameters are used on the paired samples to ensure that the samples correspond to pixels one by one.

[0075] The data-augmented polluted image sample is input into a lightweight deep learning network, which outputs a pixel residual map. After performing bilinear interpolation scaling and alignment and grayscale mapping on the pixel residual map, it is added to and fused pixel by pixel with the augmented polluted image sample to obtain the reconstructed and repaired image.

[0076] The loss value between the reconstructed and repaired image and the corresponding uncontaminated image sample is calculated using the combined loss function; backpropagation is performed based on the loss value, and the network weights are updated through the Adam optimizer; the network weights include the weight parameters of each convolutional layer in the encoding layer, the weight parameters of each convolutional layer in the decoding fusion branch, the weight parameters of the third convolutional layer in the output layer, and the scaling and offset parameters of each batch normalization layer.

[0077] During training, the learning rate is adaptively decayed based on the validation set loss; training continues iteratively, and when the validation set loss no longer decreases for several consecutive rounds, the iteration is considered to have converged, training is stopped, and the initial image dirt repair model is output.

[0078] The generalization performance of the initial image contamination restoration model was tested using a test set. No random data augmentation was performed during the test. The contaminated image samples in the test set were used to obtain the corresponding restored images. The restoration performance of the model was comprehensively evaluated using image pixel error, gradient error, and structural similarity index. If the performance test results met the preset index requirements, the initial image contamination restoration model was determined as the target image contamination restoration model.

[0079] S113, the target image dirt repair model is used to repair the current image to be repaired of the camera module to obtain a repaired image with dirt removed.

[0080] After the target image dirt restoration model is trained, in practical applications, the current image to be restored can be input into the target image dirt restoration model to obtain a restored image with dirt removed. Specifically, the current grayscale image to be repaired, captured by the camera module, is obtained. This image contains POG particle stains on the glass surface of the lens module. The current image to be repaired is then input into the input layer of the trained target image stain repair model. Multi-scale stain feature maps are extracted through the encoding layer of the lightweight UNet3 network, and then multi-scale feature alignment and fusion are completed through the decoding fusion branches at each level of the decoding layer. The output is a pixel residual map with a resolution of H / 2, W / 2, and a value range of 0 to 1.

[0081] Perform bilinear interpolation upsampling on the pixel residual image, scale and align it to the original resolution H and W of the current image to be repaired, and obtain the aligned pixel residual image; perform grayscale mapping on the aligned pixel residual image to obtain the grayscale compensation amount; add the grayscale compensation amount to the current image to be repaired pixel by pixel to obtain the repaired image with POG particle stains removed, and then output the repaired image with the dirt removed.

[0082] This invention eliminates the need for manual marking and repair of image blemishes, avoiding errors caused by manual operation and improving the repair efficiency of particle blemishes on glass surfaces. The lightweight modified network reduces computational overhead, adapting to the real-time processing needs of mobile camera devices. At the same time, the combined loss function constrains the repair effect from the pixel, gradient, and multi-dimensional structural levels, preserving the original texture details of the image while removing POG particle blemishes, resulting in a repaired image with better imaging quality.

[0083] Based on the same inventive concept as in the foregoing embodiments, this embodiment also provides an image dirt repair device based on deep learning, such as... Figure 21 As shown, the device includes: The first construction unit 211 is used to construct a glass surface particle stain image dataset; the glass surface particle stain image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; The second building unit 212 is used to build a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term. Training unit 213 is used to iteratively train the lightweight deep learning network using the glass surface particle stain image dataset and the combined loss function until the iteration converges, and output the target image stain repair model. Repair unit 214 is used to repair the current image to be repaired of the camera module using the target image dirt repair model to obtain a repaired image with dirt removed.

[0084] Since the apparatus described in the embodiments of this invention is used to implement the image contamination repair method based on deep learning, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in the embodiments of this invention, and therefore will not be repeated here. All apparatuses used in the methods of the embodiments of this invention fall within the scope of protection of this invention.

[0085] Based on the same inventive concept, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any step of the method described above.

[0086] Based on the same inventive concept, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0087] Through one or more embodiments of the present invention, the present invention has the following beneficial effects or advantages: This invention provides a method, apparatus, medium, and device for image contamination repair based on deep learning. The method includes: constructing a glass surface particle contamination image dataset; the glass surface particle contamination image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; constructing a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term; iteratively training the lightweight deep learning network using the glass surface particle contamination image dataset and the combined loss function until iterative convergence, outputting a target image contamination repair model; using the target image contamination repair model to repair the current image to be repaired by a camera module, obtaining a repaired image with the contamination removed; thus, manual marking and repair of contamination areas in the image is unnecessary, avoiding errors caused by manual operation and improving the repair efficiency of glass surface particle contamination; the lightweight modified network can reduce computational overhead, adapting to the real-time processing requirements of mobile camera devices, while the combined loss function constrains the repair effect from the pixel, gradient, and multi-dimensional structural levels, preserving the original texture details of the image while removing POG particle contamination, resulting in a repaired image with better imaging quality.

[0088] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0089] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0090] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0091] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0092] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0093] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components of the gateway, proxy server, or system according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0094] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0095] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image dirt restoration based on deep learning, characterized in that, The method includes: A glass surface particle stain image dataset is constructed; the glass surface particle stain image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; A lightweight deep learning network and a combined loss function are constructed; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term. The lightweight deep learning network is iteratively trained using the glass surface particle stain image dataset and the combined loss function until iterative convergence, and the target image stain repair model is output. The target image dirt repair model is used to repair the current image to be repaired from the camera module, resulting in a repaired image with dirt removed.

2. The method as described in claim 1, characterized in that, The constructed glass surface particle stain image dataset includes: Multiple camera modules with glass surface particle stains were used to capture images of the background panel, resulting in multiple initial images of the glass surface particle stains. For each initial image of a glass surface particle stain, glass surface particle stain features are extracted to obtain a glass surface particle stain template image sample. Multiple uncontaminated image samples are obtained, and multiple template image samples of glass surface particle stains and multiple uncontaminated image samples are randomly fused to obtain the glass surface particle stain image dataset.

3. The method as described in claim 2, characterized in that, The step of extracting glass surface particle stain features from the initial image of each of the glass surface particle stains to obtain a glass surface particle stain template image sample includes: For each initial image of the glass surface particle stain, the initial image of the glass surface particle stain is binarized to obtain the corresponding binarized image; The binarized image is traversed using pixel adjacency rules to obtain multiple foreground connected components; By filtering through the multiple foreground connected regions, the glass surface particle stain connected regions are obtained; Determine the smallest closed circle of the connected domain of the particulate contaminants on the glass surface; The neighborhood of the glass surface particle stain connected region is determined based on the smallest closed circle of the glass surface particle stain connected region; the neighborhood of the glass surface particle stain connected region is an annular area obtained by extending outward from the smallest closed circle by a preset distance, the preset distance being the radius of the smallest closed circle; The mean gray value of each pixel in the neighborhood is determined. For each pixel in the connected region of the glass surface particle stain, the mean gray value is subtracted from the gray value of each pixel in the connected region of the glass surface particle stain to obtain the corresponding gray value difference. Each gray value difference is negatively calculated to obtain the template image sample of the glass surface particle stain.

4. The method as described in claim 2, characterized in that, Multiple template image samples of glass surface particle stains and multiple uncontaminated image samples are randomly fused to obtain the glass surface particle stain image dataset, including: Each of the glass surface particle stain template image samples is randomly superimposed onto any uncontaminated image to obtain a contaminated image sample with particle stains. The contaminated image sample is paired with the corresponding uncontaminated image sample to obtain the image sample; multiple image samples are determined as the glass surface particle stain image dataset.

5. The method as described in claim 1, characterized in that, The lightweight deep learning network includes: an input layer, an encoding layer, a decoding layer, and an output layer; During training, the input layer is used to receive the image samples and transmit the image samples to the encoding layer; The encoding layer includes multiple sequentially connected first convolutional modules and mean pooling layers. The first convolutional module includes a first convolutional layer, a first batch of normalization layers, and a first activation layer. Each time an image sample passes through a mean pooling layer, its size is reduced to half of its original size. The number of output channels of each level of the first convolutional module increases progressively to extract multi-scale deep features of blemishes and output multi-scale blemish feature maps. The decoding layer includes a three-level decoding fusion branch, and each level of the decoding fusion branch includes a second convolutional module. The second convolutional module includes a second convolutional layer, a second batch normalization layer, and a second activation layer. The decoding layer is used to perform progressive upsampling resolution restoration on the multi-scale taint deep feature map output by the encoding layer to obtain a taint fusion feature map. The output layer includes a third convolutional layer and a third activation layer. The output layer is used to perform feature aggregation mapping on the taint fusion feature map and output a pixel residual map. The pixel residual map is used to repair the tainted image samples in the image samples to obtain the repaired image samples corresponding to the tainted image samples.

6. The method as described in claim 1, characterized in that, The combined loss function is determined in the following way: According to the formula Determine the loss function L ;in, The For robust pixel loss terms, The weight coefficients corresponding to the robust pixel loss term, the For the gradient consistency loss term, The weight coefficients corresponding to the gradient consistency loss term, the For the structural similarity loss term, These are the weighting coefficients corresponding to the structural similarity loss term.

7. The method as described in claim 1, characterized in that, The training set also includes a test set; the iterative training of the lightweight deep learning network using the training set and the combined loss function until convergence, outputting a target image dirt restoration model, includes: Data augmentation is performed on contaminated and uncontaminated image samples in the training set to align the pixels in the augmented contaminated and uncontaminated image samples. The enhanced contaminated image sample is input into the lightweight deep learning network to obtain a residual feature map. The residual feature map and the contaminated image sample are then fused to obtain a reconstructed and repaired image. The loss value between the reconstructed and repaired image and the uncontaminated image sample is calculated using the combined loss function; The network weights are optimized by backpropagation based on the loss value until the iteration converges and training stops, outputting the initial image dirt restoration model. The generalization performance of the image dirt restoration model is tested using the test set. If the performance test results meet the requirements, the initial image dirt restoration model is determined as the target image dirt restoration model.

8. A deep learning-based image dirt repair device, characterized in that, The device includes: The first construction unit is used to construct a glass surface particle stain image dataset; the glass surface particle stain image dataset contains multiple image samples, each image sample including a contaminated image sample and a corresponding uncontaminated image sample; The second building unit is used to build a lightweight deep learning network and a combined loss function; the combined loss function includes a robust pixel loss term, a gradient consistency loss term, and a structural similarity loss term. The training unit is used to iteratively train the lightweight deep learning network using the glass surface particle stain image dataset and the combined loss function until the iteration converges, and outputs the target image stain repair model. The repair unit is used to repair the current image to be repaired of the camera module using the target image dirt repair model, so as to obtain a repaired image with dirt removed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-7.

Images