License plate image restoration method mixed with multi-scale receptive field and attention mechanism

Abstract

The invention discloses a license plate image restoration method mixing a multi-scale receptive field and an attention mechanism, and belongs to the technical field of digital image restoration. An MHDR module is designed, feature information of different scales is extracted through three receptive fields of different scales and feature fusion is performed, so that the model can extract effective feature information from a distance while paying attention to feature information near a damaged area, and consistency of an image structure and texture is ensured; introducing a hybrid constraint multi-scale loss function, and generating a repaired image with reasonable semantics, consistent style, high pixel reconstruction precision and high visual fidelity through constraint of consistency of low-level details and high-level semantics of the generated license plate image and the stained license plate image; a middle layer of the network adopts two hole convolutions and a depth separable convolution to form a three-layer convolution cascade structure, and a dependency relationship between each pixel point on a feature map and a neighborhood after image coding is constructed through a large receptive field.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image restoration, and in particular to a license plate image restoration method that mixes multi-scale receptive field and attention mechanism. Background Art

[0002] With the rapid development of deep learning technology and the gradual intelligentization of urban traffic management, license plate recognition, as one of the key technologies of intelligent traffic management, is widely used in traffic violation management, toll booths, parking lots, and vehicle management in residential areas. However, when cars are exposed to the outdoor environment for a long time, license plates are often stained and obscured, such as dirt, leaves, or human damage. These factors directly affect the efficiency and accuracy of license plate recognition.

[0003] Image restoration refers to predicting and repairing the missing parts of an image to generate a replacement image with complete structure, reasonable semantics, and in line with human perception. In recent years, the rapid development of deep learning technology has promoted the widespread application of generative adversarial networks and convolutional neural networks, and introduced many advanced methods in the field of image restoration.

[0004] Although existing deep learning restoration methods can extract image context semantic information and achieve good restoration effects, they are mostly used in fields such as faces, street scenes, and art. Due to the simple color and strong structure of license plate images, they fail to fully integrate the semantic information and texture information of the image. Therefore, the restored images still have problems such as distortion, blurred details, and incomplete structures. To this end, a license plate image restoration method that mixes multi-scale receptive fields and attention mechanisms is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to solve the shortcomings of existing image restoration technology and the problem of low recognition accuracy of license plate recognition systems for damaged and obscured license plates. A license plate image restoration method that mixes multi-scale receptive field and attention mechanism is provided, which realizes image restoration technology tailored to the characteristics of license plates and improves the recognition accuracy of license plate recognition systems.

[0006] like Figure 4 As shown, the present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0007] S1: Network Construction

[0008] Design the MHDR module and introduce the ECA module, optimize the middle layer of the encoder and decoder, and then build the license plate image restoration network;

[0009] S2: Model training

[0010] Construct a hybrid constrained multi-scale loss function, use the hybrid constrained multi-scale loss function to train the license plate image restoration network, and obtain the license plate image restoration model;

[0011] S3: License Plate Image Restoration

[0012] The license plate image to be repaired is input into the license plate image repair model to obtain the license plate repair image.

[0013] Furthermore, in step S1, the license plate image restoration network includes an encoder and a decoder, each of the encoder and the decoder includes three MHDR modules of the same structure, and the encoder and the decoder are connected via an intermediate layer;

[0014] In the encoder, the input image is processed by the first MHDR module, then processed by the ECA module and downsampled using a max pooling layer with a stride of 2, then processed by the second MHDR module, then downsampled using a max pooling layer with a stride of 2, then processed by the third MHDR module, and then downsampled using a max pooling layer with a stride of 2;

[0015] In the decoder, upsampling is first performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the first MHDR module, then upsampling is performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the second MHDR module, then upsampling is performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the third MHDR module;

[0016] In order to realize the information interaction between semantic channels and texture channels and prevent the loss of hierarchical information after channel weighting, the feature information extracted by the encoder is fused with the information obtained by upsampling through skip connections.

[0017] Furthermore, in step S1, the intermediate layers of the encoder and the decoder include a dilated convolution layer with a dilation rate of 1, a DW convolution layer, and a dilated convolution layer with a dilation rate of 5, which are connected in sequence and have convolution kernel sizes of 3*3.

[0018] Furthermore, in step S2, the processing process of the MHDR module is as follows: first, the feature map input is input into a dilated convolution layer with a dilated rate of 2, a step size of 1, and a convolution kernel size of 3*3 to extract feature information of the image;

[0019] Then the feature information is input into three branches respectively: the first branch uses a DW convolution layer with a step size of 1 and a convolution kernel size of 3*3 to increase the original number of channels to 3 times, and the input receptive field is 7 relative to the feature map; the second branch sequentially uses a cascade of a normal convolution layer with a convolution kernel size of 1*1 and a dilated convolution layer with a convolution kernel size of 3*3, and the input receptive field is 9; the third branch sequentially uses a dilated convolution layer with a dilation rate of 5, a step size of 1, and a convolution kernel size of 5*5, a normal convolution layer with a convolution kernel size of 1*1, and a DW convolution layer with a convolution kernel size of 3*3 to increase the receptive field again, and the input receptive field is 17 relative to the feature map;

[0020] Finally, the Concat operation is used to fuse multi-scale features, and then a common convolution layer with a convolution kernel size of 1*1 is used to compress and aggregate the channel feature information after multi-scale feature fusion to output the feature map output.

[0021] Furthermore, the calculation formula of the feature map output is as follows:

[0022] output = Conv1(concat(G i (Conv3(input))+input))

[0023] Among them, Conv3(input) is a 3*3 convolution operation on the feature map input, concat is a multi-scale splicing operation, Conv1 is a 1*1 convolution operation on the feature map output after the concat splicing operation, G i Calculate the receptive field scale of the i-th branch, i = 1, 2, 3.

[0024] Furthermore, in step S1, the ECA module adaptively adjusts the feature weights of each channel in the image by introducing a learnable one-dimensional convolution, so that the model can identify the importance of different channels to the restoration task, guide the information flow between channels to the texture channel and the semantic channel, improve the feature expression capability, and filter out irrelevant information and noise.

[0025] Furthermore, in step S2, the hybrid constrained multi-scale loss function is as follows:

[0026] L=λ hole L hole +λ valid L valid +λ per L per +λ sty L sty +λ tv L tv +λ adv L adv

[0027] Among them, L hole Reconstruction loss for damaged area, L valid Reconstruct the loss for the non-damaged area, L per is the perceptual loss, L sty is the style loss, L tv is the total variational loss, L adv To counter the loss, λ hole , valid , per , sty , tv With λ adv are the weight parameters corresponding to the loss respectively.

[0028] Furthermore, in the hybrid constrained multi-scale loss function:

[0029] Reconstruct the loss L by the damaged area hole and the reconstruction loss L in the non-damaged area valid Calculate the pixel loss L pix :

[0030] L pix =λ hole L hole +λ valid L valid

[0031] L hole =||(I out -I gt )⊙I mask ||1

[0032] L valid =||(1-I mask )⊙(I out -I gt )||1

[0033] Among them, I out To repair the image, I gt is the real image, I mask is the damaged image obtained after the real image is damaged, L hole Reconstruction loss for damaged area, L valid is the reconstruction loss of the non-damaged area, λ hole , valid is the weight parameter of the reconstruction loss of the damaged area and the reconstruction loss of the non-damaged area;

[0034] Perceptual loss L per The calculation formula is:

[0035]

[0036] Among them, φ i (I) is the i-th layer feature map extracted by the VGG-19 pre-trained network, C i , H i , W i is the number of channels, height, and width of the feature map of the i-th layer;

[0037] Style loss L sty The calculation formula is:

[0038]

[0039] Among them, ψ i is the Gram matrix, which is used to quantify the style information in the image features;

[0040] The total variational loss L tv The calculation formula is:

[0041]

[0042] Among them, j, l, k are I out The spatial dimension and channel dimension index of , C, H, W are the number of channels, height and width of the feature map;

[0043] Adversarial loss L adv The formula is:

[0044]

[0045] Among them, I gt is the data distribution of real images, I out To repair the data distribution of the image, E is the mathematical expectation, G is the generator, and D is the discriminator.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. A MHDR multi-scale hybrid hole residual module is designed. It extracts feature information of different scales through three different scales of receptive fields and performs feature fusion. This enables the model to extract effective feature information from a distance while focusing on the feature information near the damaged area, thus ensuring the consistency of image structure and texture.

[0048] 2. A hybrid constrained multi-scale loss function is introduced to generate a repaired image with reasonable semantics, consistent style, high pixel reconstruction accuracy and visual fidelity by constraining the low-level details and high-level semantic consistency of the generated license plate image and the defaced license plate image.

[0049] 3. By introducing the ECA attention module, the model's ability to extract channel feature information and spatial position feature information is enhanced; the middle layer of the network adopts two hollow convolutions and one depth-separable convolution to form a three-layer convolution cascade structure, and constructs the dependency relationship between each pixel and the neighborhood on the feature map after image encoding through a large receptive field.

[0050] 4. The license plate image restored by the present invention is used in the YOLO v8 license plate recognition system experiment, which greatly improves the efficiency and accuracy of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of the structure of the MHDR module in an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of the structure of a license plate image restoration network in an embodiment of the present invention;

[0053] FIG3(a) is an example of a defaced or obscured license plate image in an embodiment of the present invention, wherein (1) is example 1, (2) is example 2, (3) is example 3, and (4) is example 4;

[0054] FIG3(b) is a restored image of the damaged or obscured license plate image in FIG3(a), wherein (1) is the restored image of Example 1, (2) is the restored image of Example 2, (3) is the restored image of Example 3, and (4) is the restored image of Example 4;

[0055] Figure 4 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0056] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0057] This embodiment provides a technical solution: a license plate image restoration method that mixes multi-scale receptive field and attention mechanism, including the following steps:

[0058] Step 1: First, you need to deploy the corresponding Pytorch environment, version number: 1.12.1, CUDA version is 11.6, and perform model training and testing to facilitate subsequent model establishment, training and testing; using commonly used license plate data sets, such as the CBLPRD-330k license plate data set, which contains 342,110 low-resolution license plate images of size 148×48, the present invention randomly selects 10,000 blue license plate images as the training set, and randomly selects the remaining blue license plate images in the data set and 300 blue license plates taken in real scenes as the test set.

[0059] Step 2: Deploy the corresponding hardware environment. Taking this embodiment as an example, the hardware environment configuration of the computing platform is: CPU-Intel-Core-i5-12490F, GPU-RTX3080-12G. The software environment is: Python 3.9. For the training of the generator and discriminator, the image size of all data sets is adjusted to 256×256, and the learning rate is 2e-4. The generator is trained using the Adam optimizer, β1=0.5 and β2=0.999, batch-size and epoch are set to 4 and 100,

[0060] Step 3: Build a license plate image restoration network with hybrid multi-scale receptive field and attention mechanism for model training;

[0061] Step 3-1:

[0062] like Figure 1 As shown in the figure, an MHDR multi-scale hybrid atrous residual module (MHDR module) is constructed. First, the feature map input is input into the atrous convolution layer with a atrous rate of 2, a stride of 1, and a convolution kernel size of 3*3 to extract the feature information of the image and preliminarily improve the receptive field; then the feature information is input into three branches respectively: the first branch uses a depthwise separable (DW) convolution layer with a stride of 1 and a convolution kernel size of 3*3 to increase the dimension of the original number of channels to 3 times, which is 7 relative to the input receptive field. The second branch uses a 1*1 ordinary convolution layer and a 3*3 atrous convolution layer in cascade to further improve the receptive field to enhance the network's ability to extract high-level semantic information, which is 9 relative to the input receptive field. The third branch uses a hole convolution layer with a hole rate of 5, a step size of 1, and a convolution kernel size of 5*5, a 1*1 ordinary convolution layer, and a 3*3 DW convolution layer in sequence to increase the receptive field again. Relative to the original input feature map, the input receptive field is 17, among which the 3*3 DW convolution layer is used to reduce the number of parameters and compensate for the local information loss caused by the excessive hole rate. Since the receptive field size of the convolution layer is crucial to the integrity of the feature information extracted by the network layer. Therefore, the present invention uses three receptive fields from small to large to extract feature information of three scales, and uses the Concat splicing operation to perform multi-scale feature fusion to enhance the semantic coherence and texture correlation between the damaged area of ​​the image and the neighborhood. In order to extract the diversified feature information obtained at three scales, the present invention compresses the aggregated feature fusion channel feature information through a 1*1 ordinary convolution layer, and outputs the feature map output. The formula is as follows:

[0063] output = Conv1(concat(G i (Conv3(input))+input))

[0064] Among them, Conv3(input) is a 3*3 convolution operation on the feature map input, concat is a multi-scale splicing operation, Conv1 is a 1*1 convolution operation on the feature map output after concat splicing, G i Calculate the receptive field scale of the i-th branch, i = 1, 2, 3.

[0065] Step 3-2:

[0066] like Figure 2 As shown, the encoder-decoder structure of the license plate image restoration network of the present invention includes 6 MHDR modules. After the first MHDR module, the ECA module (ECA attention module) is introduced. By introducing a learnable one-dimensional convolution, the feature weights of each channel in the image are adaptively adjusted, and the feature information extracted by the filter channel and the color channel are integrated to improve the network's perception of the feature information of the semantic and texture channels. After that, the maximum pooling (step length is 2) operation is used for downsampling to reduce the feature dimension (except that the other two MHDR modules in the encoder do not use the ECA module, the operations are as described above); then the middle layer of the encoder and the decoder uses three 3*3 convolution kernels (respectively, a hole convolution layer with a hole rate of 1, a DW convolution layer, and a hole convolution layer with a hole rate of 5) convolution layers to increase the receptive field and reduce the number of parameters; before each MHDR module in the decoder, a deconvolution layer with a step length of 2 and a convolution kernel of 3*3 is used to perform upsampling operations to enlarge the image. Based on the idea of ​​residual connection, in order to realize the information interaction between semantic channels and texture channels and prevent the loss of hierarchical information after channel weighting, the feature information extracted by the encoder is fused with the information obtained by upsampling through skip connections.

[0067] In this embodiment, the ECA module introduces a learnable one-dimensional convolution to adaptively adjust the feature weights of each channel in the image, so that the model can more accurately identify the importance of different channels to the restoration task, and guide the information flow between channels to key channels such as texture channels and semantic channels, effectively improving the network feature expression capability while filtering out irrelevant information and noise.

[0068] Step 4: Construct a hybrid constrained multi-scale loss function;

[0069] In order to reduce the pixel difference between the inpainted image (model-generated image) and the real image, the pixel loss is calculated using the L1 norm to evaluate the inpainted image I out and the real image I gt The difference between the two is the pixel loss L pix The calculation formula is:

[0070] L pix =λhole L hole +λ valid L valid

[0071] L hole =||(I out -I gt )⊙I mask ||1

[0072] L valid =||(1-I mask )⊙(I out -I gt )||1

[0073] Among them, I out To repair the image, I gt is the real image, I mask is the damaged image obtained after the real image is damaged, L hole Reconstruction loss for damaged area, L valid is the reconstruction loss of the non-damaged area, λ hole , valid is the weight parameter of the reconstruction loss of the damaged area and the reconstruction loss of the non-damaged area;

[0074] Perceptual loss L per Calculate the real image I through the VGG-19 pre-trained network gt and the repaired image I out The similarity between them improves the network's ability to extract high-dimensional texture features in feature images, while compensating for high-level semantic features that are difficult to obtain due to reconstruction loss, thereby further improving the structural consistency between the model-generated images and the real images, and the perceptual loss L per The calculation formula is:

[0075]

[0076] Among them, φ i (I) is the i-th layer feature map extracted by the VGG-19 pre-trained network, C i , H i , W i is the number of channels, height, and width of the feature map of the i-th layer;

[0077] Style loss L sty Constrain the noisy image to extract high-level semantic information of the image, make the damaged area semantically coherent with the neighborhood, and then perform texture synthesis to generate a consistent, beautiful and natural repaired image. The style loss L sty The calculation formula is:

[0078]

[0079] Among them, ψ i is the Gram matrix, which is used to quantify the style information in the image features. The style similarity between the generated image and the real image is measured by comparing the differences between the Gram matrices.

[0080] The total variational loss L tv Reduce the total variation to limit image noise and reduce the difference in pixel values ​​between adjacent pixels to maintain the smoothness of the generated image detail boundaries. The total variation loss L tv The calculation formula is:

[0081]

[0082] Among them, j, l, k are I out The spatial dimension and channel dimension index of , C, H, W are the number of channels, height and width of the feature map;

[0083] Adversarial loss L adv Supervise the training process of the generative adversarial network to ensure that the generator G can generate repair results that deceive the discriminator D, and the adversarial loss L adv The formula is:

[0084]

[0085] Among them, I gt is the data distribution of real images, I out is the data distribution of the repaired image, E is the mathematical expectation, and the generator G (i.e., the license plate image repair network) continuously adjusts I gt and I out The result of this formula is minimized, and the discriminator D maximizes the result of this formula, thereby prompting the repair model to be continuously optimized in the confrontation;

[0086] In summary, the hybrid constrained multi-scale loss function of the model of the present invention is:

[0087] L=λ hole L hole +λ valid L valid +λ per L per +λ sty L sty +λ tv L tv +λ adv L adv

[0088] Among them, λ hole , valid , per , sty ,tv With λ adv are the weight parameters corresponding to the loss respectively.

[0089] In this embodiment, the VGG-19 pre-trained network is used to calculate the style loss and the perceptual loss, and the weight parameters of each loss function are set as: hole =6,λ per =0.01,λ adv =0.009,λ sty =250,λ tv =0.001,λ valid =1.

[0090] Step 5: As shown in Figure 3(a), the model is tested using the simulated stained or blocked license plate image (damaged image) in the test set. The restoration result is shown in Figure 3(b). In terms of objective evaluation indicators, the mean absolute error MAE, peak signal-to-noise ratio PSNR, and structural similarity SSIM are calculated. The evaluation indicators of the restored image are shown in Table 1.

[0091] Table 1 Evaluation indicators of restored images

[0092] Number of images / proportion MAE SSIM PSNR 300 / 3％ 0.001187 0.996993 45.495712

[0093] In this embodiment, the damaged or obscured license plate image and the repaired license plate image are input into the trained open source YOLO v8 license plate recognition system. In terms of objective evaluation indicators, the recall rate, accuracy rate, and category average pixel accuracy rate are calculated. The data comparison results are shown in Table 2.

[0094] Table 2 Data comparison results

[0095] Test Set Accuracy Recall mPA Deface or obscure the license plate image 91.5％ 71.1％ 82.3％ Repair license plate image 98.1％ 97.6％ 99.1％

[0096] It can be seen from the experimental data that the proposed license plate image restoration model that mixes multi-scale receptive field and attention mechanism generates images with clear texture, complete structure and reasonable semantics; the recognition effect enhancement experiment shows that after restoration by the model of the present invention, the efficiency and accuracy of license plate image recognition by the license plate recognition system can be greatly improved.

[0097] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A license plate image restoration method that mixes multi-scale receptive field and attention mechanism, characterized in that: The following steps are involved: S1: Network Construction Design the MHDR module and introduce the ECA module, optimize the middle layer of the encoder and decoder, and then build the license plate image restoration network; S2: Model training Construct a hybrid constrained multi-scale loss function, use the hybrid constrained multi-scale loss function to train the license plate image restoration network, and obtain the license plate image restoration model; S3: License Plate Image Restoration The license plate image to be repaired is input into the license plate image repair model to obtain the license plate repair image.

2. According to claim 1, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: In step S1, the license plate image restoration network includes an encoder and a decoder, each of which includes three MHDR modules of the same structure, and the encoder and the decoder are connected via an intermediate layer; In the encoder, the input image is processed by the first MHDR module, then processed by the ECA module and downsampled using a max pooling layer with a stride of 2, then processed by the second MHDR module, then downsampled using a max pooling layer with a stride of 2, then processed by the third MHDR module, and then downsampled using a max pooling layer with a stride of 2; In the decoder, upsampling is first performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the first MHDR module, then upsampling is performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the second MHDR module, then upsampling is performed using a deconvolution layer with a stride of 2 and a kernel size of 3*3, then processed by the third MHDR module; In order to realize the information interaction between semantic channels and texture channels and prevent the loss of hierarchical information after channel weighting, the feature information extracted by the encoder is fused with the information obtained by upsampling through skip connections.

3. According to claim 2, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: In step S1, the intermediate layers of the encoder and the decoder include a dilated convolution layer with a dilation rate of 1, a DW convolution layer, and a dilated convolution layer with a dilation rate of 5, which are connected in sequence and have convolution kernel sizes of 3*3.

4. According to claim 3, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: In step S2, the processing process of the MHDR module is as follows: first, the feature map input is input into a dilated convolution layer with a dilated rate of 2, a step size of 1, and a convolution kernel size of 3*3 to extract feature information of the image; Then the feature information is input into three branches respectively: the first branch uses a DW convolution layer with a step size of 1 and a convolution kernel size of 3*3 to increase the original number of channels to 3 times, and the input receptive field is 7 relative to the feature map; the second branch sequentially uses a cascade of a normal convolution layer with a convolution kernel size of 1*1 and a dilated convolution layer with a convolution kernel size of 3*3, and the input receptive field is 9; the third branch sequentially uses a dilated convolution layer with a dilation rate of 5, a step size of 1, and a convolution kernel size of 5*5, a normal convolution layer with a convolution kernel size of 1*1, and a DW convolution layer with a convolution kernel size of 3*3 to increase the receptive field again, and the input receptive field is 17 relative to the feature map; Finally, the Concat operation is used to fuse multi-scale features, and then a common convolution layer with a convolution kernel size of 1*1 is used to compress and aggregate the channel feature information after multi-scale feature fusion to output the feature map output.

5. According to claim 4, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: The calculation formula of the feature map output is as follows: output=Conv1(concat(G i (Conv3(input))+input)) Among them, Conv3(input) is a 3*3 convolution operation on the feature map input, concat is a multi-scale splicing operation, Conv1 is a 1*1 convolution operation on the feature map output after the concat splicing operation, G i Calculate the receptive field scale of the i-th branch, i = 1, 2, 3.

6. According to claim 5, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: In step S1, the ECA module adaptively adjusts the feature weights of each channel in the image by introducing a learnable one-dimensional convolution, so that the model can identify the importance of different channels to the restoration task, guide the information flow between channels to the texture channel and the semantic channel, improve the feature expression capability, and filter out irrelevant information and noise.

7. According to claim 5, a license plate image restoration method combining multi-scale receptive field and attention mechanism is characterized in that: In step S2, the hybrid constrained multi-scale loss function is as follows: L=λ hole L hole +λ valid L valid +λ per L per +λ sty L sty +λ tv L tv +λ adv L adv Among them, L hole Reconstruction loss for damaged area, L valid Reconstruct the loss for the non-damaged area, L per is the perceptual loss, L sty is the style loss, L tv is the total variational loss, L adv To counter the loss, λ hole , valid , per , sty , tv With λ adv are the weight parameters corresponding to the loss respectively.

8. The license plate image restoration method of claim 7, characterized in that: In the hybrid constrained multi-scale loss function: Reconstruct the loss L by the damaged area hole and the reconstruction loss L in the non-damaged area valid Calculate the pixel loss L pix : L pix =λ hole L hole +λ valid L valid L hole =||(I out -I gt )⊙I mask ||1 L valid =||(1-I mask )⊙(I out -I gt )||1 Among them, I out To repair the image, I gt is the real image, I mask is the damaged image obtained after the real image is damaged, L hole Reconstruction loss for damaged area, L valid is the reconstruction loss of the non-damaged area, λ hole , valid is the weight parameter of the reconstruction loss of the damaged area and the reconstruction loss of the non-damaged area; Perceptual loss L per The calculation formula is: Among them, φ i (I) is the i-th layer feature map extracted by the VGG-19 pre-trained network, C i , H i , W i is the number of channels, height, and width of the feature map of the i-th layer; Style loss L sty The calculation formula is: Among them, ψ i is the Gram matrix, which is used to quantify the style information in the image features; The total variational loss L tv The calculation formula is: Among them, j, l, k are I out The spatial dimension and channel dimension index of , C, H, W are the number of channels, height and width of the feature map; Adversarial loss L adv The formula is: Among them, I gt is the data distribution of real images, I out To repair the data distribution of the image, E is the mathematical expectation, G is the generator, and D is the discriminator.

Images