Film patch repairing method

Abstract

The invention is suitable for the technical field of image restoration, and provides a film patch restoration method. The invention provides a film movie patch repairing method, which comprises the following steps of: firstly, adopting a patch mask generation technology based on a U-net network, effectively reducing a false drop rate, accurately identifying a patch area in a picture and generating a patch mask for accurately segmenting the patch area; furthermore, the invention also introduces a patch repairing method based on three-domain conversion, and the method comprises the following steps: firstly, carrying out coding processing on a damaged picture, and then, carrying out fitting mapping on coding information and codes of similar intact pictures, so as to realize image restoration. According to the strategy, effective information in a time domain and a space domain is fully utilized, so that the restored image effect is more natural and smoother, and the overall quality of a movie picture is remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image restoration, and in particular relates to a method for restoring film patches. Background Art

[0002] The restoration of film patches mainly relies on two methods: physical and digital. The traditional physical restoration method is mainly to repair film by hand, such as manual painting and cutting and pasting. This method is not only time-consuming and laborious, but may also cause additional damage to the film. The digital restoration method uses telecine equipment to digitally store film movies, and then uses digital algorithms to repair the patches in the film screen.

[0003] Digital restoration methods are mainly divided into one-stage and two-stage methods. The one-stage method directly uses filters, such as median filtering, multi-level median filtering, LUM filtering, topological median filtering, etc., which have a good restoration effect on small-area patches, but the restoration effect on large-area patches is very limited. The two-stage method divides the restoration process into two stages: detection and restoration. First, the patch area is segmented to form a patch area mask, and then the patch mask is used to repair the patch area.

[0004] Typical methods for generating patch masks include: 1) Artificially draw patch masks, draw a rectangular area at the patch, and form a binary mask of the patch in the area. The mask generated by this method usually contains a large number of normal pixels, which affects the repair speed and effect of subsequent repair work. 2) Mask generation method based on edge detection, which uses the grayscale mutation and discontinuity of the patch to generate the patch mask. This method has poor segmentation ability for patches with unclear edges and small areas, and may cause misjudgment for some scenes with similar physical properties to patches. 3) Patch detection algorithm based on the difference between previous and next frames. Since there is only a small movement between consecutive adjacent frames in the movie projection screen, the single-frame burst of the patch can be used to segment the patch area. Typical algorithms include SDI (Spike Detection Index) and ROD (Rank Ordered Difference). Due to the influence of the composite degradation of film movies, a large number of false detections often occur in mixed degradation areas and complex scene areas in the picture.

[0005] Typical patch repair methods include: 1) A repair method based on bilinear interpolation. This method calculates cubic single linear interpolation in the horizontal and vertical directions, and uses the four surrounding neighboring pixels to calculate the filled interpolated pixels. This method has a good repair effect in repairing small areas with small pixel differences, but the repair effect is relatively fuzzy for large areas with large pixel differences. 2) TELEA repair method. This method is based on the (FMM) fast marching algorithm. It starts from the known pixels and advances to the missing area to repair the missing area. Therefore, the closer to the center of the missing area, the fuzzier the repaired image. This method has a general repair effect when repairing large missing areas. 3) Navier-Stokes repair algorithm. This method uses the equations of fluid mechanics to restore the image by solving partial differential equations. It can retain texture detail information, but it is difficult to complete high-quality repair tasks.

[0006] In view of the above problems, the present invention proposes a method for repairing film spots. Summary of the invention

[0007] The purpose of the present invention is to provide a method for repairing film spots, aiming to solve the problems raised in the above background technology.

[0008] The purpose of the present invention is achieved through the following technical solutions: A method for repairing film spots comprises the following steps: Step 1: Obtain the film frame to be restored, and digitally transcribe the film using telecine equipment; Step 2: Use the U-net network to segment the patch area in the image to form a patch mask; Step 3: Use a plaque repair method based on three-domain conversion to repair and reconstruct the plaque damage image; Step 4: Add the reconstructed image frame back to the movie clip to complete the restoration work.

[0009] Furthermore, the specific operations of step 2 are as follows: Step 21: preprocessing of input image; Image normalization: Set the scale_tensor parameter to normalize the size of the input image to 256×256 pixels; Step 22: Network structure configuration; Depth setting: Set the downsampling and upsampling depth of the U-net network to 4; Channel setting: both input channel and output channel are set to 1; Setting the number of feature channels: Set the initial number of feature layers of the U-net network to 64; Convolution layer configuration: In each convolution block, set the number of convolution layers to 2; Upsampling method: In the upsampling process, the deconvolution method is used; Step 23: Network training; Activation function: Select ReLU as the activation function; Normalization: After each convolution layer, the Batch Norm normalization method is used to normalize the convolutional data; Select and define the loss function: Select BCEWithLogitsLoss as the loss function to solve the binary classification problem of pixels. The loss function combines the Sigmoid function and BCELoss, and the expression is as follows: ; Among them, x is the original output of the model; y is the target value; sigmoid is the sigmoid function; Optimizer configuration: Use Adam optimizer for network training and set learning rate α is 0.001, the exponential decay rate of the first-order moment estimate β 1 is 0.9, the exponential decay rate of the second-order moment estimate β 2 is 0.999; Step 24: Plaque segmentation: Use the trained U-net network to segment the plaques in the input damaged image to obtain a binary mask for plaque segmentation.

[0010] Furthermore, the specific operations of step 3 are as follows: Step 31, define the domain; Defining the Real Damaged Image Domain R , mask covers the screen area X and intact image area Y , where the intact and undamaged image area Y It contains adjacent and close frames of the damaged picture, which are used to provide image features required for restoration; Step 32: Using VAE encoder E Encode the image into the latent space, the VAE encoder E include E R , E X and E Y ; Using VAE generator G The image encoding is restored, and the VAE generator G include G R , G X and GY ; The real damaged image domain R Images in r Through the encoder E R Encoding into latent space Z R , and get the encoded representation z r ; Cover the screen area with mask X Images in x Through the encoder E X Encoding into latent space Z X , and get the encoded representation z x ; The intact and undamaged image area Y Images in y Through the encoder E Y Encoding into latent space Z Y , and get the encoded representation z y ; In hidden space Z R and Z X The intersection of the domains forms a shared domain, and the damaged image encoding in the shared domain is represented as z x∩r ; G R , G X and G Y Decode the encoded representation of the image to assist the encoder E R , E X and E Y The training is as follows: G R The encoded representation z r Decode and restore the image r ; G X The encoded representation z x Decode and restore the image x ; G Y The encoded representation z y Decode and restore the imagey ; Step 33, feature mapping; Independent training of the damaged domain Z R and Z X Shared domains and intact domains Z Y The mapping between T Z , mapping T Z Used to convert the encoded information of the shared domain to the intact domain Z Y ; Step 34: Using the VAE generator G Will be mapped T Z Map to Z Y The information of the domain is restored to an image: through mapping T Z Will z x∩r Towards z y Perform fitting mapping, and then G Y Decoding is performed to restore the image; Damaged image r The repair process is expressed as follows: ; in, is the result of transforming the damaged image r from the R domain to the Y domain; is the VAE generator; is the feature mapping transformation; is the VAE encoder; " indicates the connection of each process.

[0011] Furthermore, in the VAE encoder, assuming that the image data in the latent space obeys a Gaussian distribution, the real damaged image domain R The VAE loss function is expressed as: ; in, For the real damaged image domain R VAE loss function; the first term of the loss function is the encoder loss, KL is the KL divergence, is encoded under the condition of sample image r The mathematical expectation of The mean is 0 and the variance is The second term of the loss function is the generator loss, is the weight coefficient of the generator loss, is the mathematical expectation, is the generated result of the image, is an image sample; the third term of the loss function is , represents the LSGAN loss.

[0012] Furthermore, a discriminator is introduced to expand the shared domain, and the overall objective function for encoding images r and x is obtained as: ; in, Represents the real damaged image domain R、 Mask covers the screen area X The encoder loss of Represents the real damaged image domain R、 Mask covers the screen area X The generator loss of For the real damaged image domain R、 Mask covers the screen area X The discriminator loss of For the real damaged image domain R VAE loss function; Mask covers the screen area X VAE loss function; is the LSGAN loss introduced in the latent space.

[0013] Furthermore, by learning to train the encoded image pairs { z x∩r , z y} T Z , to repair and reconstruct the image; mapping T Z The loss function is as follows: ; in, z x∩r is the encoded representation of the damaged image in the shared domain; is the weight coefficient of the L1 loss of mapping T; is the L1 loss of mapping T, , using L1 norm for constraint; is the mapping transformation of the encoding latent space, E is the mathematical expectation; The least squares loss LSGAN is introduced; is the weight coefficient of feature matching loss; is the introduced feature matching loss, Matching Discriminator And the multi-level feature layer in the VGG network, the expression is as follows: ; in, Representation Discriminator The i-th feature layer; Represents the i-th feature layer of the VGG network; Representation Discriminator The number of activations of the i-th feature layer; Represents the number of activations of the i-th feature layer of the VGG network.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method for repairing film and movie patches. The method first adopts a patch mask generation technology based on a U-net network, which can effectively reduce the false detection rate, accurately identify the patch area in the picture, and generate a patch mask that accurately segments the patch area. Furthermore, the present invention also introduces a patch repair method based on three-domain conversion. This strategy first encodes the damaged picture, and then fits and maps the encoded information with the encoding of adjacent intact pictures to achieve image restoration. This strategy makes full use of the effective information in the time domain and the space domain, making the repaired image effect more natural and smooth, and significantly improving the overall quality of the movie picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 This is the structural diagram of the U-net network.

[0017] Figure 3 Schematic diagram of the plaque repair method based on three-domain transformation.

[0018] Figure 4 The results of the U-net-based patch mask generation method proposed in the present invention are compared with those of the traditional detection method; (A) is the test image, (B) is the SDI detection result, (C) is the ROD detection, (D) is the edge detection, and (E) is the U-net detection.

[0019] Figure 5 The results of the plaque repair method based on three-domain conversion proposed in the present invention are compared with those of the traditional repair method; (A) is the test image, (B) is the TELEA repair algorithm, (C) is the Navier-Stokes repair algorithm, and (D) is the plaque repair method based on three-domain conversion.

[0020] Figure 6 for Figure 5The enlarged image in the circle; (A) Figure 5 The enlarged image in the circle (A) and (B) Figure 5 The enlarged image in circle (B) and (C) Figure 5 The enlarged image in circle (C) and the Figure 5 Magnified image of the circle in (D). DETAILED DESCRIPTION

[0021] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be understood as limiting the scope of the present invention. The experimental methods described in the following examples are conventional methods unless otherwise specified; the reagents and materials can be obtained from commercial sources unless otherwise specified.

[0022] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0023] like Figure 1 As shown, a method for repairing film spots provided by an embodiment of the present invention comprises the following steps: Step 1: Obtain the film frame to be restored, and digitally transcribe the film using telecine equipment; Step 2: Use the U-net network to segment the patch area in the image to form a patch mask; The structure of the U-net network is as follows Figure 2 As shown. Figure 2The explanation is divided into two parts, the left part is the downsampling process of the network. Input a 256×256 image, and perform convolution feature extraction operation through 64 3×3 convolution kernels to obtain 64 256×256×1 feature layers. Then use 3×3 convolution kernels to perform convolution operation on the 64 feature layers to obtain 64 256×256×1 feature layers. After the first layer is processed, a 256×256×64 feature layer is finally obtained. Then, the 256×256×64 feature layer is downsampled through a 2×2 pooling layer, so that the feature layer size becomes half of the original, and then the same feature extraction operation is performed on the obtained feature layer. For each downsampling layer, the number of feature layers doubles and the size is halved, and so on. Finally, 512 layers of feature layers with a size of 16×16 are obtained. The right part is the upsampling process of the network. Starting from the lower right corner, the 16×16×1024 feature layer is deconvolved using 512 2×2 convolution kernels to expand the image to 32×32×512. Since deconvolution can only expand the feature layer but cannot fill in data. In order to reduce data loss, the jump connection method is used to supplement the data. After the matrix is ​​concatenated, the feature layer becomes 32×32×1024 in size, and then the convolution kernel is used to convolve the feature layer to fuse it. With each upsampling layer, the size of the feature layer doubles and the number is reduced by half. The final upsampling result is 256×256×64. In the last step, the convolution kernel is used to reduce the 64 feature layers to 2, completing the binary classification of the image and dividing the entire image into two categories: background and target.

[0024] The specific operation of step 2 is as follows: Step 21: preprocessing of input image; Adjust the input size of the U-net network to 256×256 pixels; Image standardization: For images to be segmented of different sizes, set the scale_tensor parameter to standardize the size of the input image to make it suitable for network processing.

[0025] Step 22: Network structure configuration; Depth setting: Set the downsampling and upsampling depth of the U-net network to 4; Channel settings: Both the input channel (in_channels) and the output channel (out_channels) are set to 1, that is, the input is a single-channel grayscale image, and the output is a single-channel binary segmentation mask; Setting the number of feature channels: The initial wf (width factor) is set to 6, that is, the initial number of feature layers = 2 6 =64; Convolution layer configuration: In each convolution block, set the number of convolution layers conv_num to 2 to increase the complexity of feature extraction and enable the network to learn more complex feature representations.

[0026] Upsampling method: In the upsampling process, the deconvolution method is used instead of bilinear interpolation; Step 23: Network training; Activation function: Select ReLU as the activation function; Normalization: After each convolution layer, the Batch Norm normalization method is used to normalize the convolutional data; Select and define the loss function: BCEWithLogitsLoss is selected as the loss function to solve the binary classification problem of patch pixels. The loss function combines the Sigmoid function and BCELoss, and the expression is as follows: ; Among them, x is the original output of the model; y is the target value (0 or 1); sigmoid is the sigmoid function; Optimizer configuration: Use Adam optimizer for network training and set learning rate α is 0.001, the exponential decay rate of the first-order moment estimate β 1 is 0.9, the exponential decay rate of the second-order moment estimate β 2 is 0.999; Step 24: Plaque segmentation: Use the trained U-net network to segment the plaques in the input damaged image to obtain a binary mask for plaque segmentation.

[0027] Step 3: Use the plaque repair method based on three-domain conversion to repair and reconstruct the plaque damage image. The principle is as follows: Figure 3 As shown; The specific operations of step 3 are as follows: Step 31, define the domain; Define the domain of real damaged images (instances) R , Mask covering screen (label) area X and intact picture (feature) domain Y , where the intact picture (feature) domain Y It contains adjacent and close frames of the damaged picture, which are used to provide image features required for restoration; Step 32: Using VAE encoder E Encode the image into the latent space, the VAE encoder E include E R , E X and EY ; Using VAE generator G The image encoding is restored, and the VAE generator G include G R , G X and G Y ; The real damaged image domain R Images in r Through the encoder E R Encoding into latent space Z R , and get the encoded representation z r ; Cover the screen area with mask X Images in x Through the encoder E X Encoding into latent space Z X , and get the encoded representation z x ; The intact and undamaged image area Y Images in y Through the encoder E Y Encoding into latent space Z Y , and get the encoded representation z y ; In hidden space Z R and Z X The intersection of the domains forms a shared domain, and the damaged image encoding in the shared domain is represented as z x∩r ; G R , G X and G Y Decode the encoded representation of the image to assist the encoder E R , E X and E Y The training is as follows: G R The encoded representation z r Decode and restore the image r ; GX The encoded representation z x Decode and restore the image x ; G Y The encoded representation z y Decode and restore the image y ; Step 33, feature mapping; Independent training of the damaged domain Z R and Z X Shared domains and intact domains Z Y The mapping between T Z , mapping T Z Used to convert the encoded information of the damaged domain to the intact domain Z Y ; Step 34: Using the VAE generator G Will be mapped T Z Map to Z Y The information of the domain is restored to an image: through mapping T Z Will z x∩r Towards z y Perform fitting mapping, and then G Y Decoding is performed to restore the image; Damaged image r The repair process is expressed as follows: ; in, is the result of transforming the damaged image r from the R domain to the Y domain; is the VAE generator; is the feature mapping transformation; is the VAE encoder; " indicates the connection of each process.

[0028] VAE encoder E, generator G; We first encode the damaged image into the latent space, assume that the image data in the latent space follows a Gaussian distribution, and use the reparameterization technique to sample the image data in the latent space so that the image can be reconstructed in the latent space.

[0029] The loss function for encoding images r, x, y can be expressed as follows: For example: ; The loss function consists of three parts: the first part uses KL divergence to measure the error between the two probability distributions and constrains the encoder to obtain z r , so that it obeys Gaussian distribution. The second part is to constrain the image information restored by the generator so that the encoding can extract the main features of the image. The third part introduces the least squares loss LSGAN to solve the over-smoothing phenomenon that often occurs in VAE encoders and improve the quality of encoded images.

[0030] in, is the loss function of the real damaged picture domain R; the first term of the loss function is the encoder loss, and KL is the KL divergence; is encoded under the condition of sample image r The mathematical expectation of The mean is 0 and the variance is The second term of the loss function is the generator loss, is the weight coefficient of the generator loss, is the mathematical expectation, is the generated result of the image, For image samples, the third term of the loss function is , represents the LSGAN loss.

[0031] In order to expand the shared domain, a discriminator is introduced, and the loss function is as follows: ; in, LSGAN loss introduced in the latent space; is the mathematical expectation of the loss of the mask coverage area X; For the real damaged image domain R Mathematical expectation of loss; Represents the real damaged image domain R、 Mask covers the screen area X The discriminator loss of For the real damaged image domain R、 Mask covers the screen area X The encoding of the image x; For the real damaged image domain R、 Mask covers the screen area X The encoding of the image r; The total objective function for encoding images r and x is obtained as: ; in, Represents the real damaged image domain R、 Mask covers the screen area X The encoder loss of Represents the real damaged image domain R、 Mask covers the screen area X The generator loss of Represents the real damaged image domain R、 Mask covers the screen area X The discriminator loss of For the real damaged image domain R The loss function of Mask covers the screen area X The loss function of is the LSGAN loss introduced in the latent space.

[0032] (2) Mapping T Z ; By learning to train the encoded image pairs { z x∩r , z y} T Z , to repair and reconstruct the image; mapping T Z The loss function is as follows: ; in, z x∩r Encode representations for corrupted images in a shared domain; is the weight coefficient of the L1 loss of mapping T; is the L1 loss of mapping T, , using L1 norm for constraint; is the mapping transformation of the encoding latent space; E is the mathematical expectation; The least squares loss LSGAN is introduced; is the weight coefficient of feature matching loss; is the introduced feature matching loss, Matching Discriminator And the multi-level feature layer in the VGG network, the expression is as follows: ; in, Representation Discriminator The i-th feature layer; Represents the i-th feature layer of the VGG network; Representation Discriminator The number of activations of the i-th feature layer; Represents the number of activations of the i-th feature layer of the VGG network.

[0033] Step 4: Add the reconstructed image frame back to the movie clip to complete the restoration work.

[0034] Example 1: Comparison of mask generation effects; like Figure 4 As shown in (A)-(E), traditional detection methods (SDI detection, ROD detection, edge detection) are easily affected by complex background environments and noise, and there are a lot of false detections in the generated patch mask images. Figure 4 As shown in (E), the patch mask generation technology based on U-net proposed in the present invention can effectively reduce false detection, accurately identify the patch area in the picture, and generate a patch mask that accurately segments the patch area.

[0035] like Figure 5 As shown in (A)-(D), the details are magnified to show Figure 6 From the results of (A)-(D), we can see that traditional repair methods (TELEA repair algorithm, Navier-Stokes repair algorithm) have good repair effects for small spots, but when repairing large patches, the repair effect is poor due to the lack of sufficient image information. The underlying principle of the algorithm is to start from the edge of the image and repair towards the center of the image. Therefore, the closer to the center of the patch, the worse the repair effect. In the repaired area, unnatural, disconnected, and non-smooth areas can be observed. Figure 6 As shown in (D), the plaque repair method based on three-domain conversion proposed in the present invention utilizes the intact picture information of adjacent frames to fit and map the damaged image to the intact image, which has a better image repair effect and makes the repaired image more natural and smooth.

[0036] We use three indicators, PSNR, SSIM, and MAE, to measure the image restoration effect. The indicators are shown in Table 1.

[0037] Table 1 Image restoration effects method PSNR SSIM MAE TELEA 28.7934 0.6403 0.0627 Navier-Stokes 29.7781 0.6714 0.0586 Plaque repair method based on three-domain transformation 30.6683 0.6819 0.0558 It can be seen from Table 1 that the plaque repair method based on three-domain conversion proposed in the present invention has the highest PSNR peak signal-to-noise ratio, the highest SSIM structural similarity and the lowest MAE mean absolute error, and the overall repair performance is the best.

[0038] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for repairing film plaques, characterized in that: The following steps are involved: Step 1: Obtain the film frame to be restored, and digitally transcribe the film using telecine equipment; Step 2: Use the U-net network to segment the patch area in the image to form a patch mask; Step 3: Use a plaque repair method based on three-domain conversion to repair and reconstruct the plaque damage image; Step 4: Add the reconstructed image frame back to the movie clip to complete the restoration work.

2. The method for repairing film spots according to claim 1, characterized in that: The specific operation of step 2 is as follows: Step 21: preprocessing of input image; Image normalization: Set the scale_tensor parameter to normalize the size of the input image to 256×256 pixels; Step 22: Network structure configuration; Depth setting: Set the downsampling and upsampling depth of the U-net network to 4; Channel setting: both input channel and output channel are set to 1; Setting the number of feature channels: Set the initial number of feature layers of the U-net network to 64; Convolution layer configuration: In each convolution block, set the number of convolution layers to 2; Upsampling method: In the upsampling process, the deconvolution method is used; Step 23: Network training; Activation function: Select ReLU as the activation function; Normalization: After each convolution layer, the Batch Norm normalization method is used to normalize the convolutional data; Select and define the loss function: Select BCEWithLogitsLoss as the loss function to solve the binary classification problem of pixels. The loss function combines the Sigmoid function and BCELoss, and the expression is as follows: ； Among them, x is the original output of the model; y is the target value; sigmoid is the sigmoid function; Optimizer configuration: Use Adam optimizer for network training and set learning rate α is 0.001, the exponential decay rate of the first-order moment estimate β 1 is 0.9, the exponential decay rate of the second-order moment estimate β 2 is 0.999; Step 24: Plaque segmentation: Use the trained U-net network to segment the plaques in the input damaged image to obtain a binary mask for plaque segmentation.

3. The method for repairing film spots according to claim 1, characterized in that: The specific operations of step 3 are as follows: Step 31, define the domain; Defining the Real Damaged Image Domain R , mask covers the screen area X and intact image area Y , where the intact and undamaged image area Y It contains adjacent and close frames of the damaged picture, which are used to provide image features required for restoration; Step 32: Using VAE encoder E Encode the image into the latent space, the VAE encoder E include E R , E X and E Y ; Using VAE generator G The image encoding is restored, the VAE generator G include G R , G X and G Y ; The real damaged image domain R Images in r Through the encoder E R Encoding into latent space Z R , and get the encoded representation z r ; Cover the screen area with mask X Images in x Through the encoder E X Encoding into latent space Z X , and get the encoded representation z x ; The intact and undamaged image area Y Images in y Through the encoder E Y Encoding into latent space Z Y , and get the encoded representation z y ; In hidden space Z R and Z X The intersection of the domains forms a shared domain, and the damaged image encoding in the shared domain is represented as z x∩r ; G R , G X and G Y Decode the encoded representation of the image to assist the encoder E R , E X and E Y The training is as follows: G R The encoded representation z r Decode and restore the image r ; G X The encoded representation z x Decode and restore the image x ; G Y The encoded representation z y Decode and restore the image y ; step 33. Feature mapping; Independent training of the damaged domain Z R and Z X Shared domains and intact domains Z Y The mapping between T Z , mapping T Z Used to convert the encoded information of the shared domain to the intact domain Z Y ; Step 34: Using the VAE generator G Y Will be mapped T Z Map to Z Y The information of the domain is restored to an image: through mapping T Z Will z x∩r Towards z y Perform fitting mapping, and then G Y Decoding is performed to restore the image; Damaged image r The repair process is expressed as follows: ； in, is the result of transforming the damaged image r from the R domain to the Y domain; is the VAE generator; is the feature mapping transformation; is the VAE encoder; " indicates the connection of each process.

4. The method for repairing film spots according to claim 3, characterized in that: In the VAE encoder, assuming that the image data in the latent space obeys Gaussian distribution, the real damaged picture domain R The VAE loss function is expressed as: ； in, For the real damaged image domain R VAE loss function; the first term of the loss function is the encoder loss, KL is the KL divergence, Under the condition of sample image r The mathematical expectation of The mean is 0 and the variance is The second term of the loss function is the generator loss, is the weight coefficient of the generator loss, is the mathematical expectation, is the generated result of the image, is an image sample; the third term of the loss function is , represents the LSGAN loss.

5. The method for repairing film spots according to claim 3, characterized in that: The discriminator is introduced to expand the shared domain, and the overall objective function for encoding the image r, x is obtained as: ； in, Represents the real damaged image domain R、 Mask covers the screen area X The encoder loss of Represents the real damaged image domain R、 Mask covers the screen area X The generator loss of For the real damaged image domain R、 Mask covers the screen area X The discriminator loss of For the real damaged image domain R VAE loss function; Mask covers the screen area X VAE loss function; is the LSGAN loss introduced in the latent space.

6. The method for repairing film spots according to claim 3, characterized in that: By learning to train the encoded image pairs { z x∩r , z y } T Z , to repair and reconstruct the image; mapping T Z The loss function is as follows: ； in, z x∩r Encode representations for corrupted images in a shared domain; is the weight coefficient of the L1 loss of mapping T; is the L1 loss of mapping T, , using L1 norm for constraint; is the mapping transformation of the encoding latent space, E is the mathematical expectation; The least squares loss LSGAN is introduced; is the weight coefficient of feature matching loss; is the introduced feature matching loss, Matching Discriminator And the multi-level feature layer in the VGG network, the expression is as follows: ； in, Representation Discriminator The i-th feature layer; Represents the i-th feature layer of the VGG network; Representation Discriminator The number of activations of the i-th feature layer; Represents the number of activations of the i-th feature layer of the VGG network.

Images