A Mongolian pattern repair method

Abstract

The present invention relates to the field of image restoration technology, and specifically provides a Mongolian pattern restoration method, comprising the steps of: constructing and training a pattern restoration model; obtaining an original image of a Mongolian pattern to be restored; inputting the original image into a trained global coarse restoration network for a first restoration, and obtaining a preliminary restoration image of the Mongolian pattern to be restored; inputting the preliminary restoration image into a trained local refinement network for a second restoration, and obtaining a secondary restoration image of the Mongolian pattern to be restored; inputting the secondary restoration image into a trained global refinement network for a third restoration, and obtaining a final restoration image of the Mongolian pattern to be restored. Through the above technical solution, the technical problems of inaccurate feature recognition, lack of style consistency, difficulty in model training, and limited restoration effect in the existing image restoration technology when processing Mongolian patterns are solved.

Description

Technical Field

[0001] The invention relates to the technical field of image restoration, and specifically provides a Mongolian pattern restoration method. Background Art

[0002] Image restoration refers to the process of reconstructing lost or damaged parts of images and videos. For the restoration of Mongolian patterns, existing image restoration technologies have some limitations, including the following issues:

[0003] 1. Inaccurate feature recognition: Existing technical models have difficulty recognizing the unique cultural symbols and complex pattern structures of Mongolian patterns, which may lead to deformation, distortion of elements, or distortion of patterns during the restoration process.

[0004] 2. Lack of style consistency: Current technology makes it difficult to maintain the unique artistic style and color matching rules of Mongolian patterns, which will lead to a break in the style of the restored area and the original image, thus affecting the overall aesthetic and cultural identity.

[0005] 3. Difficulty in model training: Currently, professional data on Mongolian patterns are very scarce, which may cause overfitting of the model during training and fail to fully learn the diversity and complexity of Mongolian patterns, thus affecting its generalization ability.

[0006] 4. The restoration effect is limited: there is a lack of targeted training data, so when faced with unseen or uncommon Mongolian patterns, the restoration effect of the model will be significantly reduced, resulting in failure to meet actual application needs.

[0007] Accordingly, the art needs a repair solution for Mongolian patterns to solve the above problems. Summary of the invention

[0008] In order to overcome the above-mentioned defects, the present invention is proposed to solve the technical problems of the existing image restoration technology in processing Mongolian patterns, such as inaccurate feature recognition, lack of style consistency, difficulty in model training and limited restoration effect.

[0009] The present invention provides a Mongolian pattern repair method, comprising the following steps:

[0010] S1: constructing a pattern repair model, wherein the pattern repair model includes a global coarse repair network, a local refinement network and a global refinement network connected in sequence;

[0011] S2: training pattern inpainting model;

[0012] S3: obtaining the original image of the Mongolian pattern to be restored;

[0013] S4: inputting the original image into the trained global coarse restoration network for a first restoration, to obtain a preliminary restoration image of the Mongolian pattern to be restored;

[0014] S5: inputting the preliminary repaired image into the trained local refinement network for a second repair, to obtain a secondary repaired image of the Mongolian pattern to be repaired;

[0015] S6: Inputting the secondary repaired image into the trained global refinement network for a third repair, to obtain a final repaired image of the Mongolian pattern to be repaired.

[0016] Furthermore, the global coarse repair network includes a pyramid context encoder and a multi-scale decoder connected in sequence; the local refinement network includes a downsampling block, an MPD downsampling block, a RIC-C convolution block, a residual block and an upsampling block connected in sequence; the global refinement network includes three attention modules and eight downsampling modules and eight upsampling modules connected in sequence, and the three attention modules are respectively interspersed between the fourth upsampling module to the seventh upsampling module. The MPD downsampling block is composed of a 3×3 convolution, a pixel shuffle downsampling PD, interlaced scanning and a mask separable convolution, and the interlaced scanning is a series connection of features and mask slices; the RIC-C convolution block is a rotation-invariant coordinate convolution block, which processes the rotated image by using the rotation invariance of the convolution operation.

[0017] Further, the step S4 comprises the steps of:

[0018] S41: Learn the latent feature information of each layer of the original image through the pyramid context encoder, and use the attention transfer network to learn the regional affinity of each layer of the original image;

[0019] S42: According to the regional affinity of each layer, feature restoration is performed on the missing region of the adjacent low level in the original image to obtain reconstructed feature information;

[0020] S43: fusing the latent feature information and the reconstructed feature information through a multi-scale decoder to obtain a preliminary restored image.

[0021] Further, the step S5 comprises the steps of:

[0022] S51: enlarging the preliminary repaired image through a downsampling block;

[0023] S52: using a mask to identify important information in the enlarged preliminary restoration image through the MPD downsampling block, and then retaining the important information through rearrangement;

[0024] S53: performing rotation invariance processing on the image output from step S52 through a RIC-C convolution block;

[0025] S54: outputting residual information of the image through residual block learning step S53;

[0026] S55: The image outputted in step S54 is enlarged by an upsampling block to obtain a secondary repaired image.

[0027] Further, the step S3 comprises the steps of:

[0028] Get a training set of Mongolian patterns;

[0029] The training set of Mongolian patterns is input into the global coarse restoration network to obtain the preliminary restoration image of the training set;

[0030] The preliminary repaired images of the training set are input into the discriminator, divided into multiple small blocks, and each small block is evaluated whether it is a real image;

[0031] Input the initial restoration image of the training set into the local refinement network to obtain the secondary restoration image of the training set;

[0032] Input the secondary repaired image of the training set into the global refinement network to obtain the final repaired image of the training set;

[0033] The pattern inpainting model is trained according to the total loss function, Among them, the first loss function The second loss function is used to measure the degree of training of the global coarse repair network. Used to measure the degree of training of the local refinement network, the third loss function Used to measure the degree of training of the global refinement network, the fourth loss function Used to measure how well the discriminator is trained.

[0034] Furthermore, the first loss function The first reconstruction loss function The adversarial loss function is used to represent the pixel-level difference between the initial repaired image and the original image of the training set. Used to indicate the authenticity of the discriminator output data, where

[0035] The first reconstruction loss function

[0036]

[0037] Adversarial Loss Function

[0038]

[0039] I gt represents the original image, represents the initial repair image, I in represents the image input to the network, express and M, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, and λ h and λ g Represents the balance factor.

[0040] Furthermore, the second loss function The second reconstruction loss function Used to represent the pixel-level difference between the secondary repaired image and the original image of the training set, the first total score variable loss function The first perceptual loss function is used to represent the difference between adjacent pixels in the secondary restoration image of the training set. The first style loss function is used to represent the semantic similarity between the secondary repaired image and the original image of the training set. It is used to represent the style characteristics between the secondary restoration image and the original image of the training set, where:

[0041] The second reconstruction loss function

[0042]

[0043] The first total score variable loss function

[0044] First Perceptual Loss Function

[0045] First style loss function

[0046] Lagum Matrix

[0047] k, j represent the coordinates of the synthesized image, i represents the number of layers of the feature map, and I gt represents the original image, represents the secondary repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv ,per , sty Both represent balance factors.

[0048] Furthermore, the third loss function The third reconstruction loss function The second total score variable loss function is used to represent the pixel-level difference between the final repaired image of the training set and the training set of Mongolian patterns. The second perceptual loss function is used to represent the difference between adjacent pixels in the final repaired image of the training set. The second style loss function is used to represent the semantic similarity between the final restored image of the training set and the training set of Mongolian patterns. It is used to represent the style characteristics between the final restoration image of the training set and the training set of Mongolian patterns, where

[0049] The third reconstruction loss function

[0050]

[0051] Second total score variable loss function

[0052] Second Perceptual Loss Function

[0053] Second style loss function

[0054] Lagum Matrix

[0055] k, j represent the coordinates of the synthesized image, i represents the number of layers of the feature map, and I gt represents the original image, represents the final repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv , per , sty Both represent balance factors.

[0056] Furthermore, the fourth loss function

[0057] Among them, Pdata(I) represents the distribution of the original image, Represents the expected original image I gt The loss of images recognized as real by the discriminator, Indicates the expected generated image I mer ~p is the loss of images identified as fake by the discriminator, I gt represents the original image, express and the merged image of M.

[0058] Working principle and beneficial effects of the present invention:

[0059] In the technical solution of the present invention, the constructed pattern repair model is divided into a network structure with three stages of processing. First, the network in the first stage mainly performs a global rough repair process on the entire image; secondly, after being processed by the global rough repair network, it is input into the network in the second stage, and the network mainly repairs the local features of the image; finally, the image processed by the local refinement network is input into the network in the third stage, so as to perform an overall refined repair on the global structure of the image, which can achieve accurate repair of the texture and color structure in the Mongolian pattern and reduce the probability of texture heterogeneity, color loss, distortion, distortion and other phenomena in the pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0061] Figure 1 It is a schematic flow chart of the main steps of a Mongolian pattern repair method of the present invention;

[0062] Figure 2 It is a network overall framework diagram of a Mongolian pattern repair method of the present invention;

[0063] Figure 3 is a schematic diagram of the network structure of the global rough repair network in the present invention;

[0064] Figure 4 is a schematic diagram of the network structure of the local refinement network in the present invention;

[0065] Figure 5 is a schematic diagram of the network structure of the global refinement network in the present invention;

[0066] Figure 6 It is a schematic diagram of the module structure of the MPD downsampling block in the present invention. DETAILED DESCRIPTION

[0067] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0068] Figure 1 This is a schematic flow chart of the main steps of a Mongolian pattern repair method of the present invention. Figure 2 It is a network overall framework diagram of a Mongolian pattern repair method of the present invention.

[0069] The model structure proposed in this invention is divided into a network model processed in three stages, such as Figure 2 As shown in the figure, firstly, the network in the first stage mainly performs a global rough repair process on the whole image; secondly, after being processed by the global rough repair network, it is input into the network in the second stage, which mainly repairs the local features of the image; finally, the image processed by the local refinement network is input into the network in the third stage, so as to perform an overall refined repair on the global structure of the image, which can achieve accurate repair of the texture and color structure in the Mongolian pattern and reduce the probability of texture heterogeneity, color loss, distortion, distortion and other phenomena in the pattern.

[0070] like Figure 1-2 As shown, a Mongolian pattern repairing method in this embodiment mainly includes the following steps S1 to S6.

[0071] S1: constructing a pattern repair model, which includes a global coarse repair network, a local refinement network and a global refinement network connected in sequence;

[0072] S2: training pattern inpainting model;

[0073] S3: obtaining the original image of the Mongolian pattern to be restored;

[0074] S4: inputting the original image into the trained global coarse restoration network for the first restoration, and obtaining a preliminary restoration image of the Mongolian pattern to be restored;

[0075] S5: inputting the preliminary repaired image into the trained local refinement network for a second repair, thereby obtaining a secondary repaired image of the Mongolian pattern to be repaired;

[0076] S6: Input the secondary repaired image into the trained global refinement network for the third repair, and obtain the final repaired image of the Mongolian pattern to be repaired.

[0077] Specifically, the construction, training and image processing processes of the global coarse inpainting network, the local refinement network and the global refinement network are described below.

[0078] 1. Global coarse repair network

[0079] Figure 3 Schematic diagram of the network structure of the global rough repair network in the present invention. Figure 3 As shown, the global rough repair network in this embodiment is represented as Net P , including a pyramid context encoder and a multi-scale decoder connected in sequence. The purpose of this network architecture is to effectively combine the damaged Mongolian pattern with the context information to achieve a preliminary global rough repair effect, and provide more accurate results for the subsequent two-stage network repair.

[0080] For the image input to the global coarse restoration network, it first enters the pyramid context encoder, which uses a hierarchical pyramid process strategy to learn the potential feature information of each layer in the Mongolian pattern, and uses the attention transfer network to learn the regional affinity of each layer in the Mongolian pattern, and uses the attention transfer network (ATN) layer by layer to fill the missing areas from the adjacent high-level semantic features to the low-level semantic features. After being processed by the pyramid context encoder, it will be processed by the multi-scale decoder. The multi-scale decoder will simultaneously accept the reconstructed feature information generated by the ATN and the potential feature information obtained from the pyramid context encoder, and fuse them. The final generated preliminary restoration image is highly coordinated with the context of the Mongolian pattern in terms of semantic level and texture.

[0081] In the present embodiment, the processing process of the pyramid context encoder is based on the assumption that pixels with similar semantics should have similar detail characteristics, so that the attention transfer network (ATN) learns and quantifies the relationship between the regional affinity of each layer, and then reflects the semantic association between the missing area and the surrounding pixels through the regional affinity information. Specifically, the attention transfer network (ATN) first extracts the regional affinity from the semantic features of the current level, and uses this as a basis to guide the internal and surrounding pixel features of the missing area part in the adjacent low-level features to migrate and merge. The migration process here is mainly to complete the feature information related to the context of the missing area, so as to ensure that the filled content is consistent with the semantics in the overall Mongolian pattern, and to maximize the capture of the detail texture in the pattern.

[0082] When training the global coarse inpainting network, in order to reduce the blurring effect of the image output by the network and enhance the authenticity of the image inpainting results, a patch-based discriminator is added after the network to continue processing the training set of Mongolian patterns. The main processing process of the discriminator is: for the image output by the global coarse inpainting network, it is divided into many small image blocks (or patches), the size of each image block is usually fixed, and then each image block is independently identified, and the image block is judged by binary classification whether it is a real image (from the real data distribution) or a forged image (generated by the pyramid context encoding network). In this way, the quality of the generated image can be comprehensively evaluated, and feedback can be provided to guide model training, and then the quality of the generated image can be gradually improved, thereby helping the global coarse inpainting network to better learn to generate real images.

[0083] In the global coarse repair network (Net P ), the first loss function is used to measure the difference between the global coarse repair network output and the true label, and the parameters of the global coarse repair network are optimized to minimize the first loss function. By the first reconstruction loss function With adversarial loss function composition.

[0084] The first reconstruction loss function is used to measure the pixel-level difference between the initial restoration image of the training set and the original image. The first reconstruction loss function uses the weighted L1 loss to reconstruct pixels. Weighted L1 means that when calculating the L1 loss, the loss value of each training set sample will be weighted.

[0085] Specifically, the first reconstruction loss function is

[0086] Reconstruction loss of the first effective area It measures the reconstruction accuracy of the network in the undamaged area

[0087] Reconstruction loss in the first hole region It measures the reconstruction accuracy of the network in the damaged area (hole area).

[0088] The adversarial loss function is a loss function proposed by the discriminator in the generative adversarial network (GAN) to represent the authenticity of the discriminator output data. Its purpose is to make the global coarse repair network: this network is responsible for the preliminary generation of the repaired image, which can also be understood as the generator part of the GAN. The output generated is more realistic, thereby deceiving the discriminator. The global coarse repair network in this embodiment is responsible for the preliminary generation of the repaired image, which can also be understood as the generator part of the GAN. The minimization of the adversarial loss prompts the global coarse repair network to generate samples similar to the real data distribution, thereby improving the generation ability of the global coarse repair network and the authenticity of the generated samples.

[0089] Specifically, the adversarial loss function Indicates expected value calculation. mer ~PI mer (I mer ) This part represents the distribution PI mer Medium Sampling I mer I mer is the merged image, which refers to the image that contains the repaired area in the first stage and the undamaged area of ​​the input image. mer (I mer ) represents the distribution of these merged images. ( In this paper, D represents the discriminator, is the merged image. The overall is to calculate the squared difference between the discriminator output and the target value.

[0090]

[0091] The fourth loss function is used to measure the degree of training of the discriminator. Specifically, the fourth loss function

[0092] Pdata(I) represents the distribution of the original image, Represents the expected original image I gt The loss for images that are recognized as real by the discriminator. Indicates the expected generated image I mer ~p is the loss of images identified as fake by the discriminator.

[0093] I gt represents the original image, represents the initial repair image, I in represents the image input to the network, express and the merged image of M, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the undamaged area (valid area), Sum(M) represents the total number of pixels in the damaged area (hole area), and λ h and λ g Represents the balance factor. Specifically, the parameter is set to λ through experiments. h =6 and λ g =0.1.

[0094] 2. Local Refinement Network

[0095] Figure 4 Schematic diagram of the network structure of the local refined network in the present invention. Figure 4 As shown, the local refinement network in this embodiment is represented as Net L , including a downsampling block, two MPD downsampling blocks, a RIC-C convolution block, four residual blocks and two upsampling blocks connected in sequence, where each downsampling block and upsampling block consists of two convolution blocks. The purpose of this network architecture is to perform further local refinement and restoration work on the image processed by the global coarse restoration network.

[0096] For the image input to the local refinement network, a downsampling block is first used to enlarge the preliminary repaired image, and then the MPD downsampling block uses a mask to identify the important information in the enlarged preliminary repaired image, and then the important information is retained by rearranging. The MPD downsampling block has the characteristics of more refined downsampling operations and the ability to maintain spatial details. It can retain all valid information and can greatly reduce the loss of information in the image. The image output by the MPD downsampling block is then processed for rotation invariance through the RIC-C convolution block, and the residual information of the image output by the RIC-C convolution block is learned through the residual block; finally, the image output by the residual block is enlarged through two upsampling blocks to obtain a secondary repaired image.

[0097] In this embodiment, since the training set is expanded by rotation, enlargement, etc. in the model training, the RIC-C convolution block can have a good effect mainly for such expanded images. RIC-C (rotation-invariant coordinate convolution) is a new type of convolution operation, which mainly uses the rotation invariance of the convolution operation to process the rotated images in the data. This is because the traditional convolution operation is very sensitive to the input rotated image, and the convolution kernel will respond differently to different angles of the rotated image, which will affect the performance of the model in the end. In response to this problem, the RIC-C convolution block has a special design and rotation operation, which allows the convolution kernel to rotate the input image and keep the convolution response the same, thereby achieving the feature processing of the rotation invariance of the rotated image.

[0098] In the local refinement network Net L In the training, the second loss function is used to measure the difference between the local refinement network output and the true label, and the parameters of the local refinement network are optimized to minimize the second loss function. By the second reconstruction loss function The first total score variable loss function First Perceptual Loss Function And the first style loss function composition.

[0099] The second reconstruction loss function It is used to measure the difference between the secondary repaired image of the training set and the original image. Usually, a pixel-level difference metric is used. By using weighted reconstruction loss, the model will try to make the generated image as close to the target image as possible at the pixel level, thereby preserving the structure and details in the image.

[0100] Specifically, the second reconstruction loss function

[0101] Reconstruction loss of the second effective area It measures the reconstruction accuracy of the network in the undamaged area

[0102] Reconstruction loss in the second hole region It measures the reconstruction accuracy of the network in the damaged area (hole area).

[0103] The first total score variable loss function Used to measure the difference between adjacent pixels in the secondary repaired images of the training set. Promote the smoothness and continuity of the generated image. By minimizing the first total variation loss, the local refinement network will minimize the noise and discontinuity in the generated image, making the image smoother and more natural.

[0104] Specifically, the first total score becomes the loss function

[0105] First Perceptual Loss Function It is used to measure the semantic similarity between the secondary restoration image and the original image of the training set. It usually uses the features extracted in the middle layer of the local refinement network to compare the difference between the two images. The first perceptual loss helps to maintain the consistency of the generated image with the original image at the semantic level, making the generated image more realistic and credible.

[0106] Specifically, the first perceptual loss function

[0107] First style loss function The first style loss function is used to represent the style characteristics between the secondary restoration image and the original image of the training set. It is a loss function based on deep features. It focuses on maintaining the style characteristics of the image rather than the content. It is measured by extracting the Genge features of the image. By minimizing the first style loss function, the model will try to make the style of the generated image match the original image, thereby achieving tasks such as artistic style transfer or image style conversion.

[0108] Specifically, the first style loss function

[0109] Lagum Matrix

[0110] I gt represents the original image, represents the secondary repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the undamaged area (valid area), and Sum(M) represents the total number of pixels in the damaged area (hole area). express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv , per , sty Both represent the balance factor. Specifically, the parameter is set to λ through experiments. h =6,λ tv =0.1,λ per =0.05,λ sty =120.

[0111] 3. Global Refinement Network

[0112] Figure 5 Schematic diagram of the network structure of the global refined network in the present invention. Figure 5 As shown, the global refinement network in this embodiment is represented as Net G The U-Net-based network includes three attention modules and eight downsampling modules and eight upsampling modules connected in sequence. The three attention modules are interspersed between the fourth upsampling module to the seventh upsampling module. Among them, the eight downsampling modules constitute the encoder, and the eight upsampling modules and three attention modules constitute the decoder.

[0113] Specifically, the first attention module is added to the back of the fourth layer of the decoder, the second attention module is added to the back of the fifth layer of the decoder, and the third attention module is added to the back of the sixth layer of the decoder. In this way, the global refinement network can better understand the global relationship in the image, expand the perception range, and reduce information loss. This ensures that the global refinement network can focus on the reconstruction connection between contextual information and missing areas from different scales and depths, allowing the network to learn and utilize global information more effectively, thereby achieving a refined restoration result for the global image.

[0114] For the images input to the global refinement network, the global structural information and local texture detail information in the image are relatively completely restored. However, for some images with large missing areas, various artifacts still exist. In order to further refine these images, global refinement processing is performed.

[0115] First, an encoder consisting of eight downsampling modules is used. Each downsampling module is mainly composed of convolutional layers and pooling layers. The convolutional layers are used to extract features and convert high-dimensional image information into low-dimensional feature representations. The function is to gradually reduce the size of the feature map while increasing the abstraction level of the features. The high-dimensional features of the image are compressed into a low-dimensional space, which helps capture the global context information of the image.

[0116] Then, the decoder is composed of eight upsampling modules and three attention modules. Each upsampling module is mainly composed of upsampling layers and convolutional layers. Its function is to gradually restore the low-dimensional features compressed by the encoder back to the size of the original image. In the restoration process, the high-dimensional features in the encoder are combined with the low-dimensional features in the decoder through skip connections, so as to restore the image details while retaining the global context information. Self-attention modules were previously added to the decoder. These modules were introduced to enable the model to take into account the global context information when processing each position of the image. The self-attention mechanism improves the model's reconstruction ability when processing missing areas by calculating the correlation between each pixel in the image and all other pixels.

[0117] After being processed by the decoder and the self-attention module, the global structure and local details of the image are finely restored. The final output is converted back to the image space through an upsampling module containing a convolutional layer to obtain the final restored image.

[0118] In the global refinement network Net G In the training, the third loss function is used to measure the difference between the global refinement network output and the true label, and the parameters of the global refinement network are optimized to minimize the third loss function. By the third reconstruction loss function Second total score variable loss function Second Perceptual Loss Function And the second style loss function composition.

[0119] The third reconstruction loss function It is used to measure the difference between the final repaired image of the training set and the original image. Usually, a pixel-level difference metric is used. By using weighted reconstruction loss, the model will try to make the generated image as close to the target image as possible at the pixel level, thereby preserving the structure and details in the image.

[0120] Specifically, the third reconstruction loss function

[0121] Reconstruction loss of the third effective area It measures the reconstruction accuracy of the network in the undamaged area

[0122] Reconstruction loss in the third hole region It measures the reconstruction accuracy of the network in the damaged area (hole area).

[0123] Second total score variable loss function It is used to represent the difference between adjacent pixels in the final repaired image of the training set. It promotes the smoothness and continuity of the generated image. By minimizing the second total variation loss, the local refinement network will minimize the noise and discontinuity in the generated image, making the image smoother and more natural.

[0124] Specifically, the second total score becomes the loss function Represents a synthetic image The pixel value at position (i,j+1).

[0125] Second Perceptual Loss Function It is used to measure the semantic similarity between the final restored image and the original image of the training set. It usually uses the features extracted in the middle layer of the local refinement network to compare the difference between the two images. The second perceptual loss helps to maintain the consistency of the generated image with the original image at the semantic level, making the generated image more realistic and credible.

[0126] Specifically, the second perceptual loss function

[0127] Second style loss function The second style loss function is used to represent the style characteristics between the final restored image and the original image of the training set. It is a loss function based on deep features. It focuses on maintaining the style characteristics of the image rather than the content. It is measured by extracting the Genge features of the image. By minimizing the second style loss function, the model will try to make the style of the generated image match the original image, thereby achieving tasks such as artistic style transfer or image style conversion.

[0128] Specifically, the second style loss function

[0129] Lagum Matrix

[0130] I gt represents the original image, represents the final repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the undamaged area (valid area), and Sum(M) represents the total number of pixels in the damaged area (hole area). express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv , per , sty Both represent the balance factor. Specifically, the parameter is set to λ through experiments. h =6,λ tv =0.1,λ per =0.05,λ sty =120.

[0131] In summary, the total loss function is The pattern repair model is trained according to the total loss function. When the total loss function is minimized, the model training ends.

[0132] Some terms involved in the present invention are explained below.

[0133] 1. Attention Transfer Network (ATN)

[0134] The pyramid encoder mainly fills in the missing area by using attention. The attention is mainly achieved by measuring the regional affinity between the inside and outside pixel blocks (size 3x3) of the missing area. This affinity allows the external relevant features to be transferred from the surrounding context to the internal missing area through weights. The attention transfer network (ATN) first passes through the high-level feature map It learns the affinity of regions from Extract small patches and calculate the cosine similarity between small patches inside and outside the missing area, as shown in Formula 1:

[0135]

[0136] p l i From outside the mask The i-th patch extracted from l j From inside the mask The j-th patch extracted at , and then apply the softmax function to the similarity to obtain the attention score of each patch, as shown in Formula 2:

[0137]

[0138] After obtaining the attention score, the holes in the adjacent lower-level features can be filled with the context weighted by the attention score, as shown in Formula 3:

[0139]

[0140] From φ l-1 The i-th patch extracted outside the mask area, is the jth patch filled in the missing area. After calculating all the patches, we finally pass l Transfer attention to obtain filling feature φ l-1 ,In particular, all of these operations can be formulated as convolution operations that can be trained end-to-end.

[0141] 2.MPD (Mask-aware Pixel Shuffle) downsampling block:

[0142] Figure 6 Schematic diagram of the module structure of the MPD downsampling block in the present invention. Figure 6 As shown in Figure 1, the MPD module is a downsampling method specifically proposed for image restoration. The MPD module consists of a 3×3 convolution, a traditional PD (pixel shuffle downsampling), interlaced scanning (concatenation of feature and mask slices), and a mask separable convolution.

[0143] The overall process is first given the characteristics and mask Initially, a 3×3 convolution operator h(.) is used to project feature X onto X ′ In the example above, the number of channels is halved but the size is the same, and then X ′ And M perform PD operation, as shown in Formula 4:

[0144]

[0145] And in The positions of missing pixels in the drift are also discontinuous between channels, however Each channel sequence is indicated separately The positions of valid and invalid pixels in . Can accurately act on the corresponding channel and slice and Scatter and connect to the channel, and finally get As shown in formulas 5 and 6:

[0146]

[0147] Here, Slice(.) is slicing the channel, ∥ is the channel-level connection, and % represents the modulo operator. Therefore, each feature will have a paired mask as an indicator. Finally, a separable convolutional layer is used to encode the features and masks, so as to learn local prior knowledge from the features indicated by the shuffled mask, and enable the encoder to accurately model the effective information in the visible area.

[0148] 3. RIC-C (Rotation Invariant Coordinate Convolution) Convolution Block

[0149] RIC-C (rotation-invariant coordinate convolution) is a new type of convolution operation, which mainly uses the rotation invariance of the convolution operation to process rotated images. The core idea of ​​RIC-C is to build a convolution mechanism based on a rotation-invariant coordinate system that is independent of the input image content, so that the feature extraction process can remain invariant no matter how the input image rotates around the center. RIC-C implementation process:

[0150] (1) Rotation-invariant coordinate system: RIC-C designs a rotation-invariant coordinate system, in which the corresponding position of each coordinate at different rotation angles is predetermined. This means that given an original coordinate point and its corresponding rotated coordinate point, the offset between the two is fixed and has nothing to do with the content of the input image.

[0151] (2) Matching of convolution kernel and input: When performing convolution, RIC-C no longer uses the traditional sliding window method to translate the fixed-shape convolution kernel on the input feature map. Instead, it uses the above-mentioned rotation-invariant coordinate system to sample and weighted sum each position of the convolution kernel with the pixel at the corresponding rotation coordinate point on the input feature map. In this way, even if the input image is rotated, the result of the convolution operation will remain consistent due to the rotation invariance of the coordinate system and the offset. The calculation formula of RIC-C is shown in Equation 7:

[0152]

[0153] Among them, Φ RIC-C represents the convolution operation using RIC-C, X0 is the center point on the input feature map, F(X) is the input feature map to be processed, is the set of all discrete positions covered by the convolution kernel, W(P) is the weight of the convolution kernel at position P, Q is the rotation coordinate point corresponding to P determined according to the rotation coordinate invariant system, and P+(QP) represents the offset from the original position P to its rotation corresponding position Q.

[0154] Evaluation metrics used:

[0155] PSNR (Peak Signal-to-Noise Ratio): is one of the indicators to measure image quality. PSNR is defined based on MSE (Mean Square Error). PSNR calculates the mean square error (MSE) between the original image and the processed image, and divides it by the square of the maximum possible pixel value (i.e., peak power) of the image, giving the signal-to-noise ratio in decibels (dB). This method assumes that image distortion can be regarded as noise superimposed on the original image, and measures the degree of distortion by comparing the noise power with the signal power. The higher the value of this indicator, the better the effect. Its MSE can be defined as formula 8:

[0156]

[0157] Then PSNR can be defined as Equation 9:

[0158]

[0159] SSIM (Structural Similarity Index): An index used to evaluate the similarity between two images. It is often used to measure the similarity between images before and after distortion, and is also used to measure the authenticity of images generated by the model. The higher the index, the better the effect. The calculation formula is shown in 10:

[0160]

[0161] Here μ x and μ y Expressed as the mean of x and y, σ x and σ y Expressed as the variance of x and y, σ xy represents the covariance between x and y, c1=(k1L) 2 、c2=(k2L) 2 and c3=c 2 / 2, represents three constants, k1 and k2 are 0.01 and 0.03 by default. L represents the range of image pixel values, i.e. 2 B -1.

[0162] LPIPS (Learning Perceptual Patch Similarity): Also known as "perceptual loss", it is used to measure the difference between two images. It uses a pre-trained deep network (such as VGG, AlexNet) to extract image features and then calculates the distance between these features to evaluate the perceptual similarity between images. The lower the index, the better the effect, as shown in calculation formula 11:

[0163]

[0164] Where d is the distance between x0 and x. The feature stack is extracted from layer L and unit-normalized in the channel dimension. Using the vector To scale the number of activated channels, finally calculate the L2 distance. Finally, average in space and sum over channels.

[0165] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A Mongolian pattern repair method, characterized in that: The following steps are involved: S1: constructing a pattern repair model, wherein the pattern repair model includes a global coarse repair network, a local refinement network and a global refinement network connected in sequence; S2: training pattern inpainting model; S3: obtaining the original image of the Mongolian pattern to be restored; S4: inputting the original image into the trained global coarse restoration network for a first restoration, to obtain a preliminary restoration image of the Mongolian pattern to be restored; S5: inputting the preliminary repaired image into the trained local refinement network for a second repair, to obtain a secondary repaired image of the Mongolian pattern to be repaired; S6: inputting the secondary restoration image into the trained global refinement network for a third restoration, to obtain a final restoration image of the Mongolian pattern to be restored; The global coarse repair network includes a pyramid context encoder and a multi-scale decoder connected in sequence; the local refinement network includes a downsampling block, an MPD downsampling block, a RIC-C convolution block, a residual block and an upsampling block connected in sequence; the global refinement network includes three attention modules and eight downsampling modules and eight upsampling modules connected in sequence, and the three attention modules are interspersed between the fourth upsampling module to the seventh upsampling module respectively. The MPD downsampling block is composed of a 3×3 convolution, a pixel shuffle downsampling PD, interlaced scanning and a mask separable convolution, and the interlaced scanning is a series connection of feature and mask slices; the RIC-C convolution block is a rotation invariant coordinate convolution block, which processes the rotated image by using the rotation invariance of the convolution operation.

2. A Mongolian pattern repair method according to claim 1, characterized in that: Step S4 comprises the steps of: S41: Learn the latent feature information of each layer of the original image through the pyramid context encoder, and use the attention transfer network to learn the regional affinity of each layer of the original image; S42: According to the regional affinity of each layer, feature restoration is performed on the missing region of the adjacent low level in the original image to obtain reconstructed feature information; S43: fusing the latent feature information and the reconstructed feature information through a multi-scale decoder to obtain a preliminary restored image.

3. A Mongolian pattern repair method according to claim 1, characterized in that: Step S5 comprises the steps of: S51: enlarging the preliminary repaired image through a downsampling block; S52: using a mask to identify important information in the enlarged preliminary restoration image through the MPD downsampling block, and then retaining the important information through rearrangement; S53: performing rotation invariance processing on the image output from step S52 through a RIC-C convolution block; S54: outputting residual information of the image through residual block learning step S53; S55: The image outputted in step S54 is enlarged by an upsampling block to obtain a secondary repaired image.

4. A Mongolian pattern repair method according to claim 1, characterized in that: Step S3 includes the steps of: Get a training set of Mongolian patterns; The training set of Mongolian patterns is input into the global coarse restoration network to obtain the preliminary restoration image of the training set; The preliminary repaired images of the training set are input into the discriminator, divided into multiple small blocks, and each small block is evaluated whether it is a real image; Input the initial restoration image of the training set into the local refinement network to obtain the secondary restoration image of the training set; Input the secondary repaired image of the training set into the global refinement network to obtain the final repaired image of the training set; The pattern inpainting model is trained according to the total loss function, Among them, the first loss function The second loss function is used to measure the degree of training of the global coarse repair network. Used to measure the degree of training of the local refinement network, the third loss function Used to measure the degree of training of the global refinement network, the fourth loss function Used to measure how well the discriminator is trained.

5. A Mongolian pattern repair method according to claim 4, characterized in that: The first loss function The first reconstruction loss function The adversarial loss function is used to represent the pixel-level difference between the initial repaired image and the original image of the training set. Used to indicate the authenticity of the discriminator output data, where The first reconstruction loss function Adversarial Loss Function I gt represents the original image, represents the initial repair image, I in represents the image input to the network, express and M, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, and λ h and λ g Represents the balance factor.

6. A Mongolian pattern repair method according to claim 4, characterized in that: The second loss function The second reconstruction loss function Used to represent the pixel-level difference between the secondary repaired image and the original image of the training set, the first total score variable loss function The first perceptual loss function is used to represent the difference between adjacent pixels in the secondary restoration image of the training set. The first style loss function is used to represent the semantic similarity between the secondary repaired image and the original image of the training set. It is used to represent the style characteristics between the secondary restoration image and the original image of the training set, where: The second reconstruction loss function The first total score variable loss function First Perceptual Loss Function First style loss function Lagum Matrix k, j represent the coordinates of the synthesized image, i represents the number of layers of the feature map, and I gt represents the original image, represents the secondary repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv , per , sty Both represent balance factors.

7. A Mongolian pattern repair method according to claim 4, characterized in that: The third loss function The third reconstruction loss function The second total score variable loss function is used to represent the pixel-level difference between the final repaired image of the training set and the training set of Mongolian patterns. The second perceptual loss function is used to represent the difference between adjacent pixels in the final repaired image of the training set. The second style loss function is used to represent the semantic similarity between the final restored image of the training set and the training set of Mongolian patterns. It is used to represent the style characteristics between the final restoration image of the training set and the training set of Mongolian patterns, where The third reconstruction loss function The second total score variable loss function Second Perceptual Loss Function Second style loss function Lagum Matrix k, j represent the coordinates of the synthesized image, i represents the number of layers of the feature map, and I gt represents the original image, represents the final repaired image, ⊙ represents the element-by-element product operation, M represents the binary mask, Sum(1-M) represents the total number of pixels in the valid area, Sum(M) represents the total number of pixels in the hole area, express The image to be merged with M, represents the feature map of the i-th layer in the pre-trained VGG-16 network, λ h , tv , per , sty Both represent balance factors.

8. The Mongolian pattern repair method according to claim 4, characterized in that: The fourth loss function Among them, Pdata(I) represents the distribution of the original image, Represents the expected original image I gt The loss of images recognized as real by the discriminator, Indicates the expected generated image I mer ~p is the loss of images identified as fake by the discriminator, I gt represents the original image, express and the merged image of M.

Images