A method for defect-free jointing of sawn timber

Abstract

The present application relates to the technical field of image processing, and more particularly to a texture splicing method for removing defects of solid wood sawing materials, which comprises introducing an asymmetric encoder in MAE into a Vit model to reduce the calculation amount of the model; constructing an RSwin decoder with multi-scale characteristics to adjust the size of the divided image blocks and complete the coarse-to-fine repair of the image blocks; and using a distance-defect-center-weighted L2 loss function to give different weights to the unmasked areas and make full use of the effective pixels to repair the missing areas. The present application solves the problem that the existing Transformer model uses a global attention mechanism, which makes the calculation amount too large; when the method of dividing an image into multiple non-overlapping image blocks is used for image repair tasks, pixel-level modeling cannot be performed, especially when the Vit model is used as a decoder, the repaired result will have a clear sense of fragmentation, and the marginal effect is obvious.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a texture splicing method for removing defects of solid wood sawn timber. BACKGROUND

[0002] During the growth and processing of wood, various kinds of defects such as knots, dead knots, insect holes and saw damage will be generated, which not only affect the physical properties such as mechanical properties of the wood product, but also affect the appearance of the product and reduce the grade of the product. The defects in wood are an important standard for evaluating the quality and commercial value of wood. With the rapid development of sensor and computer technology, related non-destructive testing technologies such as laser technology, infrared technology and machine vision technology have been gradually applied to the field of wood detection. Existing researches mostly focus on the detection and positioning of defects, but little consideration is given to the visual consistency of wood texture after splicing of solid wood veneer after positioning of defects. Therefore, image inpainting technology is introduced to generate the texture of the defect part of the wood, and a matching algorithm is used to find similar wood for splicing in the later stage, thereby improving the use value of the wood.

[0003] The difficulty of image inpainting lies in generating detailed and realistic texture in the missing part while maintaining the consistency of the image texture structure. At present, the main ideas of image generation tasks include variational encoder (VAE), denoising diffusion model (DDPM) and generative adversarial network (GAN). In recent years, the method based on the combination of GAN+CNN to generate the texture of the missing part of the image is the main research direction of image inpainting. However, the existing image inpainting network still has problems such as obvious marginal effect, low repair accuracy caused by discontinuous visual texture, long model training time and the like, and the irregularity and complexity of wood texture deepen the difficulty of repair, making it difficult to achieve ideal results using existing models.

[0004] In recent years, the Transformer network has achieved great success in the visual field due to its strong feature extraction capability, solving the problem of limited receptive field of the CNN model. However, there are still some deficiencies in the application of the Transformer model to the visual field. On the one hand, the use of global attention mechanism by the Transformer model makes the calculation amount too large; on the other hand, the method of dividing the image into multiple non-overlapping image blocks used by the Vision Transformer (Vit) cannot be used for pixel-level modeling, especially when the Vit model is used as a decoder, the repaired result will have a clear sense of fragmentation, and the marginal effect is obvious.

[0005] Different from other image inpainting tasks, wood defect texture inpainting needs to consider the following points: wood texture presents irregularity, the texture around the defect area will be deformed due to the existence of defects, in order to ensure the continuity of the repaired texture, the mask area of the defect part will be larger than other inpainting tasks; in addition, repairing the texture of the wood defect area is to prepare for the later wood splicing, and the wood splicing is carried out in a regular shape. SUMMARY

[0006] In view of the shortcomings of the prior art, the technical scheme adopted by the present application is as follows: a texture splicing method for removing defects of solid wood sawn timber comprises the following steps:

[0007] Step 1, collect wood texture images, and perform rotation, translation, mirroring and brightness transformation preprocessing on the images to construct a wood texture image training set and a verification set;

[0008] Step 2, input the training set data into the MRS-Transformer model for training.

[0009] Further, the construction of the MRS-Transformer model comprises:

[0010] Step 21, introducing an asymmetric encoder in the MAE into the Vit model;

[0011] Further, the asymmetric encoder adopts a fixed mask strategy, discards the mask part, and takes the unmasked visible block as input.

[0012] Further, the asymmetric encoder comprises a plurality of Transformer modules, each Transformer module being composed of a multi-head self-attention mechanism layer and a full connection layer, and the multi-head self-attention mechanism layer and the full connection layer being connected through a Norm layer and a residual connection.

[0013] Step 22, constructing an RSwin decoder with multi-scale characteristics to adjust the size of the divided image block and complete the repair of the image block from coarse to fine;

[0014] Further, the RSwin decoder adopts W-MSA and SW-MSA sliding window attention of the Swin Transformer model, and connects a Patch Diverging layer after the SW-MSA.

[0015] Step 23, by using a distance defect center far and near weighted L2 loss function, different weights are given to the unmasked area, and the effective pixels are fully utilized to repair the missing area.

[0016] Further, step 23 specifically comprises:

[0017] First, set a matrix with the same size as the original image, set the pixels in the masked area to 0, calculate the Euclidean distance between the pixel and its nearest mask area in the unmasked area, and obtain a matrix template;

[0018] Then take the matrix template as the exponent with alpha as the base, and obtain the weight matrix.

[0019] Further, the formula of the L2 loss function is:

[0020]

[0021] Wherein, L2 loss between the repaired image and the original image, The weight matrix is divided by , which represents the average loss, I r is the repaired image, I o is the original image.

[0022] The beneficial effects of the present application are:

[0023] 1. An asymmetric encoder and decoder structure is designed, that is, the encoder only takes the unmasked visible block as input, and the input to the decoder is the output of the encoder and the mask block. The dimensions of the encoder and decoder inputs are different, forming an asymmetric structure. Since the mask block can only provide position information, the texture information of the wood image is not lost while the computational load of the model encoder is reduced.

[0024] 2. A decoder with a multi-scale RSwin module is designed, which can flexibly adjust the size of the divided image block and restore the texture of the wood image from coarse to fine in the decoding stage. The characteristics of Swin mobile window attention can better benefit the repair of defect parts by the texture features of the surrounding wood defects, and solve the fragmentation between image blocks.

[0025] 3. The L2 loss weighted by the distance from the defect center is proposed, which gives different weights to the unmasked area. The closer the position to the defect center, the greater the weight, which realizes the full use of effective texture features of the wood defect edge and solves the problem of semantic incoherence at the connection between the repaired area and the unmasked area, and improves the repair accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a schematic diagram of a Vision Transformer model repairing wood texture;

[0027] Figure 2 is a schematic diagram of the steps of repairing the texture of a defective image;

[0028] Figure 3 is an asymmetric encoder-decoder architecture diagram;

[0029] Figure 4 is a decoder RSwin architecture diagram and W-MSA and SW-MSA layer principle;

[0030] Figure 5 is a Patch Diverging layer principle;

[0031] Figure 6 is MRS-Transformer architecture diagram;

[0032] Figure 7 (a)-(f) are respectively original image, right rotation 90°, left rotation 90°, brightness transformation, mirror image and contrast transformation diagram;

[0033] Figure 8 is a part of textured defect sample;

[0034] Figure 9 are respectively original image of solid wood sawn timber, defect image after cutting, mask strategy processing diagram, repaired image, image mapping to original position of image;

[0035] Figure 10 are respectively original image, mask strategy processing diagram, A, B, C, D group ablation experiment diagram;

[0036] Figure 11 are respectively original image, mask strategy processing diagram, DeepFill v2, TG-Net and MRS contrast diagram. DETAILED DESCRIPTION

[0037] The present application will be further described below in conjunction with the drawings and examples, which are simplified schematic diagrams and only schematically show the basic structure of the present application, and thus only show the configurations related to the present application.

[0038] A texture splicing method for removing defects of solid wood sawn timber includes the following steps:

[0039] Vit is a model that applies the Transformer to image classification. Due to its advantages such as simplicity, good effect and strong scalability, it has attracted the attention of many researchers in the field of vision to the Transformer model. In NLP, an input sequence is input into the Transformer, while in the field of vision, an image is two-dimensional, so before input, a two-dimensional picture needs to be converted into a one-dimensional sequence. The method adopted by VIT is to divide the image into multiple patches of the same size and non-overlapping blocks, then project each patch into a fixed-length vector, and then add position encoding to each vector to mark the position information between the picture blocks, and then input it into the Transformer for training; the wood texture structure diagram repaired by using the VIT model is as shown in Figure 1

[0040] The MRS-Transformer model of the application is a basic framework for using VIT as an image repair model, and is optimized in three aspects of mask strategy (MASK), encoder input form (Encode) and decoder structure design (Decode).

[0041] Step one, collect wood texture images, and perform rotation, translation, mirroring and brightness transformation preprocessing on the images to construct a wood texture image training set and a verification set;

[0042] Step two, input the training set data into the MRS-Transformer model for training.

[0043] Step 21, introduce an asymmetric encoder and decoder based on the Vit model, wherein the encoder adopts a fixed mask (MASK) strategy, discards the mask part and only takes the unmasked visible block as input to reduce the model calculation amount;

[0044] ​MASK: Image inpainting tasks usually need to mask the image to simulate the defect area to train the network model, and ensure that the model can reconstruct the complete image by learning part of the known information in the image. In the industrial scene of solid wood board splicing, the defective area of the board is removed in a regular rectangle. Therefore, the MRS-Transformer model adopts a rectangular mask method during training. In addition, due to the deformation of the wood texture due to the presence of defects, from the perspective of visual texture consistency, the defect area and the surrounding deformed texture need to be masked, so that the area of the mask is much larger than that of other image inpainting algorithms. Therefore, an asymmetric encoding and decoding model structure in MAE is introduced when designing the algorithm, that is, the encoder only takes the visible block as input, and the input to the decoder is the output of the encoder and the mask block. The dimensions of the encoder and the decoder input form an asymmetric structure, so that the larger the mask area is, the fewer the blocks input to the encoder are, and the less the amount of calculation required by the encoder is. However, the less the remaining unmasked area is, the more difficult it is to reconstruct the texture. After several experiments, it is found that when the mask rate is 0.5, that is, the area of the mask is about 50% of the picture, the mask area can be maximized while ensuring the repair accuracy, and the calculation amount is minimized.

[0045] Since the size and position of the wood texture defect are different, in order to improve the accuracy of the model in repairing actual wood texture defects, a strategy of training the model with a fixed mask size is proposed. The strategy determines the mask range by positioning the defect position, crops the appropriate wood texture image, expands or scales the defect image to a fixed size, and then inputs it into the model to get the repaired texture image. Then, through the reverse operation, it is mapped to the original position of the picture, as shown in Figure 2

[0046] The encoder is composed of multiple Transformer modules, each of which is composed of a multi-head self-attention mechanism layer (Attention) and a fully connected layer (MLP). They are connected through the Norm layer and the residual connection to pass between them. The multi-head self-attention mechanism is the core of the Transformer, which realizes the global modeling of the image through this mechanism. The formula of the attention mechanism is:

[0047]

[0048] where Q, K, and V are query, key, and value matrices, is the scaling factor.

[0049] ​Although this mechanism achieves global relationship modeling by calculating the similarity between all image block vectors, it also consumes a large amount of computation. Since the image blocks of the masked wood image cannot provide information for image inpainting, inputting them into the network model will only increase the computation and occupy additional memory space. The present application designs an asymmetric encoding and decoding structure, as shown in Figure 3 , wherein the encoder only inputs the unmasked blocks as input, extracts features through the encoder, and then inputs the masked blocks into the decoder to output the inpainted image.

[0050] Step 22, a RSwin (RS) module decoder with multi-scale characteristics is designed to adjust the size of the divided image blocks, complete the inpainting of the image blocks from coarse to fine;

[0051] Decoder: Since Vit takes image blocks as input and output of the model, when used in the decoder process of the model, the size of the image block is too large to affect the inpainting accuracy; the size of the image block is too small to increase the computation and prolong the inpainting time, and it cannot adaptively adjust the size of the divided image blocks. In view of the shortcomings in the design of the Vit model, a new decoder module RSwin-Transformer for wood texture image inpainting is designed, as shown in Figure 4 .

[0052] This module retains the sliding window attention mechanism (W-MSA and SW-MSA) of the Swin Transformer model, which not only guarantees the global modeling characteristics but also reduces the computational complexity of the model. The computational complexity of the ordinary attention mechanism (MSA) and the window size M window attention mechanism (W-MSA) is respectively:

[0053]

[0054] Among them, h w represents the number of patch blocks on the image, M represents the size of the window attention, C represents the number of channels; from the above formula, it can be seen that the window attention mechanism can make the model computation complexity increase linearly with the image size.

[0055] The RSwin module improves the Swin multi-scale design method, and designs a Patch Diverging layer in the decoder; the principle is as shown in Figure 5 , which can double the output size, reduce the dimension of the channel, achieve the size of the image, and reduce the size of the basic block in the window attention, effectively alleviate the direct fragmentation of the inpainted image block, and improve the inpainting accuracy.

[0056] The MRS-Transformer model structure of the present application is as shown in the figure Figure 6 As shown in the figure, on the image of the defective solid wood board, the image of appropriate size is cut out as the center of the defect according to the mask strategy, and the standard input model is made into the size for training, then the image is divided into non-overlapping blocks and the defect block in the center is masked, then the block not masked is mapped to a feature vector through a linear mapping layer and is input into the encoder with position coding, the encoder is composed of 12 layers of Vit modules, the output after the feature extraction of the encoder is input into the decoder together with the vector mapped by the block at the beginning of the mask, the decoder is composed of 8 layers of RSwin modules, and the output after the decoder is the repaired wood texture image, and the repaired image is mapped back to the corresponding position of the original image, which is the final result.

[0057] Step 23, the unmasked area is given different weights by a distance-weighted L2 loss function from the center of the defect, so that the model can make full use of the effective pixels to repair the missing area.

[0058] Weighted L2 loss: traditional image repair calculates the loss of only the defect part or the whole image, which may cause the semantic inconsistency of the repaired image at the connection between the defect and the non-defect, and the effective pixels closer to the defect position can help more in the repair, and vice versa.

[0059] The MRS-Transformers model designs a distance-weighted L2 loss from the center of the missing part; first, a matrix with the same size as the original image is set, the pixels in the masked area are set to 0, the Euclidean distance between the pixel and the nearest masked area is calculated for the pixel not masked, and a matrix template is obtained; take alpha as the base ( 0.999), the matrix template is exponential, and a weight matrix is obtained, the value of the weight matrix ranges from 0 to 1, the weight matrix is used as the weight of the pixel point when calculating the L2 loss, the weight of the defect part of the matrix is 1, the weight of the pixel closer to the defect is close to 1, and the weight of the pixel farther from the defect is smaller.

[0060] The formula is as follows:

[0061]

[0062] Wherein, L2 loss between the repaired image and the original image, W represents the weight matrix, divided by , represents the average of the loss; I r is the repaired image, I o is the original image.

[0063] Texture generation evaluation metrics:

[0064] Four image quality evaluation indicators, MSE (mean square error), PSNR (peak signal to noise ratio), SSIM (structural similarity), and LPIPS (learned perceptual image patch similarity), and the model calculation amount GFLOPs are used as quantitative observation data. Figure X The mean of the squared differences between the generated image Y

[0065]

[0066] in, Indicates the height of the image. Indicates the width of the image. The MSE value range is [0,1]. The smaller the value, the smaller the image distortion.

[0067] PSNR (Peak Signal to Noise Ratio) is a full-reference image quality evaluation indicator.

[0068]

[0069] in, is the color depth of each pixel in the image, here it is 8, The unit is The larger the value, the smaller the image distortion.

[0070] SSIM (structural similarity) is also a full-reference image quality evaluation metric that measures image similarity from three aspects: luminance, contrast, and structure.

[0071]

[0072] in, Indicates brightness, Indicates contrast, Represents the structure, the SSIM value range is The larger the value, the smaller the image distortion.

[0073] LPIPS (Learned Perceptual Image Patch Similarity) is a reference image quality evaluation index that learns image patch perceptual loss. It is more consistent with human perception than methods such as MAE, PSNR, and SSIM. The lower the LPIPS value, the more similar the two images are, and vice versa, the greater the difference.

[0074] Given a real image reference patch and a noisy image distorted patch , the perceptual similarity measure is as follows:

[0075]

[0076] where, is the distance between and . Feature stacks are extracted from the feature layer and unit-normalized in the channel dimension. The vector is used to scale the number of activation channels, and the L2 distance is finally calculated. Finally, it is averaged in space and summed in channels. When , it is equivalent to the cosine similarity.

[0077] Data collection:

[0078] The experiment used Oscar F810 CIRF industrial cameras to collect 3000 wood texture images, of which 500 had defects. Among them, 2500 defect-free images were expanded to 10000 through data augmentation methods such as rotation, translation, mirroring, and brightness transformation, and were divided into training and validation sets in a ratio of 8:2. To improve model training speed and recognition speed, high-resolution images were uniformly processed to 256 pixels. Some samples are shown in Figure 7 (a)-(f).

[0079] The size and defect location of the defective wood texture images collected in the data set are different. The defect location is positioned in the center of the intercepted sample, and is uniformly processed to 256 pixels. Some processed samples are shown in Figure 8 .

[0080] Experimental environment and key parameters:

[0081] The experimental environment is as follows: the system is Ubuntu 20.4; the deep learning framework is Pytorch; the GPU is Tesla V100; the running memory is 32G. The main parameters of the model: BatchSize is 32, the training process uses Adam optimizer, the learning rate is 1.8e-4, and the model converges to the best state after training for 1200 epochs.

[0082] The image size used in the experiment is uniformly 256 256pixel; in the encoding stage, the image is divided into non-overlapping image blocks, and the image block size (PatchSize) is 16 16pixel, a total of N (256) image blocks. The dimension (Encode_Embed_dim) mapped by each image block is 768, representing the sum of the number of pixel points in each image block in three channels. The encoder uses the Vit model, and the depth (Depth) is 12 layers. The decoder uses 2 groups of RSwin models, each group has a depth (Depth) of 4 layers, and after passing through a group of RSwin, a layer of Diverging layer is used to reduce the size of the divided image block by 4 times, the size of the image is expanded by 4 times, and the number of channels is reduced by 16 times.

[0083] Defect texture generation experiment of solid wood board

[0084] In order to test the repair effect of the model on real wood samples with defects, an experiment of defect area texture generation was conducted. First, the defect position was determined on the original image of solid wood sawn timber (origian), and then the cutting range was determined according to the size of the defect and the deformation of the texture. The cropped defect image was processed to the corresponding size (input) and input into the MRS-Transformers model (mask is a mask), and the repaired image (output) was obtained. Then the image was mapped to the original position of the image (inpaint). As shown in Figure 9 , the wood texture generated by the MRS-Transformer model is natural and coherent, which can meet the visual texture consistency requirement of solid wood veneer splicing. In the later stage, the texture matching algorithm can be used to find suitable solid wood boards with similar texture from the database for splicing.

[0085] Ablation experiment

[0086] In order to prove the effectiveness of the MRS-Transformers model, an ablation experiment was designed.

[0087] Group A experiment is to use Vit model as encoder and decoder to train the data set, using L2 loss as loss function, encoder 12 layers, decoder 8 layers.

[0088] The experiments in group B introduce an asymmetric encoder-decoder structure based on group A. The encoder only inputs unmasked image blocks, uses L2 loss as the loss function, and the decoder structure remains unchanged.

[0089] Group C replaced the decoder with the proposed RSwin structure based on Group B, and also used L2 loss as the loss function, with an encoder of 12 layers and a decoder of 8 layers.

[0090] Group D is the model of the present invention. Based on Group C, weighted L2 loss is used as the loss function, and the codec is the same as that of Group C.

[0091] The experimental results are shown in Table 1. The models were evaluated from multiple perspectives, including four image evaluation metrics and model computational complexity. Group B, after improving the encoder input method, showed no significant changes in the MSE, PSNR, and SSIM metrics. Although the LPIPS metric was slightly lower than that of Group A, the computational complexity of the model was reduced by 28.3%. Group C, after redesigning the RSwin module as the decoder, showed no significant changes in the other three metrics, but a 21% decrease in LPIPS, indicating that the generated image details were improved from a human visual perspective, and the computational complexity was further reduced by 26.3%. Group D, building on Group C's approach, used a weighted L2 loss to improve the accuracy of restoration of the edges of missing wood areas. MSE and LPIPS were reduced by 51.7% and 34.2%, respectively, compared to Group C. PSNR and SSIM were higher by 12.2% and 7.5%, respectively, compared to Group C.

[0092] Table 1 Ablation experiment

[0093]

[0094] In addition to objective indicators, visual images of each improvement stage are selected as subjective references to make the experimental results more intuitive; Figure 10 As shown, by comparing the restored images of the three experiments, it can be clearly observed that Figure 10 The restoration result is closest to the original image in terms of texture structure and semantic information. Figure 10 Compared with the asymmetric encoding and decoding structure, the blockiness of the repaired image is significantly reduced after the RSwin structure is introduced, and the repaired texture is more coherent and detailed. After the weighted L2 loss is introduced, the blockiness basically disappears, the edge connections become semantically coherent, and the texture connections become more natural.

[0095] Comparative experiment

[0096] In order to verify the superiority of the model, the present invention uses two image restoration models, Deepfill v2 and TG-Net, to perform a comparison on a solid wood veneer dataset for fixed-size mask image restoration. Among them, the Deepfill v2 model and the TG-Net model are both two-stage restoration methods based on the GAN model. They first restore the outline of the missing area in the coarse generation network, and then input the end of the coarse generation into the fine generation network to restore a more refined result. Among them, the Deepfill v2 model proposes a gated convolution operation. The gated convolution solves the problem that ordinary convolution regards all pixels as valid pixels. It generalizes some convolutions by providing a learnable dynamic feature selection mechanism for each channel at each spatial position in all layers, making the restoration effect more outstanding and achieving relatively leading results in many image restoration tasks. The TG-Net model also studies the removal of texture defects in solid wood sawn timber, and proposes the idea of ​​normalizing the foreground and background separately to improve the texture generation ability of missing areas. The experimental results are shown in Table 2. Our proposed method achieved scores of 0.0003, 40.1233, 0.154, and 0.9173 on the four metrics of MSE, PSNR, IPIPS, and SSIM. Compared to the Deepfillv2 and TG-Net models, the MSE decreased by 47.0% and 66.9%, the LPIPS decreased by 60.6% and 42.5%, the PSNR increased by 16.1% and 26.2%, and the SSIM increased by 7.3% and 5.8%, respectively. In addition to the objective evaluation metrics, our model only takes 0.05 seconds to restore a single wood texture image, nearly five times faster than the other two algorithms.

[0097] Table 2 Comparative experiment

[0098]

[0099] The present invention also makes a visual comparison of the results of the above model repair, such as Figure 11 In order to ensure the authenticity of the experimental results, all repaired images have not been processed in any way. By comparison, it can be seen that the module of the present invention can better generate the texture of the missing part of the solid wood board, and has achieved very good results in terms of the semantic consistency of the edge of the missing area. The other two models cannot remove the artifacts at the edge and maintain the semantic consistency of the connection, and the generated wood texture is not coherent enough.

[0100] from Figure 11The experimental results show that the MRS-Transformer model is superior to the DeepFill v2 and TG-Net models in objective indexes and visual effects, because the mask area of the solid wood sawing material texture repair task is large, which causes certain difficulty for image generation; the present application is optimized separately for this problem, so the repair result is better than that of the other two models. In addition, although the gating convolution operation proposed by the Deepfill v2 model provides the network with the ability of dynamic feature selection and optimizes the effect of part of the convolution, the learning ability of effective features and the modeling of long-distance scenes are still worse than those of the Transformer model, so the final repair effect of the model is also worse than that of the MRS-Transformer model. The TG-Net model is also a model for researching wood texture repair, and proposes a front and back background normalization method to increase the weight of the mask area features in the model, so that the missing part of the texture features can be better generated. However, this makes the generated texture and background area not coherent enough, and color difference phenomenon is easy to occur. The present application model weights the L2 loss by the distance from the missing center, and the weight proportion decreases from the mask area to the background area, so that the transition of the weight has a gradual change process, so that the generated texture features are more coherent.

[0101] Based on the above ideal embodiments according to the present application, through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the contents of the specification, and must be determined according to the scope of the claims.

Claims

1. A method for defect-free jointing of sawn timber, characterized in that, The method comprises the following steps: Step one, collect wood texture images, and pre-process the images to construct a wood texture image training set and a verification set; Step two, input the training set data into the MRS-Transformer model for training; The construction of the MRS-Transformer model comprises: Step 21, introduce an asymmetric encoder in the MAE into the Vit model to reduce the calculation amount of the model; The asymmetric encoder comprises a plurality of Transformer modules, each of which is composed of a multi-head self-attention mechanism layer and a full connection layer, and the multi-head self-attention mechanism layer and the full connection layer are connected through a Norm layer and a residual connection; Step 22, construct a RSwin decoder with a multi-scale feature to adjust the size of the divided image block and complete the repair of the image block from coarse to fine; The RSwin decoder adopts the W-MSA and SW-MSA sliding window attention mechanism of the Swin Transformer model, and connects a Patch Diverging layer after the SW-MSA; Patch Diverging layer changes the feature map size N C W H to N (C / r2) (H r) (W r); where C represents the number of channels; Step 23, set different weights for the unmasked area by using the L2 loss function weighted by the distance from the defect center, and repair the missing area by using the effective pixels.

2. The method of claim 1, wherein the method further comprises: The asymmetric encoder adopts a fixed mask strategy, discards the mask part, and takes the unmasked visible block as input.

3. The method of claim 1, wherein the method further comprises: Step 23 specifically comprises: First, set a matrix with the same size as the original image, set the pixels of the masked area to 0, calculate the Euclidean distance between the pixels and the nearest mask area at the unmasked position, and obtain a matrix template; Then take the matrix template as the exponent and alpha as the base to obtain a weight matrix, and take the weight matrix as the weight of the pixel point when calculating the L2 loss.

4. The method of claim 3, wherein the method further comprises: The formula of the L2 loss function is: wherein, denotes the L2 loss of the repaired image and the original image, denotes the weight matrix, I r is the repaired image, I o is the original image.

Images