Image restoration method based on controllable receptive field

Abstract

The invention discloses an image restoration method based on a controllable receptive field, and relates to the technical field of image restoration. The method comprises the following steps: refining the edge of a to-be-restored image through expansion and erosion operations to obtain a binary mask edge enhanced image, and splicing the to-be-restored image and the binary mask edge enhanced image; constructing an image restoration model through a multilayer receptive field module; in the controllable receptive field module, the input features are split according to scale factors, the features of different receptive fields are extracted through local branches and global branches and fused, the features are extracted layer by layer through an image restoration model, learning and mapping from an image to be restored to a reconstructed image are achieved, and a restored image is obtained; a mixed loss function including controllable receptive field adversarial loss is adopted in the image restoration model training process. According to the method, the edge details of the mask region are captured and enhanced, and the balance of local details and a global structure is considered through the controllable receptive field, so that the quality of the repaired image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image inpainting, and in particular relates to an image inpainting method based on a controllable receptive field. BACKGROUND

[0002] Image inpainting aims to reconstruct damaged or missing regions in an image, making the repaired image visually natural and coherent. As a key technology in the field of computer vision, it has been widely applied in various scenarios such as image editing, object removal, historical photo restoration, medical image processing, etc.

[0003] In recent years, with the rapid development of deep learning, convolutional neural networks and generative adversarial networks, Transformer and other technologies have achieved good results in the field of image inpainting. The local receptive field features of convolutional neural networks and the global receptive field provided by the attention mechanism have solved many challenges in the current image inpainting field. However, the shortcomings of these existing methods are that they usually use a single type of receptive field in the image inpainting task process. In addition, when merging local or global features, traditional models can only use a serial method, thereby preventing the use of multiple receptive fields at the same layer. This design severely limits the model's ability to balance local detail extraction and global structure understanding, thereby reducing its effectiveness in scenarios involving high-texture details or requiring comprehensive structure understanding, resulting in poor image inpainting results. SUMMARY

[0004] Therefore, it is necessary to provide an image inpainting method based on a controllable receptive field, which captures and enhances the edge details of the mask region and simultaneously considers the balance of local details and global structures through a controllable receptive field method, thereby improving the quality of the repaired image.

[0005] The present application adopts the following technical solutions: The present application provides an image inpainting method based on a controllable receptive field, comprising: The edges of the image to be repaired are refined through dilation and erosion operations to obtain a binary mask edge-enhanced image, and the image to be repaired and the binary mask edge-enhanced image are spliced to obtain a spliced image; An image inpainting model is obtained; the image inpainting model is constructed based on a multi-layer controllable receptive field module and a semi-jump connection; The spliced image is input into the image inpainting model, in each controllable receptive field module of the image inpainting model, input features of the controllable receptive field module are split along a feature channel dimension according to a controllable scale factor to obtain first split features and second split features; convolution operations are respectively performed on the first split features and the second split features to obtain first convolution features and second convolution features, and feature extraction in a channel dimension is respectively performed on the first split features and the second split features by a cross-covariance attention mechanism to obtain first attention features and second attention features; the first convolution features and the second convolution features are added to obtain local features; the first attention features and the second attention features are added to obtain global features; the local features and the global features are weighted along the feature channel dimension to obtain fused features. The fused features output by the last controllable receptive field module are spliced with the spliced image to obtain an inpainting image corresponding to the image to be inpainted.

[0006] Optionally, edges of the image to be inpainted are refined by dilation and erosion operations to obtain a binary mask edge enhanced image, including: The binary mask image of the image to be inpainted is subjected to a dilation operation, and edges of the binary mask image after the dilation operation are refined by an erosion operation to obtain a binary mask edge enhanced image.

[0007] Optionally, the image inpainting model includes an encoder and a decoder connected in series; the encoder includes multiple controllable receptive field modules containing down-sampling layers; the decoder includes multiple controllable receptive field modules containing up-sampling layers. The controllable receptive field module includes a feature splitting unit, a local branch, a global branch and a feature fusion unit, the local branch includes two parallel first convolution units and second convolution units in sequence, and a first addition unit; the global branch includes two parallel first attention units and second attention units in sequence, and a second addition unit; output ends of the feature splitting unit are connected to input ends of the local branch and the global branch, an input of the local branch is an input of the two convolution units, an input of the global branch is an input of the two attention units, output ends of the first convolution unit and the second convolution unit are connected to an input end of the first addition unit, output ends of the first attention unit and the second attention unit are connected to an input end of the second addition unit, an output end of the first addition unit and an output end of the second addition unit are connected to an input end of the feature fusion unit, and an output end of the feature fusion unit is an output end of the controllable receptive field module.

[0008] Optionally, the output channel number of the plurality of controllable receptive field modules connected in sequence in the encoder is sequentially increased, and the output size is sequentially reduced; the output channel number of the plurality of controllable receptive field modules connected in sequence in the decoder is sequentially reduced, and the output size is sequentially increased; the implementation process of the controllable receptive field module specifically includes: The input feature is split along the feature channel dimension by the controllable scaling factor in the feature splitting unit to obtain first split features and second split features; The first split features are subjected to convolution operation by the first convolution unit to obtain first convolution features, the second split features are subjected to convolution operation by the second convolution unit to obtain second convolution features, the first split features are subjected to feature extraction in the channel dimension by the cross-covariance attention mechanism in the first attention unit to obtain first attention features, and the second split features are subjected to feature extraction in the channel dimension by the cross-covariance attention mechanism in the second attention unit to obtain second attention features; The first convolution features and the second convolution features are added by the first addition unit to obtain local features; the first attention features and the second attention features are added by the second addition unit to obtain global features; The local features and the global features are weighted along the feature channel dimension by the feature fusion unit to obtain fused features.

[0009] Optionally, the first attention unit and the second attention unit are both attention units constructed by the cross-covariance attention mechanism, and the attention unit includes a first cross-covariance attention mechanism unit, a third addition unit, a second cross-covariance attention mechanism unit and a fourth addition unit connected in sequence, and the implementation process of the attention unit includes: The input feature of the attention unit is input into the first cross-covariance attention mechanism unit to obtain first features; The first features and the input feature of the attention unit are input into the third addition unit to obtain second features; The second features are input into the second cross-covariance attention mechanism unit to obtain third features; The third features and the second features are input into the fourth addition unit to obtain attention features, i.e., the output features of the attention unit.

[0010] Optionally, the cross-covariance attention mechanism is: ; wherein, is the attention feature obtained by the cross-covariance attention mechanism, represents a normalization function, , and respectively are value vector, query vector and key vector, is and vector dimension of is transpose.

[0011] Optionally, in the image inpainting model training process, the objective function adopted by the image inpainting model is a multi-loss function mixed objective function; the construction process of the multi-loss function mixed objective function specifically includes: According to the predicted inpainting image and the real sample image, a pixel-level reconstruction loss is determined; According to the discriminative output of the controllable receptive field condition discriminator when the real sample image and the to-be-inpainted sample image are input as conditions, and the discriminative output of the controllable receptive field condition discriminator when the model output image and the to-be-inpainted sample image are input as conditions, a controllable receptive field adversarial loss is determined; the controllable receptive field condition discriminator is constructed by using a controllable receptive field module; Features of the predicted inpainting image and the real sample image at different levels are extracted respectively, and a feature matching loss is determined according to the feature difference between the predicted inpainting image and the real sample image; A plurality of pre-training high receptive field network features of the predicted inpainting image and the real sample image are extracted respectively, and a high receptive field loss is obtained by capturing the difference between the predicted inpainting image and the real sample image in the global structure according to the pre-training high receptive field network features; Based on the total variation theory, a total variation loss of the predicted inpainting image is determined; The structural similarity between the predicted inpainting image and the real sample image is calculated to determine a structural similarity loss; According to the pixel-level reconstruction loss, the controllable receptive field adversarial loss, the feature matching loss, the high receptive field loss, the total variation loss and the structural similarity loss, an objective function is determined.

[0012] The application provides an image inpainting device based on a controllable receptive field, comprising: The splicing module is used for refining the edge of the to-be-inpainted image through inflation and erosion operations to obtain a binary mask edge enhanced image, and splicing the to-be-inpainted image and the binary mask edge enhanced image to obtain a splicing image; The image reconstruction module is configured to obtain an image inpainting model, the image inpainting model is constructed based on a multi-layer controllable receptive field module and a half-jump connection, the spliced image is input into the image inpainting model, in each layer of the controllable receptive field module of the image inpainting model, input features of the controllable receptive field module are split along a feature channel dimension according to a controllable scale factor to obtain first split features and second split features, convolution operations are respectively performed on the first split features and the second split features to obtain first convolution features and second convolution features, and cross-covariance attention mechanisms are respectively used to extract features in a channel dimension of the first split features and the second split features to obtain first attention features and second attention features, the first convolution features and the second convolution features are added to obtain local features, the first attention features and the second attention features are added to obtain global features, the local features and the global features are weighted along the feature channel dimension to obtain fused features, and the fused features output by the last layer of the controllable receptive field module are spliced with the spliced image to obtain a to-be-inpainted image corresponding to an inpainted image.

[0013] The application provides a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the image inpainting method based on the controllable receptive field.

[0014] The application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the image inpainting method based on the controllable receptive field when executing the program.

[0015] The above at least one technical scheme adopted by the application can achieve the following beneficial effects: In the application, first, the edge of the to-be-inpainted image is enhanced through inflation and erosion operations, and the enhanced edge features are clearer and more continuous than the original edge, especially in a complex background or a structure-intensive area, the edge information can be more accurately captured, and more accurate features can be provided for subsequent image inpainting, then the image is reconstructed through the image inpainting model constructed based on the multi-layer controllable receptive field module and the half-jump connection, in each layer of the controllable receptive field module of the image inpainting model, the spliced image is split according to a channel dimension through a controllable scale factor to determine first split features and second split features, local features are learned through convolution operations, that is, local receptive fields, global features are learned through cross-covariance attention mechanisms, that is, global receptive fields, the limitation of a single receptive field is broken, then the local features and the global features are weighted and fused along the channel dimension, the contributions of the local features and the global features are dynamically balanced, bidirectional information complementation is realized, and the “serial merging” in the prior art is replaced, so that the effectiveness of the method in scenes involving high-texture details or requiring comprehensive structure understanding is improved, and the image inpainting effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 A controllable receptive field-based image inpainting method provided by the present application is shown in the flowchart; Figure 2 A structure diagram of an image inpainting model provided by the present application is shown in the flowchart; Figure 3 A structure diagram of a controllable receptive field module provided by the present application is shown in the flowchart; Figure 4 A structure diagram of an attention unit provided by the present application is shown in the flowchart; Figure 5 Another controllable receptive field-based image inpainting method provided by the present application is shown in the flowchart; Figure 6 The image inpainting effects of the image inpainting model provided by the present application in four complex scenes are shown in the figures, wherein, (a) is a real image of a building scene, (b) is an image to be inpainted of a building scene, (c) is an inpainted image of a building scene, (d) is a real image of another building scene, (e) is an image to be inpainted of another building scene, (f) is an inpainted image of another building scene, (g) is a real image of an animal scene, (h) is an image to be inpainted of an animal scene, (i) is an inpainted image of an animal scene, (j) is a real image of an indoor scene, (k) is an image to be inpainted of an indoor scene, and (l) is an inpainted image of an indoor scene; Figure 7 The three groups of visual comparison results of the image inpainting model provided by the present application and several existing methods on the Indoor data set are shown in the figures, wherein, (a) represents three groups of real images, (b) represents three groups of images to be inpainted, (c) represents three groups of inpainted images obtained by the EC method, (d) represents three groups of inpainted images obtained by the RGM method, (e) represents three groups of inpainted images obtained by the MST method, (f) represents three groups of inpainted images obtained by the LAMA method, and (g) represents three groups of inpainted images obtained by the image inpainting model provided by the present application; Figure 8 The effect diagrams corresponding to the ablation experiment results of three different network architectures are shown in the figures, wherein, (a) represents two groups of real images, (b) represents two groups of images to be inpainted, (c) represents two groups of inpainted images obtained by the VC network, (d) represents two groups of inpainted images obtained by the CRFN network, and (e) represents two groups of inpainted images obtained by the CRFN-Adv network; Figure 9 Typical inpainting results obtained under different global receptive field scale settings are shown, wherein (a) shows two groups of real images, (b) shows two groups of inpainting images obtained under controllable scale factor , (c) shows two groups of inpainting images obtained under controllable scale factor ; Figure 10 The schematic diagram of mask enhanced ablation experimental results is shown, wherein (a) shows three groups of images to be inpainted, (b) shows three groups of images without enhancement, (c) shows the edge images corresponding to the three groups of images without enhancement, (d) shows three groups of enhanced inpainting images, and (e) shows the edge images corresponding to the three groups of enhanced images. Figure 11 The structural schematic diagram of an image inpainting system based on a controllable receptive field provided by the present application is shown. DETAILED DESCRIPTION

[0017] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0019] Figure 1 The flowchart of the image inpainting method based on a controllable receptive field in the present application is shown, which specifically includes the following steps: S101, the edges of the image to be inpainted are refined through dilation and erosion operations to obtain a binary mask edge enhanced image, and the image to be inpainted and the binary mask edge enhanced image are spliced to obtain a spliced image.

[0020] In one embodiment, the edges of the image to be inpainted are refined through dilation and erosion operations to obtain a binary mask edge enhanced image, which includes: performing dilation operation on the binary mask image of the image to be inpainted, and refining the edges of the dilated binary mask image through erosion operation to obtain a binary mask edge enhanced image.

[0021] When image inpainting is needed for the to-be-repaired image, first, a binary mask image of the to-be-repaired image is extracted, and an inflation operation is performed on the binary mask image, the edge of the inflated binary mask image is thinned through an erosion operation, and then the binary mask image after the erosion operation is multiplied by the to-be-repaired image to obtain a binary mask edge enhanced image; the to-be-repaired image and the corresponding binary mask edge enhanced image are spliced to obtain a spliced image.

[0022] The to-be-repaired image is obtained , the binary mask edge enhanced image is obtained splicing to obtain a spliced image .

[0023] The to-be-repaired image is a 3-channel image, the binary mask edge enhanced image is a 1-channel image, the obtained spliced image is a 4-channel image, H is the height, W is the width, represents a splicing operation.

[0024] In the image inpainting task, the mask edge information is crucial for generating realistic and coherent inpainting results. However, traditional image inpainting methods usually ignore the complexity of the mask edge, resulting in unnatural transitions or artifacts at the mask boundary, thereby affecting the overall quality of the inpainting effect. This lack of edge details in the inpainting result, especially when dealing with complex images, often cannot maintain visual consistency and structural coherence. The mask edge enhancement method provided by the present application uses an adaptive inflation kernel to inflate the mask image to expand the mask area and enhance its edge representation, thereby improving the learning ability of the image inpainting model for boundary shape and details. This process helps the model to identify the edge information within the mask image and provides more rich structural clues for subsequent inpainting. Considering that separate inflation may cause excessive expansion of the mask, an erosion operation is further introduced to refine and adjust the edge. While retaining key edge features, it effectively suppresses redundant expansion and avoids problems such as artifacts or edge smoothing. Through the joint processing of inflation and erosion, the enhanced mask edge guides the inpainting network to pay more attention to edge details, especially in complex structure scenarios, which helps to improve the accuracy and coherence of edge recovery.

[0025] S102, an image inpainting model is obtained; the image inpainting model is constructed based on a multi-layer controllable receptive field module and a semi-jump connection.

[0026] In the complex task of image inpainting, especially when dealing with images containing complex structures or large-scale damage, how to effectively capture global context information to generate high-quality inpainting results is a key challenge. Although traditional Transformer-based generation networks have shown significant advantages in capturing long-range dependencies, they usually limit the Transformer module to the bottleneck layer, which limits the network's ability to effectively capture global information in early layers. Specifically, the bottleneck layer usually only focuses on the extraction of local features, which prevents the network from effectively integrating global information when dealing with images with large-scale damage or complex structures, thereby affecting the final inpainting effect. To overcome this bottleneck, the present invention introduces a controllable receptive field module in the early layers of the model to eliminate the bottleneck restriction, enabling the image inpainting model to fully capture global context information in early layers. Through this design, the image inpainting model can flexibly switch between global and local receptive fields, allowing it to retain the advantages of convolutional neural networks in local feature extraction while fully utilizing the global information capturing capabilities of the attention mechanism in Transformers. This design enables the network to adaptively adjust the receptive field size when dealing with different scale computing tasks, effectively addressing the problem of early layers being unable to capture long-range dependencies in previous methods.

[0027] In one embodiment, as shown in Figure 2 , Figure 2 is a structural diagram of an image inpainting model; the image inpainting model includes an encoder and a decoder connected in series; the encoder includes multiple layers of controllable receptive field modules containing down-sampling layers; the decoder includes multiple layers of controllable receptive field modules containing up-sampling layers. Optionally, the image inpainting model can use a controllable receptive field conditional discriminator during training; the controllable receptive field conditional discriminator is constructed according to multiple controllable receptive field modules. The image inpainting model also introduces a semi-hop connection, which transmits high-frequency information from the early layers without down-sampling to the subsequent decoding stage, thereby preserving fine details.

[0028] In Figure 2 , the image to be inpainted and the binary mask edge enhancement image are spliced (the first splicing operation in Figure 2 ), to obtain a spliced image; then, after the encoder and decoder, the spliced image and the output of the decoder are spliced (the second splicing operation in Figure 2 ) through a semi-hop connection, to obtain an inpainting image. During the model training phase, the controllable receptive field conditional discriminator discriminates the inpainting image predicted by the decoder.

[0029] It should be noted that Figure 2The controllable receptive field modules in the encoder and the decoder are both 2, and the number of the controllable receptive field modules in the encoder and the decoder is not limited, and can be set according to actual needs.

[0030] As shown in Figure 3 , the controllable receptive field module is a structure diagram, and the controllable receptive field module includes a feature splitting unit, a local branch, a global branch and a feature fusion unit. Figure 3 The local branch includes two parallel first convolution units and second convolution units in sequence, and a first addition unit; the global branch includes two parallel first attention units and second attention units in sequence, and a second addition unit; the output end of the feature splitting unit is connected with the input end of the local branch and the global branch, the input of the local branch is the input of the two convolution units, the input of the global branch is the input of the two attention units, the output end of the first convolution unit and the second convolution unit is connected with the input end of the first addition unit, the output end of the first attention unit and the second attention unit is connected with the input end of the second addition unit, the output end of the first addition unit and the output end of the second addition unit are connected with the input end of the feature fusion unit, and the output end of the feature fusion unit is the output end of the controllable receptive field module.

[0031] The output channel number of the controllable receptive field modules in the encoder is increased in sequence, and the output size is reduced in sequence; the output channel number of the controllable receptive field modules in the decoder is reduced in sequence, and the output size is increased in sequence.

[0032] It should be noted that the feature fusion unit of the controllable receptive field module in the encoder further includes a down-sampling layer, which down-samples the output feature of the feature fusion unit, and the down-sampled feature is the output feature of the controllable receptive field module; the feature fusion unit of the controllable receptive field module in the decoder further includes an up-sampling layer, which up-samples the output feature of the feature fusion unit, and the up-sampled feature is the output feature of the controllable receptive field module.

[0033] The first attention unit and the second attention unit are both attention units constructed by cross-covariance attention mechanism, as shown in Figure 4 . Figure 4It is a structural schematic diagram of an attention unit; the attention units each comprise a first cross-covariance attention mechanism unit, a third adding unit, a second cross-covariance attention mechanism unit and a fourth adding unit connected in sequence, the input of the first cross-covariance attention mechanism unit is the input of the attention unit, the output end of the first cross-covariance attention mechanism unit is connected with the first input end of the third adding unit, the second input end of the third adding unit is the input end of the attention unit, the output end of the third adding unit is connected with the input end of the second cross-covariance attention mechanism unit and the first input end of the fourth adding unit respectively, the output end of the second cross-covariance attention mechanism unit is connected with the second input end of the fourth adding unit, and the output end of the fourth adding unit is the output end of the attention unit.

[0034] Optionally, the cross-covariance attention mechanism is: (1); wherein, is an attention feature obtained by the cross-covariance attention mechanism, denotes a normalization function, , and are a value vector, a query vector and a key vector respectively, is a vector dimension of and , is a transpose of .

[0035] S103, input the spliced image to the image inpainting model, in each controllable receptive field module of the image inpainting model, split the input feature of the controllable receptive field module along the feature channel dimension according to the controllable scale factor to obtain first split features and second split features; perform convolution operation on the first split features and the second split features respectively to obtain first convolution features and second convolution features, and perform feature extraction on the first split features and the second split features in the channel dimension through the cross-covariance attention mechanism respectively to obtain first attention features and second attention features; add the first convolution features and the second convolution features to obtain local features; add the first attention features and the second attention features to obtain global features; weight the local features and the global features along the feature channel dimension to obtain fusion features; and splice the fusion features output by the last controllable receptive field module and the spliced image to obtain an inpainting image corresponding to the to-be-inpainted image.

[0036] In the controllable receptive field module of each layer of the image inpainting model, the implementation process of the controllable receptive field module specifically includes: the input feature is split along the feature channel dimension through the controllable scale factor in the feature splitting unit to obtain first split features and second split features; the first split features are subjected to convolution operation through the first convolution unit to obtain first convolution features, the second split features are subjected to convolution operation through the second convolution unit to obtain second convolution features, the first split features are subjected to feature extraction in the channel dimension through the cross-covariance attention mechanism in the first attention unit to obtain first attention features, and the second split features are subjected to feature extraction in the channel dimension through the cross-covariance attention mechanism in the second attention unit to obtain second attention features; the first convolution features and the second convolution features are added through the first addition unit to obtain local features; the first attention features and the second attention features are added through the second addition unit to obtain global features; and the local features and the global features are weighted along the feature channel dimension through the feature fusion unit to obtain fused features.

[0037] It should be noted that the first convolution unit and the second convolution unit each include a convolution layer, a batch normalization layer and a nonlinear activation function layer connected in sequence.

[0038] It should be noted that the controllable scale factor in the image inpainting model can be adjusted according to different persons and different scene requirements. According to the task and scene requirements, the image inpainting model is trained to determine the controllable scale factor. The parameters in the first convolution unit and the second convolution unit are different, and the parameters in the first attention unit and the second attention unit are also different, which are obtained according to model training optimization.

[0039] Specifically, the input feature of the controllable receptive field module is , and the feature tensor is .

[0040] The local branch learns neighborhood features through convolution operation, and the global branch aims to capture long-distance context information. The channel allocation of the local branch and the global branch is controlled by the controllable scale factor . The controllable scale factor is able to flexibly adjust the proportion of local and global receptive fields, thereby optimizing the performance of different inpainting tasks.

[0041] After obtaining the controllable scale factor, the input feature is split along the feature channel dimension , that is , the first split feature is , , and the second split feature is , .

[0042] Local branch: first convolution feature and second convolution feature , to obtain a local feature , wherein, represents a convolution operation in the first convolution unit, represents a convolution operation in the second convolution unit.

[0043] Global branch: first attention feature and second attention feature , to obtain a global feature , wherein, represents a feature extracted by the first attention unit, represents a feature extracted by the second attention unit, represents a first attention feature, represents a second attention feature.

[0044] For any one of the first attention unit and the second attention unit, the implementation process of the attention unit includes: inputting the input feature of the attention unit into the first cross-covariance attention mechanism unit to obtain a first feature; adding the first feature and the input feature of the attention unit through a third addition unit to obtain a second feature; inputting the second feature into the second cross-covariance attention mechanism unit to obtain a third feature; adding the third feature and the second feature through a fourth addition unit to obtain an attention feature, that is, the output feature of the attention unit.

[0045] wherein, in the cross-covariance attention mechanism unit, the value vector of the input feature of the cross-covariance attention mechanism unit , the query vector and the key vector , and then the output feature of the cross-covariance attention mechanism unit is calculated according to

[0046] The local feature and the global feature are input into a feature fusion unit; the fusion feature output by the feature fusion unit is: , is the global feature, is the local feature.

[0047] In the present application, the controllable receptive field module adaptively adjusts the receptive field size. After processing by the controllable receptive field module, the feature tensor shape is usually consistent with the input, but it contains more rich context information, which helps to restore the global structure and details.

[0048] ​In one embodiment, in the encoder, the controllable receptive field module also includes a downsampling layer, the input end of the downsampling layer is connected to the output end of the feature fusion unit, and the output end of the downsampling layer is the output end of the controllable receptive field module; after each layer of the controllable receptive field module is executed, the fusion features need to be downsampled through the downsampling layer, and the downsampled fusion features are the input features of the controllable receptive field module of the next layer, that is, the input features of the feature splitting unit of the controllable receptive field module of the next layer, and the input features are split along the feature channel dimension through the controllable proportional factor in the feature splitting unit of the controllable receptive field module of the next layer.

[0049] In the decoder, the controllable receptive field module also includes an upsampling layer. The input end of the upsampling layer is connected to the output end of the feature fusion unit, and the output end of the upsampling layer is the output end of the controllable receptive field module. The fusion features output by the feature fusion unit are upsampled by the upsampling layer, and the upsampled fusion features are used as the input features of the next layer of controllable receptive field module to continue feature extraction.

[0050] Finally, a semi-skip connection is introduced into the image restoration model. Specifically, the image obtained by concatenating the spliced ​​image with the fusion features output by the last layer of controllable receptive field module is used as the restoration image corresponding to the image to be restored.

[0051] In one embodiment, during the training process of the image restoration model, the objective function used by the image restoration model is a mixed objective function of multiple loss functions; the process of constructing the mixed objective function of multiple loss functions specifically includes: S201, determining pixel-level reconstruction loss based on the predicted restoration image and the real sample image.

[0052] The sample image set includes the sample image to be repaired and the corresponding real image, where the given sample image to be repaired is , the decoder in the image restoration model outputs the predicted restoration image corresponding to the sample image to be restored: , where the encoder and decoder can be regarded as the generator of the image restoration model. This represents a generator.

[0053] The pixel-level reconstruction loss calculates the mean absolute error between the predicted value and the true value. It is sensitive to pixel errors and insensitive to outliers, so it can better maintain the clarity of the image structure. The mathematical expression is as follows: (2); in, is the number of pixels in the real sample image, To predict the repaired image The pixel value of each pixel, pixel value of the i-th pixel point of the real sample image.

[0054] S202, determine the controllable receptive field adversarial loss according to the discrimination output of the controllable receptive field condition discriminator when the real sample image and the to-be-repaired sample image are input as conditions and the discrimination output of the controllable receptive field condition discriminator when the model output image and the to-be-repaired sample image are input as conditions; the controllable receptive field condition discriminator is constructed by using the controllable receptive field module.

[0055] The image repairing model further comprises a controllable receptive field condition discriminator, and the controllable receptive field adversarial loss is used to train the generator and the controllable receptive field condition discriminator through a generative adversarial network, so as to improve the authenticity and visual effect of the generated image. The controllable receptive field adversarial loss is calculated as follows: (3); wherein D represents the controllable receptive field condition discriminator, G represents the generator, X represents the to-be-repaired sample image, X represents the real sample image, X represents the predicted repairing image corresponding to the to-be-repaired sample image predicted by the generator, D represents the discrimination output of the controllable receptive field condition discriminator when the real sample image and the to-be-repaired sample image are input as conditions, D represents the discrimination output of the controllable receptive field condition discriminator when the predicted repairing image and the to-be-repaired image are input as conditions, and represents expectation.

[0056] S203, extract the features of the predicted repairing image and the real sample image at different levels respectively, and determine the feature matching loss according to the feature difference between the predicted repairing image and the real sample image.

[0057] The present application incorporates the feature matching loss based on the controllable receptive field condition discriminator. The controllable receptive field condition discriminator comprises a plurality of controllable receptive field modules, specifically: features from different layers of the controllable receptive field condition discriminator are extracted, and intermediate features from the real sample image and the predicted repairing image are trained to match. The feature matching loss is calculated as follows.

[0058] (4); wherein represents expectation, N represents the number of feature extraction layers in the controllable receptive field condition discriminator, Xi represents the i-th pixel point of the real sample image.​ total number of features of the layer, denotes the feature of the real sample image extracted by the controllable receptive field condition discriminator in the i-th layer, denotes the feature of the real sample image extracted by the controllable receptive field condition discriminator in the i-th layer, denotes the feature of the predicted inpainting image extracted by the controllable receptive field condition discriminator in the i-th layer, denotes the feature of the predicted inpainting image extracted by the controllable receptive field condition discriminator in the i-th layer, denotes the first norm.

[0059] S204, a plurality of pre-training high-receptive field network features of the predicted inpainting image and the real sample image are extracted respectively, and a difference between the predicted inpainting image and the real sample image in a global structure is captured according to the pre-training high-receptive field network features, so as to obtain a high-receptive field loss.

[0060] The high-receptive field loss is a loss function designed to improve the modeling capability of the global structure and long-distance dependence in the image inpainting task. The core idea is to capture the difference between the generated image and the real image in the global structure by using the high-receptive field features extracted by the deep neural network. The high-receptive field loss is calculated as follows: (5); wherein, is the number of feature extraction layers of the pre-training high-receptive field network, denotes the feature of the real sample image extracted by the pre-training high-receptive field network in the i-th layer, denotes the feature of the real sample image extracted by the pre-training high-receptive field network in the i-th layer, denotes the feature of the predicted inpainting image extracted by the pre-training high-receptive field network in the i-th layer, which is usually derived from the high-layer convolutional layer of the pre-training model; is the predicted inpainting image; y is the real sample image; is the channel number, height and width of the i-th layer feature, denotes the second norm. The pre-training high-receptive field network is a network constructed by a plurality of controllable receptive field modules and trained in advance. S205, based on the total variation theory, a total variation loss of the predicted inpainting image is determined.

[0061] The total variation (TV) loss is based on the total variation theory of the image, which aims to smooth the image by constraining the difference between adjacent pixels in the image, and can effectively preserve the edges and other important structural information in the image. By punishing large-scale pixel changes, the TV loss can reduce noise and artifacts in the image, thereby improving the visual quality of the image. For example, the TV loss is calculated as follows.

[0062]

[0063] ​​​ (6); wherein, denotes the domain of the image , denotes the gradient of the image , and denote the partial derivatives of the image in and directions, respectively, , .

[0064] The TV loss of the predicted repair image can be calculated according to the calculation manner of the TV loss of formula (6).

[0065] S206, calculate the structural similarity between the predicted repair image and the real sample image, and determine the structural similarity loss.

[0066] The structural similarity loss is mainly used to measure the structural similarity between two images or data. This loss function not only focuses on the pixel-level difference, but also combines the features of brightness, contrast and structure, so as to more accurately reflect the visual similarity between images, so as to calculate the structural similarity loss between image and image . The calculation formula of the structural similarity loss is as follows.

[0067] (7); wherein, denotes the pixel mean of the image , denotes the pixel mean of the image , the pixel variance of the image , denotes the pixel variance of the image , denotes the pixel covariance of the image and image , and are constants introduced for stable calculation.

[0068] Therefore, the structural similarity loss between the predicted repair image and the real sample image can be calculated according to formula (7).

[0069] S207, according to the pixel-level reconstruction loss, the controllable receptive field adversarial loss, the feature matching loss, the high receptive field loss, the total variation loss and the structural similarity loss, determine the target function.

[0070] The objective function in the present application is a mixed combination. It includes pixel-level reconstruction loss , controllable receptive field adversarial loss , feature matching loss , high receptive field loss , total variation loss and structural similarity loss . Therefore, the final objective function is: (8); Wherein, , , , , and are the weights of the corresponding loss.

[0071] Parameter update: Update the controllable receptive field condition discriminator: fix the generator parameters, and calculate the gradient using the controllable receptive field condition discriminator loss, then backpropagate to update the weights of the controllable receptive field condition discriminator to make it more accurate to distinguish true / false samples.

[0072] Update the generator: fix the controllable receptive field condition discriminator parameters, and calculate the gradient using the value of the objective function, then backpropagate to update the weights of the generator to make the predicted repair image optimized under each loss.

[0073] Repeat iteration: alternately execute the above steps until convergence.

[0074] It should be noted that in model deployment or actual inference, the controllable receptive field condition discriminator and these losses do not participate in running. When inferring, only the trained generator is used, and given the input image, the predicted repair image can be output.

[0075] In one embodiment, as shown in Figure 5 , the present application also provides a controllable receptive field based image repair method, which includes the following steps: S501, obtaining an image to be repaired.

[0076] S502, performing dilation operation on the binary mask image of the image to be repaired.

[0077] S503, performing erosion operation on the edge of the dilated binary mask image to obtain a binary mask edge enhanced image.

[0078] S504, an image inpainting model is obtained; the image inpainting model comprises an encoder and a decoder connected in sequence; the encoder comprises multiple layers of controllable receptive field modules comprising down-sampling layers; the decoder comprises multiple layers of controllable receptive field modules comprising up-sampling layers; high-frequency information of an early layer without down-sampling is transmitted to a subsequent decoding stage through semi-hop connection in the image inpainting model; and a target function used in a training process of the image inpainting model is a target function mixed with multiple loss functions.

[0079] S505, the spliced image is input into the image inpainting model, in each controllable receptive field module of the image inpainting model, input features are split along a feature channel dimension through a controllable scale factor in a feature splitting unit to obtain first split features and second split features.

[0080] S506, in a local branch, a first convolution unit is used to perform convolution operation on the first split features to obtain first convolution features, a second convolution unit is used to perform convolution operation on the second split features to obtain second convolution features, and a first addition unit is used to add the first convolution features and the second convolution features to obtain local features.

[0081] S507, in a global branch, a cross-covariance attention mechanism in a first attention unit is used to perform feature extraction in a channel dimension on the first split features to obtain first attention features, a cross-covariance attention mechanism in a second attention unit is used to perform feature extraction in a channel dimension on the second split features to obtain second attention features, and a second addition unit is used to add the first attention features and the second attention features to obtain global features.

[0082] S508, a feature fusion unit is used to weight the local features and the global features along a feature channel dimension to obtain fused features.

[0083] S509, a semi-hop connection is used to splice the fused features output by the last controllable receptive field module and the spliced image to obtain an inpainting image corresponding to the to-be-inpainted image.

[0084] In an embodiment, in order to illustrate the effectiveness of the trained image inpainting model, experiments are performed on the image inpainting model. Specifically, the experiments are performed under an Ubuntu system, based on a Pytorch v1.7.1 framework, and using an NVIDIA GeForce RTX-3090 GPU to train the model, an AdamW optimizer is used, and a learning rate is set to 1e−4 to ensure the stability and efficiency of the training.

[0085] 、 、 、 、 and The weights of the image restoration model are respectively set as 10, 100, 10, 10, 10 and 0.1. The data set is selected as Place2 and Indoor, and four representative image restoration methods of introducing edge information (Edge Connect, EC), improving a residual module (ResidualGather Module, RGM), a multi-scale sketch tensor inpainting network (Multi-scale Sketch Tensorinpainting, MST) and a new complete network framework using fast Fourier convolution (Large Mask Inpainting, LAMA) are compared, and finally the peak signal-to-noise ratio (Peak Signal-to-Noise Ratio, PSNR), Fréchet inception distance (Fréchet Inception Distance, FID) and learned perceptual image patch similarity (Learned Perceptual Image Patch Similarity, LPIPS) are used as evaluation indexes of the ablation experiment, and the above indexes are respectively from multiple dimensions such as reconstruction accuracy, perceptual consistency and visual similarity, so that the restoration effect of the model is systematically and comprehensively evaluated.

[0086] As shown in Table 1, Table 1 is the test result of the image restoration model (CRFN-Adv) provided by the application on different data sets.

[0087] Table 1 is the test result of the image restoration model (CRFN-Adv) provided by the application on different data sets.

[0088] As shown in Table 1, Table 1 is the test result of the image restoration model (CRFN-Adv) provided by the application on different data sets. Figure 6 As shown in Table 1, Table 1 is the test result of the image restoration model (CRFN-Adv) provided by the application on different data sets. Figure 6 The image restoration effect of the image restoration model provided by the application in four complex scenes is shown in the figures, wherein, (a) is a real image of a building scene, (b) is a to-be-restored image of a building scene, (c) is a restored image of a building scene, (d) is a real image of another building scene, (e) is a to-be-restored image of another building scene, (f) is a restored image of another building scene, (g) is a real image of an animal scene, (h) is a to-be-restored image of an animal scene, (i) is a restored image of an animal scene, (j) is a real image of an indoor scene, (k) is a to-be-restored image of an indoor scene, and (l) is a restored image of an indoor scene; it can be seen from the results that the image restoration model can effectively restore various semantic contents such as building, animal and indoor structure. In the regular occlusion scene, the texture reconstructed by the model is natural and the structure is continuous; in the face of irregular occlusion, such as a duck and its background, the image restoration model can also accurately restore key details and realize semantic consistent restoration.

[0089] As shown in Figure 7 , Figure 7 Figures (a), (b), (c), (d), (e), (f) and (g) are schematic diagrams of three sets of visual comparison results of the image inpainting model provided by the present application and several existing methods on the Indoor dataset, wherein (a) shows three sets of real images, (b) shows three sets of images to be inpainted, (c) shows three sets of inpainted images obtained by the EC method, (d) shows three sets of inpainted images obtained by the RGM method, (e) shows three sets of inpainted images obtained by the MST method, (f) shows three sets of inpainted images obtained by the LAMA method, and (g) shows three sets of inpainted images obtained by the image inpainting model provided by the present application.

[0090] In one embodiment, the effectiveness of the image inpainting module is unfolded, and a step-by-step promotion path design of three different network architectures is introduced from the basic structure to the controllable receptive field module, and then to the controllable receptive field condition discriminator collaborative optimization to realize systematic analysis and comparative evaluation of its performance, which specifically includes: a baseline network (Vanilla Convolution, VC) constructed by only using conventional convolution operation, a generation network (CRFN) constructed by introducing a complete controllable receptive field module, and a new generation network (CRFN-Adv) obtained by further combining the controllable receptive field condition discriminator for adversarial training on the basis of CRFN, i.e. the image inpainting model provided by the present application. PSNR, FID and LPIPS are used as evaluation indexes of the ablation experiment, and the above indexes are respectively from multiple dimensions such as reconstruction accuracy, perceptual consistency and visual similarity, to systematically and comprehensively evaluate the inpainting effect of the model. As shown in Table 2, Table 2 is the ablation experiment results of three different network architectures.

[0091] Table 2 Ablation experiment results of three different network architectures

[0092] In Table 2, the values that perform best on each index are highlighted in bold to facilitate intuitive comparison of the performance differences between different methods. From the results, it can be seen that in terms of PSNR, CRFN-Adv obtains the highest score, indicating that it has a significant advantage in image pixel-level reconstruction accuracy and detail restoration capability compared to other models; in terms of FID and LPIPS indexes, CRFN-Adv obtains the lowest score, further verifying its superiority in the perceptual quality and realism of generated images, and it can more effectively fit the consistency of human subjective evaluation.

[0093] It is worth emphasizing that CRFN-Adv can maintain the overall coherence of image structure and generate delicate and natural texture details when facing complex shapes, multi-semantic content and texture non-uniform distribution of occluded areas, fully demonstrating its powerful context modeling and feature expression ability. The performance improvement is due to the controllable receptive field mechanism, which enables the network to dynamically adjust the receptive field according to the semantic needs of different regions, thereby realizing the effective integration and response of multi-scale context information.

[0094] As Figure 8 shown, Figure 8 are the effect pictures corresponding to the ablation experiment results of three different network architectures, wherein, (a) figure represents two groups of real images, (b) figure represents two groups of images to be repaired, (c) figure represents two groups of repair images obtained by the VC network, (d) figure represents two groups of repair images obtained by the CRFN network, (e) figure represents two groups of repair images obtained by the CRFN-Adv network.

[0095] The present application further explores the influence of global receptive field proportion on image repair performance through systematic ablation experiment, aiming to evaluate the adaptability and effectiveness of the controllable receptive field mechanism in multi-scene image repair task. In order to ensure the wide applicability of the experiment and the representativeness of the results, the experimental samples are selected from the Places2 dataset with natural texture characteristics and the Indoor dataset with complex structure. Figure 9 Typical repair results obtained under different global receptive field proportion settings are shown, wherein, (a) figure represents two groups of real images, (b) figure represents two groups of repair images obtained when the controllable proportion factor is 2, (c) figure represents two groups of repair images obtained when the controllable proportion factor is 4; Table 3 quantitatively analyzes the model performance from multiple objective index dimensions.

[0096] Table 3 Ablation results of different proportion receptive fields

[0097] The experimental results clearly show that the adjustment of the global receptive field ratio plays a key regulatory role between the model's perception ability and the repair performance. In the Places2 dataset, the scene mainly consists of natural scenery and high-frequency textures. In contrast, the Indoor dataset emphasizes the integrity of geometric structure and spatial layout, involving a large number of indoor scene elements with clear boundaries, such as walls, furniture, and doors and windows. The experimental results show that when the global receptive field ratio is increased to 0.5, the model can more effectively construct the context semantic association, accurately restore the structural information of the missing area, and improve the spatial consistency of the scene. Especially in the face of large-scale occluded areas, the enhanced global modeling capability significantly improves the problems of structural distortion and boundary fracture, further improving the semantic coherence and visual rationality of the repaired image.

[0098] In summary, this experiment verifies the key role of the global receptive field ratio in the image inpainting task. In different scenarios, the ratio should be adjusted according to the image features to achieve a better balance between detail restoration and structure reconstruction, thereby achieving higher quality image inpainting.

[0099] To further verify the key role of the mask edge in the image inpainting process, a special experiment on mask edge enhancement was conducted to systematically evaluate its effect on improving the quality of inpainting. This experiment aims to explore the impact on the overall inpainting results after specific processing of the mask edge area. The mask enhancement ablation experiment results are as follows: Figure 10As shown in the figure, (a) represents three groups of images to be repaired, (b) represents three groups of non-enhanced images, (c) represents the edge images corresponding to the three groups of non-enhanced images, (d) represents three groups of enhanced repaired images, and (e) represents the edge images corresponding to the three groups of enhanced images. Specifically, the mask edge enhancement operation optimizes the feature extraction and representation of the boundary region, so that the repair model can obtain more accurate edge structure information in the inference stage, thereby providing more effective guidance for the structure reconstruction of the damaged region. In the experiment, by enhancing the mask edge, CRFN-Adv can effectively suppress the common artifact phenomenon in the repair process. The generation of artifacts usually occurs in the edge transition region, especially in the repair task of complex texture and structure, and the traditional method often has difficulty in maintaining accurate consistency and transition smoothness in the edge region. By enhancing the mask edge information, CRFN-Adv can better capture the edge details and ensure that the transition between the repaired region and the surrounding region is more natural and smooth, thereby significantly reducing the generation of artifacts. In addition, the experimental results also show that through the enhancement of the mask edge, the fine texture in the image is more completely preserved. The strengthening of the edge information enables the repair model to more accurately restore the tiny texture details in the image, especially in the region with complex texture or severe damage, the enhanced edge information can effectively guide the repair network to make more detailed recovery. This optimization of edge information not only improves the visual effect of image repair, but also enhances the naturalness and consistency of the repair result to a certain extent. The following figure is the ablation experiment result of mask enhancement.

[0100] In the application of the image repair method based on controllable receptive field provided by the present application, the image repair method based on controllable receptive field provided by the present application can be performed without Figure 1 The order of execution of each step shown in the figure can be determined as needed, and the present application does not limit the execution order of each step.

[0101] The above is the image repair method based on controllable receptive field provided by one or more embodiments of the present application. Based on the same idea, the present application also provides a corresponding image repair system based on controllable receptive field, as shown in the figure, which includes an image uploading module, an image preprocessing module, an image repair module, a result visualization module and an image saving module. Figure 11

[0102] The image uploading module is used to upload the image to be repaired.

[0103] The image preprocessing module is used to refine the edge of the image to be repaired through dilation and erosion operations to obtain a binary mask edge enhanced image, and to splice the image to be repaired and the binary mask edge enhanced image to obtain a spliced image.

[0104] ​An image inpainting module is configured to obtain an image inpainting model, wherein the image inpainting model is constructed based on a plurality of controllable receptive field modules and a semi-jump connection; the image inpainting module is configured to input a stitched image into the image inpainting model, split input features of the controllable receptive field module along a feature channel dimension according to a controllable scale factor to obtain first split features and second split features in each controllable receptive field module of the image inpainting model, perform convolution operations on the first split features and the second split features to obtain first convolution features and second convolution features, respectively, and perform feature extraction on the first split features and the second split features in a channel dimension through a cross-covariance attention mechanism to obtain first attention features and second attention features, respectively, add the first convolution features and the second convolution features to obtain local features, add the first attention features and the second attention features to obtain global features, weight the local features and the global features along the feature channel dimension to obtain fused features, and splice the fused features output by a last controllable receptive field module and the stitched image to obtain an inpainting image corresponding to the image to be inpainted.

[0105] An result visualization module is configured to display the obtained inpainting image.

[0106] An image saving module is configured to save the obtained inpainting image in a preset position.

[0107] For specific limitations of the image inpainting system based on the controllable receptive field, refer to the limitations of the image inpainting method based on the controllable receptive field in the foregoing, which will not be described herein again.

[0108] Based on the same idea, the application further provides a corresponding image inpainting device based on the controllable receptive field, which comprises: A stitching module is configured to refine edges of the image to be inpainted through inflation and erosion operations to obtain a binary mask edge enhanced image, and splice the image to be inpainted and the binary mask edge enhanced image to obtain a stitched image; The image reconstruction module is configured to obtain an image inpainting model, wherein the image inpainting model is constructed based on the multi-layer controllable receptive field module and the semi-jump connection; the spliced image is input into the image inpainting model; in each layer of the controllable receptive field module of the image inpainting model, the input features of the controllable receptive field module are split along the feature channel dimension according to the controllable scale factor to obtain first split features and second split features; the first split features and the second split features are subjected to convolution operations respectively to obtain first convolution features and second convolution features, and the first split features and the second split features are subjected to feature extraction in the channel dimension through the cross-covariance attention mechanism respectively to obtain first attention features and second attention features; the first convolution features and the second convolution features are added to obtain local features; the first attention features and the second attention features are added to obtain global features; the local features and the global features are weighted along the feature channel dimension to obtain fused features; and the fused features output by the last layer of the controllable receptive field module are spliced with the spliced image to obtain a repair image corresponding to the image to be repaired.

[0109] The specific limitations of the image inpainting device based on the controllable receptive field can be referred to the limitations of the image inpainting method based on the controllable receptive field in the foregoing, which will not be described herein. Each module in the image inpainting device based on the controllable receptive field can be realized by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0110] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the above Figure 1 The image inpainting method based on the controllable receptive field is provided.

[0111] The present application also provides a structural diagram of a computer device, which comprises a processor, an internal bus, a network interface, a memory, and a non-volatile memory at the hardware level, and can also comprise other hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1 The image inpainting method based on the controllable receptive field is provided.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of the methods. In the embodiments of the present application, any reference to memory, storage, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0113] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

Claims

1. A controllable receptive field based image inpainting method, characterized in that, The method comprises the following steps: The edges of the image to be repaired are refined through inflation and erosion operations to obtain a binary mask edge enhanced image, and the image to be repaired and the binary mask edge enhanced image are spliced to obtain a spliced image; An image inpainting model is obtained; the image inpainting model is constructed based on a plurality of controllable receptive field modules and a semi-jump connection; The spliced image is input into the image inpainting model, in each controllable receptive field module of the image inpainting model, the input features of the controllable receptive field module are split along the feature channel dimension according to a controllable scale factor to obtain first split features and second split features; the first split features and the second split features are respectively subjected to convolution operations to obtain first convolution features and second convolution features, and the first split features and the second split features are respectively subjected to feature extraction in the channel dimension through a cross-covariance attention mechanism to obtain first attention features and second attention features; the first convolution features and the second convolution features are added to obtain local features; the first attention features and the second attention features are added to obtain global features; the local features and the global features are weighted along the feature channel dimension to obtain fused features; The fused features output by the last controllable receptive field module are spliced with the spliced image to obtain a repaired image corresponding to the image to be repaired.

2. The method of claim 1, wherein, The edges of the image to be repaired are refined through inflation and erosion operations to obtain a binary mask edge enhanced image, comprising: The binary mask image of the image to be repaired is subjected to an inflation operation, and the edges of the inflated binary mask image are refined through an erosion operation to obtain a binary mask edge enhanced image.

3. The method of claim 1, wherein, The image inpainting model comprises an encoder and a decoder connected in sequence; the encoder comprises a plurality of controllable receptive field modules containing down-sampling layers; the decoder comprises a plurality of controllable receptive field modules containing up-sampling layers; The controllable receptive field module comprises a feature splitting unit, a local branch, a global branch and a feature fusion unit, the local branch comprises two first convolution units and two second convolution units connected in parallel in sequence, and a first addition unit; the global branch comprises two first attention units and two second attention units connected in parallel in sequence, and a second addition unit; the output end of the feature splitting unit is connected with the input ends of the local branch and the global branch, the input of the local branch is the input of the two convolution units, the input of the global branch is the input of the two attention units, the output ends of the first convolution unit and the second convolution unit are connected with the input end of the first addition unit, the output ends of the first attention unit and the second attention unit are connected with the input end of the second addition unit, the output end of the first addition unit and the output end of the second addition unit are connected with the input end of the feature fusion unit, and the output end of the feature fusion unit is the output end of the controllable receptive field module.

4. The method of claim 3, wherein, The output channel numbers of the plurality of controllable receptive field modules connected in sequence in the encoder are increased in sequence, and the output sizes are decreased in sequence; the output channel numbers of the plurality of controllable receptive field modules connected in sequence in the decoder are decreased in sequence, and the output sizes are increased in sequence; the implementation process of the controllable receptive field module comprises: The input features are split along the feature channel dimension by a controllable scale factor in the feature splitting unit to obtain first split features and second split features; The first split features are subjected to convolution operation by a first convolution unit to obtain first convolution features, the second split features are subjected to convolution operation by a second convolution unit to obtain second convolution features, the first split features are subjected to feature extraction in the channel dimension by a cross-covariance attention mechanism in a first attention unit to obtain first attention features, and the second split features are subjected to feature extraction in the channel dimension by a cross-covariance attention mechanism in a second attention unit to obtain second attention features; The first convolution features and the second convolution features are added by a first addition unit to obtain local features, and the first attention features and the second attention features are added by a second addition unit to obtain global features; The local features and the global features are weighted along the feature channel dimension by a feature fusion unit to obtain fused features.

5. The method of claim 4, wherein, The first attention unit and the second attention unit are both attention units constructed by a cross-covariance attention mechanism, and each attention unit comprises a first cross-covariance attention mechanism unit, a third addition unit, a second cross-covariance attention mechanism unit and a fourth addition unit connected in sequence, and the implementation process of the attention unit comprises: The input features of the attention unit are input into the first cross-covariance attention mechanism unit to obtain first features; The first features and the input features of the attention unit are input into the third addition unit to obtain second features; The second features are input into the second cross-covariance attention mechanism unit to obtain third features; The third features and the second features are input into the fourth addition unit to obtain attention features, which are the output features of the attention unit.

6. The method of claim 4, wherein, The cross-covariance attention mechanism is: ； wherein, is the attention feature obtained by the cross-covariance attention mechanism, denotes a normalization function, , and are a value vector, a query vector and a key vector, respectively, is the vector dimension of and , is the transpose of .

7. The method of claim 1, wherein, In the image inpainting model training process, the objective function adopted by the image inpainting model is a multi-loss function mixed objective function; the construction process of the multi-loss function mixed objective function specifically comprises: According to the predicted repair image and the real sample image, a pixel-level reconstruction loss is determined; According to the discriminative output of the controllable receptive field conditional discriminator inputting the real sample image and the to-be-repaired sample image as conditions and the discriminative output of the controllable receptive field conditional discriminator inputting the model output image and the to-be-repaired sample image as conditions, a controllable receptive field adversarial loss is determined; the controllable receptive field conditional discriminator is constructed by using a controllable receptive field module; Features at different levels of the predicted repair image and the real sample image are extracted respectively, and a feature matching loss is determined according to the feature difference between the predicted repair image and the real sample image; A plurality of pre-training high receptive field network features of the predicted repair image and the real sample image are extracted respectively, and a high receptive field loss is obtained by capturing the difference in global structure between the predicted repair image and the real sample image according to the pre-training high receptive field network features; Based on the total variation theory, a total variation loss of the predicted repair image is determined; The structural similarity between the predicted repair image and the real sample image is calculated to determine a structural similarity loss. A target function is determined according to a pixel-level reconstruction loss, a controllable receptive field adversarial loss, a feature matching loss, a high receptive field loss, a total variation loss and a structural similarity loss.

Images