Image inpainting method, apparatus, device, and medium

Abstract

The present application relates to image inpainting method, device, equipment and medium. Method includes: shallow feature extraction is carried out to original image, and initial feature is obtained;Non-overlapping window division is carried out to initial feature, and a plurality of first division features are obtained;In each window, the first division feature is scored by self-attention, the first update feature in the window is obtained, and then residual fusion is carried out with the initial feature, and the first fusion feature is obtained;Nonlinear transformation is carried out to the first fusion feature, and the first output feature is obtained;The first output feature is divided by overlapping window, and the correlation weight between the features in each window is calculated, and the second output feature is obtained;Then residual fusion is carried out with the initial feature, and the second fusion feature is output;Iterative processing is carried out with the second fusion feature as input, and the final fusion feature is output;Residual fusion is carried out with the final fusion feature and the initial feature, and the inpainting image is output. The present application can improve the calculation efficiency, and the repair quality is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image inpainting method, device, equipment and medium. BACKGROUND

[0002] Image inpainting aims to reconstruct the missing regions in an image due to factors such as occlusion, aging, compression, transmission damage, etc., so that the inpainted image is as close as possible to the original image in structure, texture and semantics. This task not only has important theoretical significance, but also shows wide value in real-world applications. For example, in the fields of cultural relic image inpainting, old photo restoration, security monitoring image inpainting, video content completion, medical image enhancement, remote sensing image processing, virtual reality and augmented reality, image inpainting technology plays an important role. The quality of image inpainting is directly related to the readability, accuracy of the final image and the influence on downstream tasks such as detection, recognition and segmentation, so it has been a research hotspot in the field of computer vision for a long time.

[0003] Traditional image inpainting methods mostly rely on the spatial consistency and texture extensibility of known pixels in the image. For example, partial differential equation (PDE) based methods fill in the missing regions by extending edge information, which is suitable for small-scale damage repair; texture synthesis based repair methods copy content from similar neighboring regions to fill in the missing part; and sparse representation and dictionary learning based methods try to restore the representation of the missing region at the image block level. Although these methods perform well in specific scenarios, they often have difficulty capturing high-level semantic information and perform poorly in large-scale damage or complex structure images, resulting in structural distortion, inconsistent texture, etc.

[0004] With the rise of deep learning, image inpainting methods based on convolutional neural networks (CNN) have developed rapidly. CNN can learn the structural rules and semantic information of images from large-scale data sets through end-to-end training, thus achieving more robust image restoration results. Such methods usually design deep network structures containing feature extraction, context modeling and image reconstruction modules, effectively alleviating the shortcomings of traditional methods in the face of complex repair tasks. In recent years, the integration of Transformer architecture, attention mechanism, multi-scale modeling and residual learning has further improved the accuracy of image inpainting. However, with the increase in model depth and complexity, the computational resource requirements of the network structure have increased dramatically, raising a series of problems:

[0005] On the one hand, current high-performance restoration models often contain a large number of parameters and complex calculation modules, which put higher requirements on storage, computing resources and deployment platforms; on the other hand, in many practical scenarios such as mobile terminals, intelligent security cameras or embedded systems, the computing resources are limited, which makes it difficult to deploy complex deep models. In addition, some existing methods only focus on global structure modeling and ignore local texture details, or lack effective feature fusion mechanism between different scales, resulting in lack of realism in detail of the restored image. SUMMARY

[0006] Therefore, it is necessary to provide an image restoration method, device, equipment and medium capable of solving the problems of low calculation efficiency and poor restoration performance of existing restoration models.

[0007] An image restoration method, the method comprising:

[0008] Step 1, obtaining an original image, performing shallow feature extraction on the original image to obtain initial features;

[0009] Step 2, performing non-overlapping window division on the initial features to obtain a plurality of first divided features; in each window, performing self-attention scoring on the first divided features to obtain first updated features within the window; performing residual fusion on the first updated features and the initial features to obtain first fused features;

[0010] Step 3, performing non-linear transformation on the first fused features to obtain first output features;

[0011] Step 4, performing overlapping window division on the first output features and calculating the correlation weight between the features in each window to obtain second output features;

[0012] Step 5, performing residual fusion on the second output features and the initial features to output second fused features;

[0013] Step 6, taking the second fused features as input, repeating steps 2 to 5 to output final fused features; performing residual fusion on the final fused features and the initial features to output a restored image.

[0014] On the other hand, an image restoration device is also provided, the device comprising:

[0015] An initial feature extraction module is configured to obtain an original image, perform shallow feature extraction on the original image, and obtain initial features;

[0016] The first fusion feature acquisition module is configured to perform non-overlapping window division on the initial feature to obtain a plurality of first division features; perform self-attention scoring on the first division features in each window to obtain first updated features in the window; and perform residual fusion on the first updated features and the initial feature to obtain first fusion features.

[0017] The first output feature acquisition module is configured to perform non-linear transformation on the first fusion features to obtain first output features.

[0018] The second output feature acquisition module is configured to perform overlapping window division on the first output features and calculate correlation weights between features in each window to obtain second output features.

[0019] The second fusion feature acquisition module is configured to perform residual fusion on the second output features and the initial feature to output second fusion features.

[0020] The repaired image output module is configured to perform iterative processing on the second fusion features as input to output final fusion features; and perform residual fusion on the final fusion features and the initial feature to output a repaired image.

[0021] In another aspect, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the image repairing method when executing the computer program.

[0022] In another aspect, a computer readable storage medium is also provided, which stores a computer program, and the computer program implements the steps of the image repairing method when executed by a processor.

[0023] Compared with the prior art, the image repairing method, device, equipment and medium provided by the present application have the following effects:

[0024] 1. The pixel-level dependency is captured by non-overlapping window division and self-attention scoring; the cross-region feature correlation is constructed by overlapping window division and correlation weight calculation. The local fine feature extraction is performed while the continuity between features is ensured, the problem of local blur is avoided, the repairing accuracy is improved, and the reality of the repaired image is ensured.

[0025] 2. The image features are decomposed into local blocks by window division, the high-dimensional calculation caused by directly processing global features is avoided, the memory and calculation overhead are reduced; at the same time, the self-attention mechanism is only calculated locally within the window, the complexity caused by global calculation is avoided, the calculation amount is greatly reduced while the repairing effect is maintained. In addition, the residual fusion is used multiple times, which can effectively avoid repeated learning of basic features, reduce the total amount of network parameters, improve the calculation efficiency, and is particularly suitable for scenes with limited computing resources.

[0026] 3. Through the method provided by the application, an end-to-end network with a small parameter scale can be constructed, and efficient deployment can be realized in a scenario with limited computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the application, and those skilled in the art can obtain other related drawings according to these drawings without any creative labor.

[0028] Figure 1 A flowchart of the image inpainting method provided in Embodiment 1 is shown in FIG. 1.

[0029] Figure 2 A structure diagram of the image inpainting model provided in Embodiment 1 is shown in FIG. 2.

[0030] Figure 3 A visual effect comparison diagram of the first group of different algorithms provided in Embodiment 1 is shown in FIG. 3; wherein, Figure 3 (a) is a to-be-inpainted image, Figure 3 (b) is a visual effect diagram of image inpainting using Bicubic, Figure 3 (c) is a visual effect diagram of image inpainting using SwinIR-L, Figure 3 (d) is a visual effect diagram of image inpainting using the method provided by the application.

[0031] Figure 4 A visual effect comparison diagram of the second group of different algorithms provided in Embodiment 1 is shown in FIG. 4; wherein, Figure 4 (a) is a to-be-inpainted image, Figure 4 (b) is a visual effect diagram of image inpainting using Bicubic, Figure 4 (c) is a visual effect diagram of image inpainting using SwinIR-L, Figure 4 (d) is a visual effect diagram of image inpainting using the method provided by the application.

[0032] Figure 5 A structure block diagram of the image inpainting device provided in Embodiment 2 is shown in FIG. 5.

[0033] Figure 6 An internal structure diagram of the computer device provided in Embodiment 3 is shown in FIG. 6.

[0034] The purposes, functional features and advantages of the application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0035] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0036] It can be understood that the technical solutions among various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0037] The embodiments of the present application will be described in detail below with reference to the drawings.

[0038] Embodiment 1

[0039] The embodiment discloses an image inpainting method. By non-overlapping window division, self-attention modeling is performed in different scale ranges, so as to effectively capture local details and global semantics. Nonlinear transformation is performed on the obtained features, which can enhance the nonlinear expression ability of the attention output features, improve the modeling depth and discrimination ability of the network. By performing overlapping window division, a local self-attention mechanism is constructed, the image texture and edge structure are refined, and the semantic relationship between different regions is strengthened. Moreover, residual connections are widely introduced to ensure the stability of deep features in the propagation process, alleviate the gradient vanishing problem, and have significant advantages in structure consistency and detail restoration.

[0040] Compared with traditional CNN methods or single-scale modeling inpainting methods, the method provided by the present application realizes the optimal balance among inpainting quality, computational efficiency and model lightness. Based on the method provided by the present application, an end-to-end image inpainting network with small parameter size, high precision and high efficiency can be constructed, which has good scalability and migration ability. It is especially suitable for dual deployment requirements of high-performance servers and low-resource edge devices, and has important practical application value and popularization potential in the fields of image processing, security monitoring, virtual reality and the like.

[0041] As shown in FIG. 1, the image inpainting method provided by the embodiment includes the following steps: Figure 1

[0042] Step 1, obtaining an original image, performing shallow feature extraction on the original image to obtain initial features.

[0043] ​It can be understood that the original image is an image to be repaired, and the initial features obtained by shallow feature extraction on the original image can retain the local texture and edge information of the original image, thereby providing support for subsequent deep modeling.

[0044] In step 2, the initial features are divided into a plurality of first divided features by non-overlapping window division; in each window, a self-attention score is calculated for the first divided features to obtain first updated features in the window; and the first updated features are fused with the initial features by residual fusion to obtain first fused features.

[0045] In this step, by non-overlapping window division and self-attention scoring in each window, local details and global semantics can be effectively captured, the representation ability of local features can be enhanced, and fine modeling of image features can be achieved.

[0046] In step 3, the first fused features are subjected to non-linear transformation to obtain first output features.

[0047] It can be understood that non-linear transformation of the first fused features can improve the representation ability of image details and avoid insufficient feature expression ability caused by linear processing.

[0048] In step 4, the first output features are divided into overlapping windows, and the correlation weights between the features in each window are calculated to obtain second output features.

[0049] It can be understood that by dividing the windows in an overlapping manner and calculating the correlation weights between the features in each window, the boundary restrictions of local regions can be broken, the interaction between features can be enhanced, the features of adjacent windows can be interacted, and the texture and edge information in local regions can be more effectively captured. In this way, the detailed information of the image is finely processed, and a richer feature basis is provided for subsequent image reconstruction.

[0050] In step 5, the second output features are fused with the initial features by residual fusion to output second fused features.

[0051] In step 6, the second fused features are taken as input, and steps 2 to 5 are repeated to output final fused features; and the final fused features are fused with the initial features by residual fusion to output a repaired image.

[0052] It can be understood that through more than two rounds of iterative processing and extensive use of residual processing in each round of processing, each round of iteration can further optimize the features based on the results of the previous round, which can effectively reduce the error accumulation caused by a single stage, improve the robustness of image repair, and finally make the repaired image closer to the true effect of the original image.

[0053] As Figure 2As shown, it is a schematic diagram of an image inpainting model constructed based on the method provided in the embodiment. A convolutional layer Conv is arranged at the front end to extract shallow features; two or more deep feature enhancement modules (DSWLI) are connected in series after the convolutional layer Conv to perform deep feature enhancement processing; finally, a convolutional layer Conv is connected for image reconstruction, which maps the high-level semantic features output by the deep feature enhancement module (DSWLI) back to the original image space to generate an inpainting image; at the same time, the output of the first convolutional layer Conv and the output of the last convolutional layer Conv are connected in residual connection, and finally an inpainting image is output. The deep feature enhancement module (DSWLI) includes a dynamic scale window attention sub-module (DSWAB) and a local information interaction sub-module (LIIM). The outputs of the dynamic scale window attention sub-module (DSWAB) and the local information interaction sub-module (LIIM) are respectively connected with a convolutional layer Conv, and feature transformation and channel adjustment are performed through the convolutional layer Conv. At the same time, the input and output of each deep feature enhancement module (DSWLI) are connected in residual connection.

[0054] It can be seen that the image inpainting model provided in the embodiment is closely connected between modules through residual connection, the overall process is coherent and clear, and has good scalability and migration ability.

[0055] In the specific implementation process of step 1, the shallow features are extracted through the first convolutional layer Conv in Figure 2 . Specifically, let the original image be: The initial features are extracted through the shallow convolutional network, and the expression is:

[0056] ;

[0057] In the formula, represents the initial features.

[0058] The initial features mainly retain the local texture and edge information of the image, providing support for subsequent deep modeling.

[0059] In the specific implementation process of step 2, the local and global features are mainly captured through the DSWAB sub-module.

[0060] It can be understood that since the deep feature enhancement module is a series structure of two or more, the output of the previous deep feature enhancement module is the input of the next deep feature enhancement module. Therefore, the input of the DSWAB sub-module in the first deep feature enhancement module is the initial feature , the input feature of the DSWAB sub-module in the second and subsequent deep feature enhancement modules is , and the output feature is Since the processing procedures of each module are completely consistent, for the convenience of description, the first module is taken as an example to be described in the following. In the deep layer feature enhancement module, the input feature of the DSWAB sub-module is , and the output feature is .

[0061] Specifically, when the feature is divided by the non-overlapping window, the feature is divided by different scales of non-overlapping windows by dynamically adjusting the window size, so as to obtain the first divided feature under different windows. Further, the way of dynamically adjusting the window size is to adjust the window size in a layer-by-layer increasing manner, for example, it can start from , and then increase the window by 8 pixels layer by layer each time to obtain the first divided feature under different scales.

[0062] In each window, first, the first divided feature is independently normalized to stabilize the statistical characteristics of the feature, so as to obtain the normalized first divided feature . Then, the normalized first divided feature is scored by self-attention to capture the mutual relationship between the features at different positions in the window. The specific operation is as follows:

[0063] In each window, the first query vector, the first key vector and the first value vector are respectively represented as:

[0064] ;

[0065] According to the first query vector and the first key vector , the normalized first divided feature is scored by self-attention, and the calculation expression is:

[0066] ;

[0067] Then, according to the self-attention scoring matrix , the first value vector is weighted and summed to obtain the first updated feature in the window, and the expression is:

[0068] ;

[0069] Then, the first updated feature is fused with the feature by residual fusion to obtain the first fused feature, and the expression is:

[0070] ;

[0071] In the formula, denotes the first query vector; denotes the first key vector; denotes the first value vector; , , denotes the learnable weight matrix for projecting the feature vector into the query, key, and value space; denotes the first partitioned feature after normalization in the first layer of DSWAB; denotes the first partitioned feature after normalization in the first layer of DSWAB; denotes the self-attention score matrix; denotes the dimension of the key vector used to scale the attention scores to stabilize gradient computation; denotes the first updated feature; denotes the vector transpose, where, denotes the replaceable variable.

[0072] It can be understood that in the DSWAB submodule, by dynamically adjusting the window size, a balance is achieved between local features and global features, realizing fine modeling of image features. This mechanism can extract and fuse feature information at different scales, providing richer context for the image inpainting task. The DSWAB submodule can effectively capture local and global features through dynamic window size adjustment and self-attention score calculation mechanism. In addition, the use of residual connection ensures the stable flow of gradients, improving the training efficiency and performance of the model.

[0073] Further, in order to make full use of information at different scales, by incrementally increasing the window size layer by layer, it is possible to realize context aggregation from local to larger range; among them, smaller windows help to capture fine local details, while larger windows can capture structural consistency. In this way, the model can extract and fuse feature information at different scales, thereby reducing the computational complexity while improving the expression ability of features, making the model have excellent structure recovery and detail restoration ability.

[0074] In the specific implementation process of step 3, in order to further enhance the expression ability of the features, a nonlinear transformation is performed on the first fused feature to obtain a first output feature.

[0075] Specifically, first, a linear transformation is performed on the first fused feature to extract the high-level representation of the feature. This process is implemented through a first weight matrix and a first bias vector , and the expression is:

[0076] ;

[0077] In the formula, denotes the feature after linear transformation.

[0078] To introduce nonlinearity, the GELU activation function is used in this embodiment. GELU can dynamically adjust the activation value according to the input data, and performs better when dealing with complex feature relationships. After linear transformation, the GELU activation function is applied for non-linear transformation to obtain the intermediate feature, expressed as:

[0079] ;

[0080] In the formula, represents the intermediate feature.

[0081] Then, the intermediate feature is linearly transformed again to obtain the feature representation . This process is implemented through a second weight matrix and a second bias vector , expressed as:

[0082] ;

[0083] Further, to alleviate the gradient vanishing problem in deep networks and speed up the training process, the first fused feature is residual fused with the first output feature to obtain the first output feature , expressed as:

[0084] ;

[0085] Through steps 2 and 3, the data processing process of the DSWAB sub-module is completed, and the first output feature is processed through the convolution layer Conv and then input into the LIIM sub-module for processing. The LIIM sub-module plays an important role in strengthening the dependence between image features in the local region, and can finely process the detailed information of the image.

[0086] In the specific implementation process of step 4, the LIIM sub-module uses overlapping image blocks to enhance the interaction between features, thereby more effectively capturing texture and edge information in the local region.

[0087] In the LIIM, the first output feature processed by convolution is divided into overlapping windows to obtain multiple second divided features, denoted as , where represents the number of image blocks, represents the dimension of the feature.

[0088] In each window, the second divided feature The linear transformation is performed to obtain a second query vector, a second key vector, and a second value vector, respectively, and the expressions are as follows:

[0089] ;

[0090] Next, the obtained second query vector , second key vector , and second value vector are used to calculate the feature interaction under the self-attention mechanism. Specifically, the correlation weight between features is determined according to the similarity between the second query vector and the second key vector , and then the second value vector is weighted and summed to obtain a second updated feature, and the calculation expression is as follows:

[0091] ;

[0092] The second updated feature is multi-head spliced to obtain the final second output feature, and the expression is as follows:

[0093] ;

[0094] Here, represents a multi-head self-attention mechanism, which enhances the learning ability of the model for different representation subspaces of features by calculating the self-attention mechanism in multiple different subspaces in parallel and splicing the results.

[0095] In the formula, represents the second query vector; represents the second key vector; represents the second value vector; , , represents a learnable weight matrix for linear projection of input features to meet the calculation requirements of the self-attention mechanism; represents the second divided feature; represents the second updated feature; The function is used to normalize the attention weight to the range (0, 1) of the probability distribution; represents a factor for scaling the attention weight, which aims to alleviate the problem of gradient disappearance in high-dimensional space; represents the second output feature; represents the vector transpose, where represents an alternative variable.

[0096] It can be understood that the overlapping window division can be achieved by means such as sliding window operation, 1x1 convolution + depthwise separable convolution, etc. When using the sliding window operation to achieve it, the window size can be set to K×K (K = 5, 7, 9, etc.), and then the sliding step size is set to S, where S < K (S = 3, 4, etc.). Starting from the starting position (0, 0) in the upper left corner of the feature map, in the height and width dimensions, a window of size K×K is slid with a step size of S. The feature region intercepted each time a slide is made constitutes a division window (i.e., the second divided feature), and there is an overlapping region in space between adjacent windows, and its size is (K - S) pixels.

[0097] When using the 1x1 convolution + depthwise separable convolution method, first use 1x1 convolution to adjust the number of channels or generate Q, K, V projections; then apply a K x K depthwise separable convolution with a stride = S (S < K); the depth convolution operation processes each channel independently at each spatial position, which is equivalent to calculating a correlation weight (i.e., the weight of the depth convolution) separately for each channel within a K x K sliding window. Using this operation method, although the weights are fixed, the calculation process occurs in overlapping local regions, so the overlapping window division can be achieved through this method.

[0098] It is worth noting that in the LIIM sub-module, the input and output are also connected by a residual connection. The second output feature and the first output feature after convolution processing are fused by residual to obtain the final output of the LIIM sub-module .

[0099] In the LIIM sub-module, through the division of overlapping windows, the interaction between features can be effectively enhanced within the local region, capturing more detailed image detail information, thereby providing a richer feature basis for subsequent image reconstruction. At the same time, since the LIIM sub-module only performs feature interaction within the local region, its computational complexity is significantly reduced compared to the global self-attention mechanism, enabling it to meet the requirements of lightweight models for computational efficiency while ensuring performance.

[0100] In the specific implementation process of step 5, it is stated that the second output feature is fused by residual with the initial feature . However, in the specific process of processing by the th deep feature enhancement module, after the final output of the LIIM sub-module is subjected to convolution processing, it is fused by residual with the feature to obtain the second fused feature.

[0101] In the implementation process of step 6, the iteration process is performed for two or more DSWLI modules connected in series. The second fusion feature is taken as input, and steps 2 to 5 are repeated to output the final fusion feature; the final fusion feature is residual fused with the initial feature to output the repaired image.

[0102] In the DSWLI module, higher-level structural semantics and contextual information are gradually extracted by the DSWAB sub-module. Let the input feature of the DSWAB sub-module in the i-th layer be Then the output feature can be expressed as:

[0103] ;

[0104] Among them, the residual connection structure ensures the stable propagation of the feature flow and enhances the expression ability of the model to the deep structure.

[0105] After the DSWAB sub-module, the LIIM sub-module is used to refine and enhance the local region of the first output feature After the DSWAB sub-module, the LIIM sub-module is used to refine and enhance the local region of the first output feature

[0106] ;

[0107] The DSWAB sub-module improves the model's perception of local content such as texture details and edge contours by constructing feature relationships between image blocks, especially in the repair of structure damaged areas.

[0108] Finally, the final repaired image is generated, that is:

[0109] .

[0110] The method and network structure proposed in this embodiment are equipped with standardized and nonlinear activation mechanisms, and widely use residual structures to ensure the stability of the gradient in the training process and the consistency of the image quality in the inference stage. The method proposed in the present application realizes the whole process modeling from low-level detail capture to high-level semantic recovery through multi-level feature abstraction and fusion mechanism, which not only guarantees the visual consistency and structural rationality of image repair, but also has good computational efficiency and network scalability, and is suitable for image restoration and reconstruction tasks in various real application scenarios.

[0111] ​In one embodiment, to verify the effectiveness and advancement of the image inpainting device and method proposed in the present application in real image degradation scenarios, systematic experimental evaluations are carried out on multiple public data sets, and comparisons are made with current mainstream image inpainting and super-resolution algorithms. The experiments are carried out from three dimensions of quantitative evaluation (PSNR / SSIM indicators), qualitative evaluation (visual effects) and model performance balance, to comprehensively evaluate the performance of the method in image quality, structure restoration, texture preservation and computational efficiency.

[0112] In the experiment, five standard image super-resolution evaluation data sets are selected for testing, including Set5, Set14, BSD100 (B100), Urban100 and Manga109, covering different types of natural images, urban structure images and cartoon images, etc., and having good representativeness. Two industry standards are used as evaluation indicators:

[0113] PSNR (Peak Signal-to-Noise Ratio): measures the overall pixel restoration ability of the image;

[0114] SSIM (Structural Similarity Index): measures the similarity of image structure and texture restoration.

[0115] The methods compared with the present application include traditional interpolation methods (Bicubic), classic CNN models (SRCNN, FSRCNN), high-performance deep models (EDSR, CARN, IMDN), lightweight networks (RFDN-L, SRPN-Lite), Transformer series models (SwinIR, SRFormer, HiT-SIR, IPG-Tiny) and the latest fusion structure models (SwinIR, OSFFNet, Omni-SR, etc.), a total of more than twenty kinds.

[0116] Table 1 lists the average PSNR / SSIM scores of each method on the five data sets. From Table 1, it can be observed that:

[0117] The method of the present application achieves excellent performance on all data sets, and achieves or exceeds the current best result on the five data sets of Set5, Set14, B100, Urban100 and Manga109;

[0118] For example, on the Set5 data set, the PSNR of the present method is 32.52 dB, slightly better than SwinIR-NG (32.44 dB) and HiT-SIR (32.51 dB), and the SSIM is 0.8999, which is also the highest in the table;

[0119] On the more challenging Urban100 dataset, the PSNR of the method reaches 26.83dB, which is significantly higher than SwinIR (26.47dB) and SRFormer (26.67dB), and the SSIM is 0.8101, which also surpasses all the comparative methods;

[0120] On the animation image set Manga109, the method continues to lead with 31.32 / 0.9184, fully demonstrating that the model has a strong advantage in texture restoration;

[0121] Compared with the classic models SRCNN (such as Set14 only 27.49 / 0.7503), FSRCNN and IMDN, the method of the application has a substantial improvement in all indicators, which reflects the obvious cross-generation technology advantage.

[0122] From the comprehensive average results, the method achieves a good balance between image quality (PSNR) and structural consistency (SSIM), and the advantage is more obvious on the data set with higher structural complexity (such as Urban100).

[0123] Table 1 Comparison of quantitative indicators of different algorithms

[0124]

[0125] Figure 3 And Figure 4 The visual restoration results of different methods in typical image restoration tasks are shown.

[0126] From Figure 3 (b) and Figure 4 (b) can be seen, the traditional method of Bicubic has obvious blur in the edge area of the image, and the details are seriously lost.

[0127] From Figure 3 (c) and Figure 4 (c) can be seen, the Transformer series, such as SwinIR-L, performs well in structure restoration, but some methods still have problems such as texture repetition and artifact expansion.

[0128] From Figure 3 (d) and Figure 4 (d) can be seen, the method of the application performs best in edge sharpness, texture restoration, color consistency, etc., especially in the complex structure detail area (such as building window frame, cartoon line), it shows a more realistic and natural restoration effect;

[0129] The design of the LIIM submodule enables complete recovery of fine structures in an image, such as building window lattices, human hair, and the like, and is significantly better than a model lacking local modeling capability; experiments show that the method has significant advantages in processing multi-scale fusion of texture and structure, and avoids the problems of excessive smoothing or excessive sharpening.

[0130] In addition, although some high-performance models (such as SwinIR-L and SRFormer-L) perform similarly to the present method in terms of image quality, they have large model parameter quantities and high inference delays, and are not suitable for edge devices. The method disclosed in the present application is not only suitable for high-performance server-side deployment, but can also be extended to edge computing platforms, such as intelligent cameras, mobile terminals, and other resource-constrained devices.

[0131] Although the steps in the present embodiment Figure 1 are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, Figure 1 At least some of the steps in the present embodiment may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least some of the other steps or sub-steps or stages of other steps.

[0132] Embodiment 2

[0133] Based on the image inpainting method in Embodiment 1, the present embodiment discloses an image inpainting device, as shown in Figure 5 The image inpainting device comprises an initial feature extraction module 401, a first fusion feature acquisition module 402, a first output feature acquisition module 403, a second output feature acquisition module 404, a second fusion feature acquisition module 405, and an inpainted image output module 406, wherein:

[0134] The initial feature extraction module 401 is configured to obtain an original image, perform shallow feature extraction on the original image, and obtain initial features.

[0135] The first fusion feature acquisition module 402 is configured to divide the initial features into a plurality of first divided features by non-overlapping window division; perform self-attention scoring on the first divided features in each window to obtain first updated features in the window; and perform residual fusion on the first updated features and the initial features to obtain first fusion features.

[0136] The first output feature acquisition module 403 is used to perform a nonlinear transformation on the first fused feature to obtain the first output feature;

[0137] The second output feature acquisition module 404 is used to divide the first output feature into overlapping windows and calculate the correlation weight between features within each window to obtain the second output feature.

[0138] The second fusion feature acquisition module 405 is used to perform residual fusion between the second output feature and the initial feature to output the second fusion feature;

[0139] The image restoration output module 406 is used to perform iterative processing with the second fusion feature as input, and output the final fusion feature; the final fusion feature is then residually fused with the initial feature to output the restored image.

[0140] In this embodiment, the specific working process and working principle of the initial feature extraction module 401, the first fusion feature acquisition module 402, the first output feature acquisition module 403, the second output feature acquisition module 404, the second fusion feature acquisition module 405, and the repaired image output module 406 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above unit modules.

[0141] Example 3

[0142] like Figure 6 The diagram illustrates a terminal device disclosed in this embodiment, comprising a transmitter, a receiver, a memory, and a processor. The transmitter transmits instructions and data, the receiver receives instructions and data, the memory stores computer-executed instructions, and the processor executes the computer-executed instructions stored in the memory to implement the method described in Embodiment 1 above.

[0143] It is important to note that the aforementioned memory can be either standalone or integrated with the processor. When the memory is set up independently, the terminal device also includes a bus for connecting the memory and the processor.

[0144] Example 4

[0145] This embodiment discloses a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method in Embodiment 1 above.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0148] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image inpainting method characterized by, The method comprises: Step 1, obtaining an original image, performing shallow feature extraction on the original image to obtain initial features; Step 2, performing non-overlapping window division on the initial features to obtain a plurality of first divided features; in each window, performing self-attention scoring on the first divided features to obtain first updated features within the window; and performing residual fusion on the first updated features and the initial features to obtain first fused features; Step 3, performing non-linear transformation on the first fused features to obtain first output features; Step 4, performing overlapping window division on the first output features and calculating the correlation weight between the features in each window to obtain second output features; Step 5, performing residual fusion on the second output features and the initial features to output second fused features; Step 6, taking the second fused features as input, repeating steps 2 to 5 to output final fused features; and performing residual fusion on the final fused features and the initial features to output a repaired image; In step 2, the initial features are divided into a plurality of first divided features through non-overlapping window division, comprising: The window size is dynamically adjusted, and the initial features are divided into first divided features under different window sizes through non-overlapping windows of different scales.

2. The image inpainting method of claim 1, wherein, When the window size is dynamically adjusted, the window size is adjusted in a layer-by-layer incremental manner.

3. The image inpainting method according to claim 1 or 2, characterized in that, In step 2, in each window, the first divided features are first normalized, and then the normalized first divided features are scored by self-attention to obtain first updated features within the window.

4. The image inpainting method of claim 3, wherein, In step 2, in each window, the first divided features are scored by self-attention to obtain first updated features within the window, comprising: In each window, the first query vector, the first key vector and the first value vector are represented as: ； According to the first query vector and the first key vector, the first divided features are scored by self-attention, and the expression is: ； Then, according to the self-attention scoring matrix, the first value vector is weighted and summed to obtain the first updated features within the window, and the expression is: ； In the formula, denotes the first query vector; denotes the first key vector; denotes the first value vector; denotes the first partition feature after normalization in the th DSWAB; , , denotes the learnable weight matrix; denotes the self-attention score matrix; denotes the dimension of the key vector; denotes the first updated feature; denotes the vector transpose, wherein, denotes the replaceable variable.

5. The image inpainting method of claim 4, wherein, In step 3, the first fused features are transformed by non-linear transformation to obtain first output features, comprising: The first fused features are linearly transformed according to the first weight matrix and the first bias vector, and after linear transformation, a GELU activation function is applied for non-linear transformation to obtain intermediate features; The intermediate features are linearly transformed twice based on the second weight matrix and the second bias vector to obtain the final first output features.

6. The image inpainting method of claim 1 or 2, wherein, In step 4, the first output features are divided into overlapping windows, and the correlation weight between the features in each window is calculated to obtain second output features, comprising: The first output features are divided into overlapping windows to obtain a plurality of second divided features; In each window, the second divided features are linearly transformed to obtain a second query vector, a second key vector and a second value vector, and the expressions are respectively: ； According to the second query vector and the second key vector, the second divided feature is subjected to correlation weight calculation, and then the second value vector is subjected to weighted summation to obtain a second updated feature, and the calculation expression is: ； The second updated feature is subjected to multi-head splicing to obtain a final second output feature, and the expression is: ； wherein denotes a second query vector; denotes a second key vector; denotes a second value vector; denotes a learnable weight matrix; denotes a second partition feature; denotes a second update feature; denotes a factor for scaling attention weights; denotes a second output feature; denotes a vector transpose, wherein denotes an alternative variable.​​ 7. An image inpainting apparatus characterized by comprising: The image inpainting method of any one of claims 1 to 6, wherein the device comprises: An initial feature extraction module configured to acquire an original image, perform shallow feature extraction on the original image, and obtain an initial feature; A first fused feature acquisition module configured to divide the initial feature into a plurality of first divided features in a non-overlapping window manner, perform self-attention scoring on the first divided features in each window, and obtain a first updated feature in the window; and perform residual fusion on the first updated feature and the initial feature to obtain a first fused feature; A first output feature acquisition module configured to perform non-linear transformation on the first fused feature to obtain a first output feature; A second output feature acquisition module configured to divide the first output feature into a plurality of second output features in an overlapping window manner, and calculate correlation weights between features in each window to obtain a second output feature; A second fused feature acquisition module configured to perform residual fusion on the second output feature and the initial feature to output a second fused feature; An inpainted image output module configured to perform iterative processing on the second fused feature as input to output a final fused feature, and perform residual fusion on the final fused feature and the initial feature to output an inpainted image.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the image inpainting method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the image inpainting method of any one of claims 1 to 6.

Images