High-resolution image inpainting method, device, and storage medium

Abstract

The application discloses a high-resolution image repairing method and device and a storage medium, relates to the technical field of computer vision, and comprises the following steps: performing structural feature extraction on a damaged image to obtain a corresponding edge graph and a line frame graph; determining a multi-channel input tensor based on the damaged image, the edge graph, the line frame graph and a binary mask graph; inputting the multi-channel input tensor into a high-resolution structure repairing network to generate a high-resolution edge graph and a high-resolution line frame graph; performing fusion processing on the high-resolution edge graph, the high-resolution line frame graph, a mask image and the binary mask graph to obtain a structural visual fusion feature graph; inputting the structural visual fusion feature graph into a structure-enhanced texture repairing network to generate a high-resolution enhanced feature graph, and generating a high-resolution repaired image according to the high-resolution enhanced feature graph, so that the structural reconstruction capability of high-resolution repairing is improved, the texture repairing quality is enhanced, and a repaired result with reasonable structure and consistent texture semantics is generated.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a high-resolution image restoration method, device, and storage medium. Background Technology

[0002] Image inpainting utilizes existing information to reconstruct a credible image structure and content to a certain extent. In recent years, image inpainting technology has become an important research area in image processing and has wide applications, such as damaged image restoration, target removal, image retouching, and text removal. To generate realistic results, the restored image should maintain the consistency of the neighborhood of the missing region and the image's texture structure. Despite significant progress in image inpainting technology in recent years, the increasing demand for high-quality, high-resolution images due to advancements in photographic equipment and display technology makes high-resolution image inpainting a considerable challenge.

[0003] High-resolution images typically contain rich details and intricate geometric structures (such as continuous edges, coherent lines, and complex textures). However, traditional convolutional neural networks, due to their limited local receptive fields, are prone to structural breaks, misalignments, or discontinuities between the generated content and the surrounding known regions when processing large-scale missing or complex structural repairs. Although subsequent research introduced the Transformer architecture to capture long-range dependencies, its self-attention mechanism still tends to model low-order spatial relationships in deep feature interactions and cannot fully simulate high-order nonlinear relationships between pixels. This often results in distortion or falsification of macroscopic structures such as repaired edge contours and object shapes, severely damaging the visual realism and spatial coherence of the image.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a high-resolution image restoration method, device, and storage medium, aiming to solve the technical problem that it is difficult to maintain the overall structural integrity of the image during the high-resolution image restoration process in the prior art.

[0006] To achieve the above objectives, this application proposes a high-resolution image restoration method, which includes:

[0007] A low-resolution damaged image and a corresponding binary mask image are obtained. Structural features are extracted from the damaged image to obtain the corresponding edge image and wireframe image.

[0008] The damaged image, the edge image, and the wireframe image are masked using the binary mask image, and the corresponding mask results are channel-concatenated with the binary mask image to form a multi-channel input tensor.

[0009] The multi-channel input tensor is input into a high-resolution structure repair network, and the edge map and wireframe map are repaired by the high-resolution structure repair network to generate a high-resolution edge map and a high-resolution wireframe map.

[0010] The high-resolution edge map, the high-resolution wireframe map, the mask image, and the binary mask image are fused to obtain the corresponding structural visual fusion feature map.

[0011] The structural visual fusion feature map is input into the structural enhancement texture restoration network, and the structural visual fusion feature map is enhanced by the structural enhancement texture restoration network to generate a high-resolution enhanced feature map.

[0012] A high-resolution repair image corresponding to the damaged image is generated based on the high-resolution enhanced feature map.

[0013] In one embodiment, the step of inputting the multi-channel input tensor into a high-resolution structure repair network, and performing structure repair on the edge map and wireframe map through the high-resolution structure repair network to generate a high-resolution edge map and a high-resolution wireframe map includes:

[0014] The multi-channel input tensor is subjected to convolutional downsampling to obtain the corresponding first downsampled feature map;

[0015] The first downsampled feature map is input into the efficient Transformer block in the high-resolution structure repair network. The efficient Transformer block performs high-order spatial interaction and feature enhancement on the first downsampled feature map to obtain the corresponding structure enhancement feature map. The efficient Transformer block is constructed based on recursive gated convolution and multi-head interactive attention mechanism.

[0016] The structure enhancement feature map is subjected to transposed convolution upsampling processing to generate a corresponding high-resolution structure sketch space, which includes the high-resolution edge map and the high-resolution wireframe map.

[0017] In one embodiment, the step of inputting the first downsampled feature map into the efficient Transformer block in the high-resolution structure repair network, and obtaining the corresponding structure-enhanced feature map through the high-order spatial interaction and feature enhancement of the first downsampled feature map by the efficient Transformer block includes:

[0018] The first downsampled feature map is normalized to obtain the corresponding first normalized feature map;

[0019] Perform a recursive gated convolution operation on the first normalized feature map to obtain the corresponding spatial blending feature map;

[0020] The spatial hybrid feature map is transformed by the first feedforward neural network to obtain the corresponding initial structure enhancement feature map. The initial structure enhancement feature map and the first downsampled feature map are then added to obtain the corresponding first structure fusion feature map.

[0021] The first structure fusion feature map is input into the hybrid attention module in the efficient Transformer block, and the first structure fusion feature map is normalized by the normalization layer in the hybrid attention module to obtain the corresponding normalized structure fusion feature map.

[0022] The normalized structure fusion feature map is subjected to structural feature enhancement processing through the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head attention processing result; and the normalized structure fusion feature map is subjected to global semantic extraction through the full attention branch in the hybrid attention module to obtain the corresponding full attention processing result.

[0023] The full attention processing result, the multi-head attention processing result, and the first structure fusion feature map are added together to obtain the corresponding second structure fusion feature map;

[0024] The second structure fusion feature map is transformed by the second feedforward neural network to obtain the corresponding intermediate structure enhancement feature map. The intermediate structure enhancement feature map and the second structure fusion feature map are then added together to obtain the structure enhancement feature map.

[0025] In one embodiment, the step of performing structural feature enhancement processing on the normalized structural fusion feature map through the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head attention processing result includes:

[0026] The normalized structure fusion feature map is segmented along the channel dimension to obtain the sub-features corresponding to each attention head, and each sub-feature is linearly transformed to generate the corresponding query tensor, key tensor and value tensor.

[0027] Based on the binary mask image, the query tensor, the key tensor, and the value tensor are respectively divided into known region features and masked region features;

[0028] Calculate the first global feature vector corresponding to the known region feature and the second global feature vector corresponding to the masked region feature respectively, and multiply the first global feature vector and the second global feature vector element by element to obtain the known region enhancement feature and the masked region enhancement feature;

[0029] Based on the known region enhancement features and the masked region enhancement features, the SENet network is used to dynamically predict the weights and obtain the corresponding weight coefficients.

[0030] Based on the weighting coefficients, the enhanced features of the known region and the enhanced features of the masked region are weighted and fused to obtain the corresponding weighted fusion result;

[0031] The known region features and the weighted fusion result are combined to obtain the output features of each attention head;

[0032] The output features of each attention head are concatenated along the channel dimension to obtain the multi-head attention processing result.

[0033] In one embodiment, the step of inputting the structural visual fusion feature map into a structural enhancement texture inpainting network, and performing feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture inpainting network to generate a high-resolution enhanced feature map includes:

[0034] The structural visual fusion feature map is subjected to convolutional downsampling processing to obtain the corresponding second downsampling feature map;

[0035] The second downsampled feature map is input into the global-local Transformer block of the structure-enhanced texture restoration network. The global-local Transformer block performs global-local feature enhancement on the second downsampled feature map to obtain the corresponding texture enhancement feature map.

[0036] The texture enhancement feature map is subjected to convolutional upsampling to generate the high-resolution enhancement feature map.

[0037] In one embodiment, the step of inputting the second downsampled feature map into the global-local Transformer block of the structure-enhanced texture inpainting network, and performing global-local feature enhancement on the second downsampled feature map through the global-local Transformer block to obtain the corresponding texture-enhanced feature map includes:

[0038] The second downsampled feature map is normalized to obtain the corresponding second normalized feature map;

[0039] A recursive gated convolution operation is performed on the second normalized feature map to obtain the corresponding initial texture enhancement feature map. The initial texture enhancement feature map and the second downsampled feature map are then added to obtain the corresponding first texture fusion feature map.

[0040] The first texture fusion feature map is normalized to obtain the corresponding normalized texture fusion feature map;

[0041] The normalized texture fusion feature map is input into the global-local fusion module of the global-local Transformer block. Global features are extracted through the global branch of the global-local fusion module and filtered by the gating unit. Local detail features are extracted through the local branch of the global-local fusion module. The filtered global features are combined with the local features to obtain the intermediate texture enhancement feature map.

[0042] The intermediate texture enhancement feature map and the first texture fusion feature map are added together to obtain the corresponding second texture fusion feature map;

[0043] The second texture fusion feature map is transformed by a third feedforward neural network, and the corresponding feature transformation result is added to the second texture fusion feature map to obtain the texture enhancement feature map.

[0044] In one embodiment, before the step of extracting structural features from the damaged image to obtain the corresponding edge map and wireframe map, the method further includes:

[0045] Obtain a training set, which includes multiple training samples and labels for each training sample. The labels for each training sample include a high-resolution target image corresponding to the training sample, a high-resolution target edge map corresponding to the training sample, and a high-resolution target wireframe map corresponding to the training sample. The training samples are low-resolution training images and training binary mask maps corresponding to the training images.

[0046] The training image is input into the structure extraction network in the high-resolution image restoration model to be trained. The structure extraction network extracts structural features from the training image to obtain the corresponding training edge map and training wireframe map.

[0047] The training image, the training edge map, and the training wireframe map are masked using the training binary mask map to obtain the corresponding training mask image, training mask edge map, and training mask wireframe map.

[0048] The training mask image, training binary mask image, training mask edge image and training mask wireframe image are channel-concatenated to form a training multi-channel input tensor.

[0049] The trained multi-channel input tensor is input into the high-resolution structural repair network in the high-resolution image repair model. The high-resolution structural repair network is used to repair the edge map and wireframe map, generating a high-resolution edge prediction map and a high-resolution wireframe prediction map.

[0050] Based on the high-resolution edge prediction map and the high-resolution target edge map, calculate the first loss value corresponding to the cross-entropy loss function, and based on the high-resolution wireframe prediction map and the high-resolution target wireframe map, calculate the second loss value corresponding to the cross-entropy loss function.

[0051] The first-stage loss value of the high-resolution image inpainting model is calculated based on the first loss value and the second loss value, and the parameters of the high-resolution structural inpainting network are updated based on the first-stage loss value.

[0052] The training image is input into the high-resolution image restoration model, and the low-resolution training image is restored by inputting the training image into the high-resolution image restoration model to be trained, so as to obtain the high-resolution prediction image corresponding to the training image.

[0053] Based on the high-resolution predicted image and the high-resolution target image, the second-stage loss value of the high-resolution image inpainting model is determined, and the parameters of the structure-enhanced texture inpainting network are updated based on the second-stage loss value.

[0054] When the loss value in the second stage is less than or equal to a preset value, the high-resolution image restoration model that has been trained is obtained.

[0055] In one embodiment, the step of determining the second-stage loss value of the high-resolution image inpainting model based on the high-resolution predicted image and the high-resolution target image includes:

[0056] Calculate the L1 norm loss between the high-resolution predicted image and the high-resolution target image to obtain the pixel reconstruction loss value;

[0057] The high-resolution predicted image and the high-resolution target image are respectively input into the discriminator, and the discriminator outputs the corresponding true or false judgment result. Based on the true or false judgment result, the discriminator loss and generator loss are calculated, and a gradient penalty term is added to obtain the adversarial loss value.

[0058] Obtain the output feature maps of the discriminator for the high-resolution predicted image and the high-resolution target image in multiple intermediate activation layers, and calculate the L1 norm loss between the output feature maps of the corresponding layers as the feature matching loss value;

[0059] The high-resolution predicted image and the high-resolution target image are respectively input into a pre-trained residual network with an expanded convolutional structure to obtain the corresponding first high-level semantic feature map and second high-level semantic feature map.

[0060] The feature similarity between the first high-level semantic feature map and the second high-level semantic feature map is calculated and used as the high receptive field perception loss value.

[0061] The second-stage loss value is obtained by weighted summing of the pixel reconstruction loss value, the adversarial loss value, the feature matching loss value, and the high receptive field perception loss value.

[0062] Furthermore, to achieve the above objectives, this application also proposes a high-resolution image restoration apparatus, which includes:

[0063] The structure extraction module is used to acquire a low-resolution damaged image and a binary mask image corresponding to the damaged image, and to extract structural features from the damaged image to obtain the corresponding edge image and wireframe image.

[0064] A multi-channel cascade module is used to perform masking processing on the damaged image, the edge image, and the wireframe image respectively using the binary mask image, and to cascade the corresponding masking results with the binary mask image to form a multi-channel input tensor;

[0065] The structure repair module is used to input the multi-channel input tensor into a high-resolution structure repair network, and perform structure repair on the edge map and wireframe map through the high-resolution structure repair network to generate a high-resolution edge map and a high-resolution wireframe map.

[0066] The feature fusion module is used to fuse the high-resolution edge map, the high-resolution wireframe map, the mask image, and the binary mask image to obtain the corresponding structural visual fusion feature map.

[0067] The texture restoration module is used to input the structural visual fusion feature map into the structural enhancement texture restoration network, and perform feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture restoration network to generate a high-resolution enhanced feature map.

[0068] The feature decoding module is used to generate a high-resolution repair image corresponding to the damaged image based on the high-resolution enhanced feature map.

[0069] In addition, to achieve the above objectives, this application also proposes a high-resolution image restoration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the high-resolution image restoration method as described above.

[0070] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the high-resolution image restoration method described above.

[0071] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the high-resolution image restoration method described above.

[0072] The present application proposes one or more technical solutions, which have at least the following technical effects: By acquiring a low-resolution damaged image and its corresponding binary mask, structural features are extracted from the damaged image to obtain corresponding edge maps and wireframes. The damaged image, edge map, and wireframe are masked using the binary mask, and the corresponding masking results are concatenated with the binary mask to form a multi-channel input tensor, providing more comprehensive structural information about the image to be repaired. This multi-channel input tensor is then input into a high-resolution structural repair network, which performs structural repair on the edge map and wireframe, generating high-resolution edge maps and high-resolution wireframes. This provides prior information for subsequent image repair tasks, ensuring that the repaired image maintains structural integrity and accuracy. Next, the high-resolution edge map, high-resolution wireframe, mask image, and binary mask are fused to obtain a corresponding structural visual fusion feature map, enhancing the expressiveness of the image structure and providing a good foundation for subsequent texture repair. The structural visual fusion feature map is input into the structural enhancement texture restoration network. The network then enhances the feature map, effectively restoring texture details in the missing areas and ensuring consistency between texture and structure during restoration. This generates a high-resolution enhanced feature map, which is then used to generate a high-resolution restored image corresponding to the damaged image. This solves the technical problem of maintaining the overall structural integrity of the image during high-resolution image restoration in existing technologies, improves the structural reconstruction capability of high-resolution restoration, enhances texture restoration quality, and generates restoration results with reasonable structure and consistent texture semantics. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart illustrating an embodiment of the high-resolution image restoration method of this application.

[0076] Figure 2 This is a schematic diagram of the structure of a high-resolution image restoration model provided in this application;

[0077] Figure 3 This application provides a schematic diagram of the structure of an efficient Transformer block;

[0078] Figure 4 This application provides a schematic diagram of the structure of a multi-head interactive attention branch in a hybrid attention module;

[0079] Figure 5 A schematic diagram of a global-local Transformer block is provided for this application;

[0080] Figure 6 This is a schematic diagram of the module structure of the high-resolution image restoration device according to an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the high-resolution image restoration method in this application embodiment. Detailed Implementation

[0082] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0083] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0084] The main solution of this application embodiment is as follows: A low-resolution damaged image and its corresponding binary mask image are obtained; structural features are extracted from the damaged image to obtain corresponding edge images and wireframe images; the damaged image, edge images, and wireframe images are masked using the binary mask image, and the corresponding masking results are concatenated with the binary mask image to form a multi-channel input tensor; the multi-channel input tensor is input into a high-resolution structure restoration network, which performs structure restoration on the edge images and wireframe images to generate high-resolution edge images and high-resolution wireframe images; the high-resolution edge images, high-resolution wireframe images, mask images, and binary mask images are fused to obtain corresponding structural visual fusion feature maps; the structural visual fusion feature maps are input into a structure enhancement texture restoration network, which performs feature enhancement on the structural visual fusion feature maps to generate high-resolution enhancement feature maps; and a high-resolution restored image corresponding to the damaged image is generated based on the high-resolution enhancement feature maps.

[0085] In this embodiment, for ease of description, the following description uses a high-resolution image restoration system as the execution subject.

[0086] Image inpainting utilizes existing information to reconstruct a believable image structure and content to a certain extent. In recent years, image inpainting technology has become an important research area in image processing and has wide applications, such as damaged image restoration, target removal, image retouching, and text removal. To generate realistic results, the inpainted image should maintain consistency in texture and structure between the missing area and its neighborhood. Despite significant progress, with advancements in photographic equipment and display technology, users' demand for high-quality, high-resolution images continues to increase, making high-resolution image inpainting a considerable challenge. Traditional diffusion-based and sample-based image inpainting methods still have limited ability to restore good texture and overall structure in high-resolution images.

[0087] The Transformer has demonstrated significant performance in natural language processing, with its self-attention mechanism better capturing long-term dependencies between inputs and effectively promoting the learning of global feature representations. Inspired by the success of the Transformer in natural language processing, we have applied the Visual Transformer and its variants to visual tasks and achieved some results. However, the Visual Transformer still faces some challenges in high-resolution image inpainting. First, while the Transformer can capture long-range dependencies and model global contextual information, the self-attention mechanism used by the Transformer model, when processing deep features, can usually only capture low-order feature space interactions and cannot fully capture the correlations between high-order features, leading to distortion or artifacts in image inpainting. Second, high-resolution images typically contain more details and complex structures, making the inpainting process more complex. Existing image inpainting methods struggle to effectively maintain the overall structural integrity of images when processing these images, resulting in broken and discontinuous edges and texture connections, severely impacting the realism and visual coherence of the image.

[0088] This application provides a solution that extracts structural features from a low-resolution damaged image and its corresponding binary mask image to obtain edge and wireframe images. The damaged image, edge image, and wireframe image are then masked using the binary mask image, and the corresponding masking results are concatenated with the binary mask image to form a multi-channel input tensor, providing more comprehensive structural information about the image to be repaired. This multi-channel input tensor is then fed into a high-resolution structural repair network, which performs structural repair on the edge and wireframe images, generating high-resolution edge and wireframe images. This provides prior information for subsequent image repair tasks, ensuring that the repaired image maintains structural integrity and accuracy. Finally, the high-resolution edge image, high-resolution wireframe image, mask image, and binary mask image are fused to obtain a corresponding structural visual fusion feature map, enhancing the expressiveness of the image structure and providing a good foundation for subsequent texture repair. The structural visual fusion feature map is input into the structural enhancement texture restoration network. The network then enhances the feature map, effectively restoring texture details in the missing areas and ensuring consistency between texture and structure during restoration. This generates a high-resolution enhanced feature map, which is then used to generate a high-resolution restored image corresponding to the damaged image. This solves the technical problem of maintaining the overall structural integrity of the image during high-resolution image restoration in existing technologies, improves the structural reconstruction capability of high-resolution restoration, enhances texture restoration quality, and generates restoration results with reasonable structure and consistent texture semantics.

[0089] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a high-resolution image restoration device capable of performing the above functions. The following description uses a high-resolution image restoration system as an example to illustrate this embodiment and the subsequent embodiments.

[0090] Based on this, embodiments of this application provide a high-resolution image restoration method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the high-resolution image restoration method of this application.

[0091] In this embodiment, the high-resolution image restoration method includes steps 101-106:

[0092] Step 101: Obtain the low-resolution damaged image and the corresponding binary mask image of the damaged image, extract the structural features of the damaged image, and obtain the corresponding edge image and wireframe image.

[0093] Specifically, the low-resolution damaged image refers to the input image to be repaired. It has a low resolution and contains missing areas (such as occlusions, scratches, or text coverage), serving as the original visual input for the repair task. The binary mask is a single-channel image of the same size as the damaged image, used to accurately identify missing areas: a pixel value of 0 in the binary mask represents a missing / repairable area, while a pixel value of 1 represents a known / reliable area. The edge map is a single-channel binarized image that retains only the detailed edges of objects in the image, such as clothing folds, leaf veins, and gaps between objects, highlighting the transition boundaries of local structures. The wireframe is a single-channel binarized image that retains the core outlines of objects in the image, such as building borders, the overall outline of a person, and the main boundaries of a scene, reflecting the overall structural framework of the image.

[0094] In some embodiments, a low-resolution damaged image and a corresponding binary mask image are first obtained. The damaged image typically contains missing or damaged areas and has a low resolution, such as 256×256 pixels, with limited detail information. The binary mask image is used to accurately mark the damaged location. The damaged image is input into the high-resolution image restoration model provided in this application. The structural feature extraction network of the high-resolution image restoration model is used to extract structural features from the damaged image to obtain key geometric prior information of the damaged image. For wireframe extraction, the damaged image can first be converted into a single-channel grayscale image to eliminate color interference. Then, the LSM-HAMP algorithm is used to analyze the grayscale value changes pixel by pixel to identify the core contours of objects, such as building frames and human torso contours. Weak signals such as textures are filtered by adaptive thresholding to generate a low-resolution wireframe image that retains only strong contour lines. For edge image extraction, the damaged image can first be preprocessed by multi-scale scaling to enhance edge signals at different scales. Then, the multi-scale convolution kernel of the DexiNed algorithm is used to capture detailed edges. After fusing the multi-scale detection results, non-maximum suppression is performed to eliminate redundancy and generate a low-resolution edge image containing complete detailed edges. By extracting structural skeletons such as edge maps and wireframes from complex, low-resolution damaged images, the implicit geometric structures in the damaged images are made explicit, forming a sketch space that provides strong guidance for subsequent high-resolution structural repair and texture generation.

[0095] Step 102: Perform masking processing on the damaged image, edge image and wireframe image respectively using a binary mask image, and concatenate the corresponding masking results with the binary mask image to form a multi-channel input tensor.

[0096] Optionally, the steps of masking the damaged image, edge image, and wireframe image separately using a binary mask image, and then concatenating the corresponding mask results with the binary mask image to form a multi-channel input tensor include:

[0097] The damaged image, edge image, and wireframe image are masked using a binary mask image to obtain the corresponding mask image, mask edge image, and mask wireframe image.

[0098] The mask image, binary mask image, mask edge image, and mask wireframe image are concatenated along the channel dimension to obtain a multi-channel input tensor.

[0099] Specifically, the mask image is the image obtained after masking the damaged image. Known areas retain their original visual pixels (such as color and texture), while pixels in damaged areas are set to 0, simultaneously carrying visual information and damage location markers. The mask edge map and mask wireframe map are images obtained after masking the edge map and wireframe map, respectively, retaining only the edge / contour information of known areas and clearly presenting the structural deficiencies in the damaged areas. The multi-channel input tensor is the final output after channel cascading, containing a multi-dimensional feature matrix of "visual information + structural information + damage location markers," which serves as the input data for the subsequent high-resolution structural repair network.

[0100] In some embodiments, a binary mask is used to mask the damaged image, edge map, and wireframe image respectively: that is, the damaged image, edge map, and wireframe image are multiplied pixel by pixel with the binary mask image, and the pixel values ​​of the missing areas are forced to zero, thereby obtaining the mask image, mask edge map, and mask wireframe image. This ensures that all input information contains only content from reliable areas and avoids information leakage. Next, the mask image, mask edge map, and mask wireframe image are concatenated with the original binary mask image in the channel dimension. If the damaged image is RGB three-channel, a 6-channel multi-channel input tensor is finally formed (3-channel mask image + 1-channel binary mask image + 1-channel mask edge map + 1-channel mask wireframe image). This multi-channel input tensor serves as the sole input to the high-resolution structural repair network, fully integrating three types of key information: visual content, spatial location constraints, and geometric structure priors, so that the visual, structural, and damaged location information correspond precisely in the same spatial location. By using unified masking and multi-source information channel fusion, a multi-channel input tensor that contains reliable visual content and embeds strong geometric priors and precise spatial constraints is constructed. This helps the high-resolution structural repair network generate repair results with continuous edges, complete wireframes, and reasonable structures based on a full understanding of the context.

[0101] Step 103: Input the multi-channel input tensor into the high-resolution structure repair network, and repair the edge map and wireframe map through the high-resolution structure repair network to generate high-resolution edge map and high-resolution wireframe map.

[0102] Specifically, in terms of structural restoration, the high-resolution structural restoration network provided in this application employs recursive gated convolution (gnConv) operations, which can utilize high-order spatial interactions to capture global feature dependencies, avoiding the secondary complexity increase brought about by self-attention mechanisms. To optimize the restoration effect, the high-resolution structural restoration network also adopts a multi-head interactive attention mechanism (HIA), which can fuse information from known regions and occluded regions, thereby helping to promote the generation of structurally consistent outputs by the high-resolution image restoration model. The high-resolution image restoration model includes the high-resolution structural restoration network.

[0103] Optionally, the steps of inputting a multi-channel input tensor into a high-resolution structure repair network, and performing structure repair on the edge map and wireframe map through the high-resolution structure repair network to generate high-resolution edge maps and high-resolution wireframe maps include:

[0104] The multi-channel input tensor is subjected to convolutional downsampling to obtain the corresponding first downsampled feature map;

[0105] The first downsampled feature map is input into the efficient Transformer block in the high-resolution structure repair network. The efficient Transformer block performs high-order spatial interaction and feature enhancement on the first downsampled feature map to obtain the corresponding structure-enhanced feature map. The efficient Transformer block is constructed based on recursive gated convolution and multi-head interactive attention mechanism.

[0106] The structure enhancement feature map is subjected to transposed convolution upsampling to generate a corresponding high-resolution structure sketch space, which includes a high-resolution edge map and a high-resolution wireframe map.

[0107] Specifically, the efficient Transformer block is the core module of the high-resolution structural repair network. Unlike the traditional self-attention Transformer, it is built upon recursive gated convolution (gnConv) and multi-head interactive attention (HIA) mechanisms. Recursive gated convolution helps model higher-order spatial interactions, while HIA explicitly models the structural dependencies between known and missing regions. Utilizing the efficient Transformer block, both global modeling capabilities and computational efficiency can be achieved, making it suitable for high-resolution tasks, capturing higher-order spatial interactions, and enhancing structural feature representation.

[0108] For example, to achieve accurate reconstruction of the missing region structure, a multi-channel input tensor (including a mask image, a binary mask image, a mask edge image, and a mask wireframe image) needs to be input into a high-resolution structure restoration network. The high-resolution structure restoration network first downsamples the multi-channel input tensor through several layers (e.g., three layers) of convolutional operations with strides to extract high-level semantic features, obtaining a first downsampled feature map. Subsequently, this first downsampled feature map is fed into the core module of the high-resolution structure restoration network—a four-layer stacked efficient Transformer block. The efficient Transformer block integrates recursively gated convolution (gnConv) and a multi-head interactive attention mechanism (HIA). The recursively gated convolution can model high-order spatial interactions through recursive unrolling to capture complex geometric relationships, while the multi-head interactive attention mechanism can use binary masks to distinguish between known and missing regions, dynamically fusing the features of both to enhance structural consistency, thereby outputting a structure-enhanced feature map with strong geometric constraints and rich semantics. Finally, the enhanced feature map is upsampled using four layers of transposed convolution to gradually restore it to the target high-resolution scale (e.g., 512×512), generating a dual-channel high-resolution structural sketch space. The first channel is a high-resolution edge map used to depict fine contours, and the second channel is a high-resolution wireframe map used to represent the global backbone structure. By using an efficient Transformer block to overcome the limitations of traditional low-order feature interaction, and balancing restoration accuracy and computational efficiency, a high-resolution structural prior with complete details and coherent logic is reconstructed based on low-resolution input. This provides reliable geometric guidance for subsequent texture restoration and helps solve common structural distortion problems in high-resolution image restoration, such as edge breakage and wireframe distortion.

[0109] Step 104: The high-resolution edge map, high-resolution wireframe map, mask image, and binary mask image are fused to obtain the corresponding structural visual fusion feature map.

[0110] Optionally, the steps of fusing the high-resolution edge map, high-resolution wireframe map, mask image, and binary mask image to obtain the corresponding structural visual fusion feature map include:

[0111] Upsampling is performed on the mask image and the binary mask image respectively to obtain the upsampled mask image and the upsampled binary image;

[0112] Convolutional processing is performed on the high-resolution edge map and the high-resolution wireframe map respectively to obtain the corresponding first deep structure feature map and second deep structure feature map;

[0113] The upsampled mask image and the upsampled binary image are convolved to obtain the corresponding first initial visual feature map and second initial visual feature map.

[0114] The first deep structural feature map, the second deep structural feature map, the first initial visual feature map, and the second initial visual feature map are added together to obtain a structural visual fusion feature map.

[0115] Specifically, the high-resolution edge map and high-resolution wireframe map are dual-channel structural sketches output by the high-resolution structural inpainting network, respectively depicting fine contours and the global skeleton, serving as the core carriers of structural priors. Additive processing refers to adding feature maps of the same size element-wise, achieving a tight coupling and fusion of structural and visual information, rather than simple splicing, which is beneficial for gradient flow and feature interaction. The structural-visual fusion feature map is a unified feature map that fuses deep structural features and initial visual features, possessing both structural coherence and visual reference information, providing the basic input for subsequent texture enhancement.

[0116] In some embodiments, to achieve effective collaboration between structural priors and visual content, deep fusion of multi-source high-resolution information (i.e., high-resolution edge maps, high-resolution wireframe maps, mask images, and binary mask images) is required. Specifically, firstly, the low-resolution mask image and binary mask image are upsampled using bilinear interpolation or lightweight transpose convolution to align their spatial resolution with the high-resolution edge map and high-resolution wireframe map, resulting in an upsampled mask image and an upsampled binary mask image. Subsequently, a 1×1 convolution operation is performed on the high-resolution edge map and high-resolution wireframe map, mapping them to a unified feature channel dimension (e.g., 64-dimensional), generating a first deep structural feature map and a second deep structural feature map to extract refined geometric semantics. Simultaneously, convolution operations are performed on the upsampled mask image and the upsampled binary mask image to extract a first initial visual feature map and a second initial visual feature map containing the original visual content and spatial location constraints, respectively. Finally, the four sets of feature maps with the same spatial size and number of channels (first deep structural feature map, second deep structural feature map, first initial visual feature map, and second initial visual feature map) are fused element-wise to achieve deep fusion of deep structural features and initial visual features, ultimately resulting in a structural-visual fusion feature map with the same number of channels and spatial size as the high-resolution image. Through the feature fusion process of upsampling alignment, convolutional mapping, and feature addition, the complete high-resolution structural features (edges, wireframes) are deeply fused with visual features and damage location markers, ensuring that the subsequent texture restoration process both follows the structural framework and references the visual style of known regions, avoiding a disconnect between texture and structure. Simultaneously, it avoids parameter inflation caused by channel splicing. The tightly coupled feature element addition operation promotes deep interaction between structural guidance and visual context at the feature level, thereby constructing a unified feature representation that contains high-fidelity edge and wireframe geometric constraints while retaining the true texture of known regions and the location information of missing regions. This provides an input foundation for the subsequent structure-enhanced texture restoration network that combines "structural rationality" and "visual realism."

[0117] Step 105: Input the structural visual fusion feature map into the structural enhancement texture restoration network, and perform feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture restoration network to generate a high-resolution enhanced feature map.

[0118] Specifically, the high-resolution image inpainting model also includes a structure-enhanced texture inpainting network. In the high-resolution structure reconstruction process, the reconstructed high-resolution structure maps (i.e., high-resolution edge maps and high-resolution wireframe maps) are used as a sketch space, providing prior information for subsequent image inpainting tasks. For texture inpainting, a structure-enhanced texture inpainting network is introduced, employing a global-local controlled Transformer structure to selectively extract and fuse global and local features, thereby effectively restoring texture details in missing areas. By extracting structural features from the sketch space, the structure-enhanced texture inpainting network further optimizes the generation process, ensuring consistency between texture and structure during the inpainting process.

[0119] Optionally, the steps of inputting the structural visual fusion feature map into the structural enhancement texture inpainting network, and performing feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture inpainting network to generate a high-resolution enhanced feature map include:

[0120] The structural visual fusion feature map is subjected to convolutional downsampling to obtain the corresponding second downsampling feature map;

[0121] The second downsampled feature map is input into the global-local Transformer block of the structure-enhanced texture restoration network. The global-local Transformer block performs global-local feature enhancement on the second downsampled feature map to obtain the corresponding texture enhancement feature map.

[0122] Convolutional upsampling is performed on the texture enhancement feature map to generate a high-resolution enhancement feature map.

[0123] Specifically, the Global-Local Transformer block is the core feature processing unit in the structure-enhanced texture inpainting network. Built upon HorBlock and GateBlock, it combines the high-order spatial interaction capabilities of recursively gated convolutions with a global-local feature filtering and fusion mechanism. The Global-Local Transformer block leverages the self-attention mechanism in the Transformer architecture to effectively capture long-range dependencies (i.e., global features) and fine-grained structural information (i.e., local features) within the feature map. Global features help understand the overall layout and contextual information, while local features focus more on the accurate representation of details, edges, and other subtle structures. By processing the second downsampled feature map through the Global-Local Transformer block, its expressiveness at different scales can be enhanced, resulting in a texture-enhanced feature map.

[0124] In some embodiments, to achieve high-quality texture reconstruction guided by structure, the structural visual fusion feature map needs to be input into a structure-enhanced texture inpainting network for deep feature enhancement. The specific implementation is as follows: First, the structural visual fusion feature map is downsampled using several layers (3 layers) of convolutional operations with stride (e.g., 3×3 convolutions with stride=2). This reduces the spatial dimension while extracting high-level semantic features, resulting in a second downsampled feature map. Then, this second downsampled feature map is fed into the core module of the structure-enhanced texture inpainting network—a four-layer stacked global-local Transformer block. The global-local Transformer block first establishes long-range relationships through recursive operations using HorBlock to achieve high-order spatial interaction. Then, it extracts and filters global features through the global branch of GateBlock, while the local branch uses convolution to extract local detail information. The global and local features are then fused to form a texture enhancement feature map that combines global context awareness with local detail preservation. Finally, three consecutive transposed convolutional layers with a stride of 2 are used to upsample the texture enhancement feature map. After each transposed convolution, the spatial size is doubled and the number of channels is halved. Interspersed convolutional layers refine the texture details, ultimately restoring it to the preset high-resolution size and generating a high-resolution enhanced feature map. Through the process of efficient downsampling, global-local collaborative enhancement, and accurate upsampling restoration, based on the known high-fidelity structural priors (edges and wireframes), global-local collaborative modeling is used to achieve semantically reasonable, detailed, and natural boundary reconstruction of the missing texture region. This balances computational efficiency and restoration accuracy, ensuring consistency between texture and structural framework, and guaranteeing the unity of local details and global style. It specifically addresses the pain points of texture blurring and structural disconnect in high-resolution image restoration in existing technologies, providing high-quality feature support for the final generation of realistic restored images.

[0125] Step 106: Generate a high-resolution repair image corresponding to the damaged image based on the high-resolution enhanced feature map.

[0126] Specifically, the high-resolution enhanced feature map is a deep feature tensor output by the structure-enhanced texture inpainting network. It typically has the same high spatial resolution as the target inpainted image (e.g., 512×512). The high-resolution enhanced feature map integrates multi-dimensional information such as global semantics, local texture, edge continuity, and wireframe structure. It contains complete structural constraints, fine texture information, and features for completing damaged areas, and is the core data for generating the final image. The high-resolution inpainted image is the final product of the image inpainting in this application. It has the same preset high-resolution size (e.g., 512×512), and features structural coherence, realistic texture, and no obvious artifacts, completely repairing the missing / occluded areas of the original damaged image.

[0127] In some embodiments, a high-resolution enhanced feature map is input into a prediction layer of a high-resolution image inpainting model. This layer can consist of a 1×1 convolutional kernel. The convolution operation reduces the number of channels in the high-resolution enhanced feature map to 3 (corresponding to RGB three channels), converting the feature vector at each spatial location into pixel values. Subsequently, a Tanh or Sigmoid activation function is applied to the output, constraining the pixel values ​​to a reasonable dynamic range (e.g., [0,1] or [-1,1]) to ensure the color fidelity of the generated image. Through pixel-level reconstruction, a high-resolution inpainted image is directly output, completing the conversion from feature space to image space. Optionally, mean filtering can be used to remove high-frequency noise, and non-maximum suppression can be used to optimize edge details, ensuring a natural transition between the inpainted area and the known area without stitching marks. The final output is a high-resolution inpainted image with complete structure, realistic texture, and conforming to visual display standards.

[0128] In some embodiments, reference Figure 2 , Figure 2This is a schematic diagram of the structure of a high-resolution image restoration model provided in this application. A low-resolution damaged image and its corresponding binary mask are obtained. The low-resolution damaged image is input into the high-resolution image restoration model. The structure extraction network of the high-resolution image restoration model extracts structural features from the damaged image. The structure extraction network can extract wireframes and edge maps from the damaged image using LSM-HAMP and DexiNed methods. Next, the damaged image, edge map, and wireframe map are masked using the binary mask, resulting in corresponding mask images, mask edge maps, and mask wireframe maps. The mask images, binary mask images, mask edge maps, and mask wireframe maps are then concatenated along the channel dimension to obtain a multi-channel input tensor. The multi-channel input tensor is input into the high-resolution structure restoration network of the high-resolution image restoration model. In the high-resolution structure restoration network, firstly, the multi-channel input tensor is subjected to convolutional downsampling to obtain the corresponding first downsampled feature map. Next, the first downsampled feature map is input into the efficient Transformer block in the high-resolution structure restoration network. The efficient Transformer block performs high-order space interaction and feature enhancement on the first downsampled feature map to obtain the corresponding structure enhancement feature map. Finally, the structure enhancement feature map is subjected to transposed convolutional upsampling to generate the corresponding high-resolution structure sketch space, which includes a high-resolution edge map and a high-resolution wireframe map. Upsampling is performed on the mask image and the binary mask image respectively to obtain the upsampled mask image and the upsampled binary image; convolution is performed on the high-resolution edge image and the high-resolution wireframe image respectively to obtain the corresponding first deep structure feature map and the second deep structure feature map; convolution is performed on the upsampled mask image and the upsampled binary image respectively to obtain the corresponding first initial visual feature map and the second initial visual feature map; the first deep structure feature map, the second deep structure feature map, the first initial visual feature map, and the second initial visual feature map are added together to obtain the structural visual fusion feature map. The structural visual fusion feature map is input into the structure-enhanced texture inpainting network of the high-resolution image inpainting model. In this network, firstly, the structural visual fusion feature map is downsampled via convolution to obtain a corresponding second downsampled feature map. Next, this second downsampled feature map is input into the global-local Transformer block of the network, where global-local feature enhancement is performed to obtain the corresponding texture enhancement feature map. Finally, the texture enhancement feature map is upsampled via convolution to generate a high-resolution enhanced feature map. A high-resolution restored image corresponding to the damaged image is then generated based on this high-resolution enhanced feature map.

[0129] Based on the high-resolution image restoration method provided in this application, a low-resolution damaged image and its corresponding binary mask image are obtained. Structural features are extracted from the damaged image to obtain corresponding edge and wireframe images. The damaged image, edge image, and wireframe image are then masked using the binary mask image, and the corresponding masking results are concatenated with the binary mask image to form a multi-channel input tensor, providing more comprehensive structural information about the image to be restored. This multi-channel input tensor is then input into a high-resolution structural restoration network, which performs structural restoration on the edge and wireframe images, generating high-resolution edge and wireframe images. This provides prior information for subsequent image restoration tasks, ensuring that the restored image maintains structural integrity and accuracy. Next, the high-resolution edge image, high-resolution wireframe image, mask image, and binary mask image are fused to obtain a corresponding structural visual fusion feature map, enhancing the expressiveness of the image structure and providing a good foundation for subsequent texture restoration. The structural visual fusion feature map is input into the structural enhancement texture restoration network. The network then enhances the feature map, effectively restoring texture details in the missing areas and ensuring consistency between texture and structure during restoration. This generates a high-resolution enhanced feature map, which is then used to generate a high-resolution restored image corresponding to the damaged image. This solves the technical problem of maintaining the overall structural integrity of the image during high-resolution image restoration in existing technologies, improves the structural reconstruction capability of high-resolution restoration, enhances texture restoration quality, and generates restoration results with reasonable structure and consistent texture semantics.

[0130] In some embodiments, the step of inputting the first downsampled feature map into an efficient Transformer block in a high-resolution structural repair network, and obtaining the corresponding structurally enhanced feature map through high-order spatial interactions and feature enhancement of the first downsampled feature map by the efficient Transformer block includes:

[0131] The first downsampled feature map is normalized to obtain the corresponding first normalized feature map;

[0132] Perform a recursive gated convolution operation on the first normalized feature map to obtain the corresponding spatial blending feature map;

[0133] The spatial hybrid feature map is transformed by the first feedforward neural network to obtain the corresponding initial structure enhancement feature map. The initial structure enhancement feature map and the first downsampled feature map are then added to obtain the corresponding first structure fusion feature map.

[0134] The first structure fusion feature map is input into the hybrid attention module in the efficient Transformer block. The first structure fusion feature map is normalized by the normalization layer in the hybrid attention module to obtain the corresponding normalized structure fusion feature map.

[0135] The normalized structure fusion feature map is subjected to structural feature enhancement processing by the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head interactive attention processing result; and the normalized structure fusion feature map is subjected to global semantic extraction by the full attention branch in the hybrid attention module to obtain the corresponding full attention processing result.

[0136] The results of full attention processing, multi-head interactive attention processing, and the first structure fusion feature map are added together to obtain the corresponding second structure fusion feature map.

[0137] The second structure fusion feature map is transformed by the second feedforward neural network to obtain the corresponding intermediate structure enhancement feature map. The intermediate structure enhancement feature map and the second structure fusion feature map are then added together to obtain the structure enhancement feature map.

[0138] Specifically, the recursive gated convolution operation is a feature processing method based on gnConv. It establishes long-range dependencies by splitting feature sub-regions and recursively calculating them, while dynamically filtering effective features through a gating mechanism to achieve high-order spatial interaction. Both the first and second feedforward neural networks are feature transformation units composed of "linear layers + activation functions + linear layers," used to perform non-linear transformations on the feature map to enhance feature representation capabilities. The hybrid attention module is the core unit of the efficient Transformer block, integrating multi-head interactive attention branches and full attention branches, taking into account both local structural correlations and global semantic information. The multi-head interactive attention branch can split the feature into multiple attention heads, capturing the structural correlations of different local regions respectively, strengthening the structural connection between damaged and known regions. The full attention branch can perform global attention calculations on the overall feature map, extracting global semantic information of the image (such as overall structural logic and style consistency), avoiding disconnections in local restoration. The structure-enhanced feature map, possessing high-order spatial interaction capabilities, local structural correlations, and global semantic consistency, is the core intermediate result of the high-resolution structure restoration network.

[0139] As an example, see reference Figure 3 , Figure 3This is a schematic diagram of the efficient Transformer block. The first downsampled feature map is input into the efficient Transformer block in the high-resolution structural inpainting network. This efficient Transformer block performs high-order spatial interactions and structural enhancements on the first downsampled feature map, improving the geometric consistency and detail fidelity of the inpainting result. Specifically, the first downsampled feature map X obtained from pre-convolution downsampling is input into the efficient Transformer block. The efficient Transformer block mainly consists of a HorNet module, a Hybrid Attention Module (HFA), and a second feedforward neural network. These three components work together to achieve a unification of local spatial modeling and global semantic understanding. First, the first downsampled feature map X is normalized using Layer Normalization to obtain a first normalized feature map, which stabilizes the training process and improves gradient propagation efficiency. Then, a recursive gated convolution operation is performed on the first pair of normalized feature maps. This operation, based on a layer-by-layer recursive mechanism, dynamically captures high-order spatial dependencies through multi-step convolution-gating sequences, generating a spatial blending feature map that effectively models complex geometric structures such as edge continuity and wireframe topology. Next, a first feedforward neural network performs a nonlinear transformation on the spatial blending feature map to obtain an initial structure-enhanced feature map. The initial structure-enhanced feature map is then added to the first downsampled feature map X using residuals to form a first structure-fusion feature map. This residual operation avoids feature information loss and ensures the integrity of the original structural features (the first downsampled feature map). Subsequently, the first structural fusion feature map is input into the Hybrid Attention Module (HFA). The HFA consists of a multi-head interactive attention branch and a full attention branch connected in parallel. The multi-head interactive attention branch focuses on local structure perception, capturing fine-grained interactions between pixels through a multi-head self-attention mechanism to generate the multi-head interactive attention processing result. The full attention branch utilizes a global attention mechanism (such as fully connected or axial attention) to extract long-range semantic context, generating the full attention processing result. Simultaneously, to ensure stability, the first structural fusion feature map is processed by a normalization layer before entering the attention module to obtain a normalized structural fusion feature map. Then, through residual connections, the full attention processing result, the multi-head interactive attention processing result, and the first structural fusion feature map are integrated. Figure 3The two are added together to obtain a second structural fusion feature map that balances local structural coherence and global semantic consistency, achieving deep fusion of local and global information. Finally, the second structural fusion feature map is further transformed by a second feedforward neural network to generate an intermediate structural enhancement feature map. This intermediate structural enhancement feature map is then added to the second structural fusion feature map via residual connections to obtain the final structural enhancement feature map. In the processing of the efficient Transformer block, recursive gated convolution is used to model higher-order spatial interactions. Combined with a hybrid attention mechanism, this achieves dual enhancement of local structural reinforcement and global semantic guidance, which can significantly improve the reconstruction capability of the structural repair network for complex geometric shapes. It is especially suitable for scenes with strong structure and helps to solve problems such as structural breakage and texture inconsistency caused by the loss of local details in traditional Transformers in high-resolution tasks. This lays a solid foundation for subsequent upsampling to generate high-fidelity structural sketches.

[0140] In some embodiments, the step of performing structural feature enhancement processing on the normalized structural fusion feature map through the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head interactive attention processing result includes:

[0141] The normalized structure fusion feature map is segmented along the channel dimension to obtain the sub-features corresponding to each attention head, and a linear transformation is performed on each sub-feature to generate the corresponding query tensor, key tensor and value tensor.

[0142] Based on the binary mask image, the query tensor, key tensor, and value tensor are respectively divided into known region features and masked region features;

[0143] Calculate the first global feature vector corresponding to the known region features and the second global feature vector corresponding to the masked region features respectively, and multiply the first global feature vector and the second global feature vector element by element to obtain the known region enhancement feature and the masked region enhancement feature.

[0144] Based on the known region enhancement features and the masked region enhancement features, the SENet network is used to dynamically predict the weights and obtain the corresponding weight coefficients.

[0145] Based on the weight coefficients, the enhanced features of the known region and the enhanced features of the masked region are weighted and fused to obtain the corresponding weighted fusion result;

[0146] The known region features and the weighted fusion results are combined to obtain the output features of each attention head;

[0147] The output features of each attention head are concatenated along the channel dimension to obtain the multi-head interactive attention processing result.

[0148] Specifically, the SENet network is a weight prediction network based on the "squeeze-excitation" mechanism. Through the GAP→FC→ReLU→FC→Sigmoid structure, it dynamically predicts the weights of the channels to predict the fusion ratio between the known region and the masked region.

[0149] refer to Figure 4 , Figure 4 This is a schematic diagram of the multi-head interactive attention branch in the hybrid attention module provided in this application. As an example, the normalized structure is fused with the feature map F, F∈R. HW×C Segmented into multiple sub-features F along the channel dimension i (e.g., i), F i ∈R HW×C Each sub-feature F i The query tensor Q is generated by linear transformation. i Key tensor K i Sum tensor V i This is to support parallel computation under a multi-head interactive attention mechanism. Generate the query tensor Q. i Key tensor K i Sum tensor V i The mathematical expression is as follows:

[0150]

[0151] in, , and To represent the weight matrix of the linear transformation, Q i Let K be the query tensor of the i-th attention head. i Let V be the key tensor of the i-th attention head. i Let be the value tensor of the i-th attention head.

[0152] Subsequently, based on the binary mask image M, the query tensor Q is... i Key tensor K i Sum tensor V i The regions are divided into known region features and masked region features to achieve explicit spatial differentiation between reliable information and missing regions. Next, global average pooling (GAP) is applied to both known and masked region features to obtain a first global feature vector representing the global features of the known region and a second global feature vector representing the global features of the masked region, representing the overall semantic information of their respective regions. Then, the first and second global feature vectors are broadcast back to the original spatial dimensions and multiplied element-wise with the value tensor of the corresponding region to generate enhanced features for the known region. and mask region enhancement features This strengthens the consistency and structural continuity within the region. The specific calculations are as follows:

[0153]

[0154]

[0155] in, Let Q represent a decomposable kernel with characteristic representations, and let ⊙ represent element-wise multiplication; i Let K be the query tensor of the i-th attention head. i Let V be the key tensor of the i-th attention head. i Let i be the value tensor of the attention head; Let i be the query sub-tensor of the known region in the i-th attention head; Let i be the query subtensor of the masked region in the i-th attention head. Let be the key tensor of the known region in the i-th attention head. Let be the key tensor of the masked region in the i-th attention head. Value tensor V i The feature vector of the j-th spatial location in the middle, For the known region bond tensor The feature vector of the j-th spatial location in the middle, For mask region key tensor The feature vector of the j-th spatial location; Enhance features for the known region of the i-th attention head. Enhance the features of the masked region for the i-th attention head.

[0156] Based on this, known region enhancement features and masked region enhancement features are input into the SENet network. The SENet network consists of global average pooling, fully connected layers, ReLU activation, and a sigmoid output, used to dynamically predict weight coefficients. , ∈[0,1], where The contribution ratio of the known region to the restoration process is represented. Then, based on all weight coefficients, the enhanced features of the known region and the enhanced features of the masked region are weighted and fused to obtain a weighted fusion result, achieving semantic guidance and structural extrapolation for the missing region. Finally, the enhanced features of the known region are... The results are combined with the weighted fusion results (e.g., added) to form the output features of each attention head. Then pay attention to all the features of the head By concatenating along the channel dimension, the final multi-head interactive attention processing result is obtained. As the output of the multi-head interactive attention branch, the specific calculation is as follows:

[0157]

[0158] in, Indicates global average pooling. Represents the ReLU activation function. This represents the Sigmoid function. and It is the first The weight matrix of the two fully connected layers in the attention head. This indicates a splicing operation. Indicates the first The weight ratio learned from each attention head; Enhance features for the known region of the i-th attention head. Enhance the features of the masked region for the i-th attention head. E i For the fused output features of the i-th attention head, The result of multi-head interactive attention processing is shown. C is the total number of channels in the feature map, H and W are the spatial height and width of the feature map, and M is the binary mask image (1 represents the mask / damaged area, and 0 represents the known / complete area).

[0159] By employing a design that combines regional association with dynamic weighting, we can, on the one hand, divide known / masked regions based on binary masks, allowing attention computation to focus on using known structures to guide mask region completion; on the other hand, by dynamically predicting weights through the SENet network, we can make the structure of the completed mask region more consistent with the global semantic logic of the known region, avoiding structural breaks or style discrepancies. The final output of the multi-head interactive attention processing results provides a structural foundation for accurate completion of subsequent global-local feature fusion, directly serving the complete generation of high-resolution structural sketch space.

[0160] In some embodiments, the step of inputting the second downsampled feature map into the global-local Transformer block of the structure-enhanced texture inpainting network, and performing global-local feature enhancement on the second downsampled feature map through the global-local Transformer block to obtain the corresponding texture-enhanced feature map includes:

[0161] The second downsampled feature map is normalized to obtain the corresponding second normalized feature map;

[0162] A recursive gated convolution operation is performed on the second normalized feature map to obtain the corresponding initial texture enhancement feature map. The initial texture enhancement feature map and the second downsampled feature map are then added to obtain the corresponding first texture fusion feature map.

[0163] The first texture fusion feature map is normalized to obtain the corresponding normalized texture fusion feature map;

[0164] The normalized texture fusion feature map is input into the global-local fusion module of the global-local Transformer block. Global features are extracted through the global branch of the global-local fusion module and filtered by the gating unit. Local detail features are extracted through the local branch of the global-local fusion module. The filtered global features are combined with the local features to obtain the intermediate texture enhancement feature map.

[0165] The intermediate texture enhancement feature map and the first texture fusion feature map are added together to obtain the corresponding second texture fusion feature map;

[0166] The second texture fusion feature map is transformed by a third feedforward neural network, and the corresponding feature transformation result is added to the second texture fusion feature map to obtain the texture enhancement feature map.

[0167] Specifically, the global-local fusion module is the core unit of the global-local Transformer block, containing global and local branches to capture global texture style and local texture details, respectively. The third feedforward neural network can be a non-linear transformation unit composed of a linear layer + ReLU activation function + linear layer, used to enhance the expressive power of texture features and enrich texture dimensions. The texture enhancement feature map is the feature map obtained after complete processing by the global-local Transformer block, possessing both consistent global texture style and fine local texture details, and is the core intermediate result for subsequent upsampling.

[0168] refer to Figure 5 , Figure 5This document provides a schematic diagram of the structure of a global-local Transformer block. In the high-resolution image inpainting method of this application, inputting the second downsampled feature map into the global-local Transformer block of the structure-enhanced texture inpainting network, and performing global-local feature enhancement through this global-local Transformer block to generate a texture-enhanced feature map, is a crucial step in achieving high-quality texture reconstruction. As an example, the second downsampled feature map G obtained by pre-convolution downsampling is input into the global-local Transformer block, which consists of a HorBlock module, a GateBlock module, and a third feedforward neural network. These three modules work together to complete multi-scale feature enhancement from local details to global semantics. The GateBlock module includes a global-local fusion module. First, the second downsampled feature map G is normalized (Layer Normalization) to obtain a second normalized feature map, which stabilizes the training of the subsequent attention mechanism. Then, a recursive gated convolution operation is performed on the second normalized feature map. This recursive gated convolution operation dynamically models high-order spatial interactions through multi-step convolution-gating sequences to generate an initial texture enhancement feature map, effectively capturing texture continuity and local structural patterns. Next, the initial texture enhancement feature map and the second downsampled feature map G are added together by residual to form a first texture fusion feature map, which retains the original information and enhances the ability to express local details. The first texture fusion feature map is the output of HorBlock.

[0169] Subsequently, the first texture fusion feature map is input into the GateBlock module. This map is first processed again by the normalization unit within the GateBlock module using LayerNorm, resulting in a stable, normalized texture fusion feature map. This normalized map is then input into the global-local fusion module within the GateBlock module. The global-local fusion module contains two parallel branches: the global branch uses a multi-head interactive attention mechanism to extract long-distance dependencies, obtain overall scene semantics, and filters global features using a gating unit (such as a learnable gating matrix activated by Sigmoid), retaining key contextual information and extracting global features; the local branch uses 3×3 convolution to extract local fine texture details, focusing on texture details and edge structures within the pixel neighborhood, generating local features. The filtered global and local features are then combined to obtain an intermediate texture enhancement feature map, achieving consistency between global semantic guidance and local details. Next, the intermediate texture enhancement feature map is added to the first texture fusion feature map to obtain a second texture fusion feature map that integrates global style and local details, further enhancing feature consistency and robustness. Finally, a third feedforward neural network is used to perform a nonlinear transformation on the second texture fusion feature map to generate the feature transformation result. This result is then added to the residual of the second texture fusion feature map to obtain the final texture enhancement feature map G′. By modeling the local texture structure through recursive gated convolution and combining it with the global-local fusion module to achieve cross-scale feature interaction, the texture inpainting network's ability to restore fine structures in complex scenes is significantly improved. It is particularly suitable for inpainting tasks with large-area missing or fine textures in high-resolution images. This helps to solve problems such as texture breaks, repetitions, or unnaturalness caused by ignoring global consistency in traditional methods, laying a solid foundation for subsequent upsampling to generate realistic and coherent high-resolution inpainted images.

[0170] In some embodiments, the normalized texture fusion feature map is input into the global-local fusion module of the global-local Transformer block. This global-local fusion module consists of a global branch and a local branch, which are processed in parallel to extract feature information at different scales. In the global branch, the normalized texture fusion feature map G is processed... h Global features of the overall texture style of the image are extracted. After dynamic selection of global features effective for texture restoration by a gating unit, the spatial correlation of the global features is enhanced by a multi-head interactive attention branch, resulting in enhanced global features. In local branches, gating units (such as learnable gating matrices with sigmoid activation) are also introduced to normalize the texture fusion feature map G. h Dynamic filtering is performed to retain effective local features, and then lightweight convolution operations (such as 3×3 convolution) are used to further refine texture details, resulting in enhanced local features. The system extracts fine texture and edge information through local branches to generate local feature representations. Then, the filtered global features output from the global branch are used... Local features of local branch output The concatenation is performed in the spatial dimension to obtain the corresponding concatenation result. To enhance the feature representation capability, the concatenation result is further transformed by a non-linear layer (BN-ReLU) containing BatchNorm and ReLU activation functions, finally generating an intermediate texture enhancement feature map. h By anchoring the overall texture style through global branches, the texture of damaged areas is prevented from becoming disconnected from the original image; local branches complete fine details and solve the problem of rough texture; and gating units filter effective features to improve fusion efficiency, providing a high-quality feature foundation for subsequent texture enhancement.

[0171] In some embodiments, before the step of extracting structural features from the damaged image to obtain the corresponding edge map and wireframe map, the method further includes:

[0172] Obtain the training set, which includes multiple training samples and labels for each training sample. The labels for the training samples include the high-resolution target image corresponding to the training sample, the high-resolution target edge map corresponding to the training sample, and the high-resolution target wireframe map corresponding to the training sample. The training samples are low-resolution training images and training binary mask maps corresponding to the training images.

[0173] The training image is input into the structure extraction network in the high-resolution image restoration model to be trained. The structure extraction network extracts structural features from the training image to obtain the corresponding training edge map and training wireframe map.

[0174] The training image, training edge map, and training wireframe map are masked by training a binary mask image to obtain the corresponding training mask image, training mask edge map, and training mask wireframe map.

[0175] The training mask image, training binary mask image, training mask edge image and training mask wireframe image are channel-concatenated to form a training multi-channel input tensor;

[0176] The multi-channel input tensor is trained and input into the high-resolution structural insulation network of the high-resolution image insulation model. The high-resolution structural insulation network is used to perform structural insulation on the edge map and wireframe map, generating high-resolution edge prediction map and high-resolution wireframe prediction map.

[0177] Based on the high-resolution edge prediction map and the high-resolution target edge map, the first loss value corresponding to the cross-entropy loss function is calculated, and based on the high-resolution wireframe prediction map and the high-resolution target wireframe map, the second loss value corresponding to the cross-entropy loss function is calculated.

[0178] The first-stage loss value of the high-resolution image inpainting model is calculated based on the first and second loss values, and the parameters of the high-resolution structural inpainting network are updated based on the first-stage loss value.

[0179] The training image is input into the high-resolution image restoration model, and the low-resolution training image is restored by the high-resolution image restoration model to be trained, so as to obtain the high-resolution prediction image corresponding to the training image.

[0180] Based on the high-resolution predicted image and the high-resolution target image, the second-stage loss value of the high-resolution image inpainting model is determined, and the parameters of the structure-enhanced texture inpainting network are updated based on the second-stage loss value.

[0181] When the loss value in the second stage is less than or equal to the preset value, a high-resolution image restoration model that has been trained is obtained.

[0182] Specifically, the training set is a dataset used to train the high-resolution image restoration model. It contains multiple sets of "training samples + corresponding labels" and serves as the fundamental data source for the high-resolution image restoration model to learn its restoration logic. The training samples consist of low-resolution training images and training binary masks, simulating the input format of low-resolution damaged images in real-world scenarios (the training binary mask marks the damaged areas of the training image). The labels of the training samples represent the ideal restoration results corresponding to the training samples, including the high-resolution target image (the final desired restored image), the high-resolution target edge map, and the high-resolution target wireframe map (a structural feature reference standard for the target image).

[0183] In the high-resolution image restoration method of this application, to ensure that the high-resolution image restoration model can reconstruct a high-resolution restoration result with complete structure and realistic texture, a systematic two-stage training process needs to be completed before extracting structural features from the damaged image to obtain a trained high-resolution image restoration model. As an example, firstly, a training set containing multiple training samples and their corresponding labels is obtained. Each training sample consists of a low-resolution training image and its corresponding training binary mask. The training binary mask is a matrix with the same size as the training image, where regions with a value of 1 identify missing parts and regions with a value of 0 identify known parts, used to accurately guide the restoration range. The labels consist of three parts: a high-resolution target image (used for final image reconstruction supervision), a high-resolution target edge map (depicting fine contours), and a high-resolution target wireframe map (representing the global skeleton). The labels can be generated offline from high-resolution real images using standard edge detection algorithms (such as Canny) and wireframe extraction methods (such as HAWP or LSD combined with post-processing), and paired with manually or program-generated masks to form a complete supervision signal. Subsequently, the low-resolution training image is input into the structure extraction network of the high-resolution image inpainting model to be trained. The structure extraction network outputs the initial training edge map and training wireframe map. Next, the training image, training edge map, and training wireframe map are masked (i.e., multiplied element-wise) using the training binary mask image to obtain the training mask image, training mask edge map, and training mask wireframe map, thus masking missing regions and preventing information leakage. The training mask image, training mask edge map, training mask wireframe map, and training binary mask image are then concatenated along the channel dimension to form a training multi-channel input tensor. This training multi-channel input tensor is then fed into the high-resolution structure inpainting network (containing efficient Transformer blocks). After downsampling, it undergoes deep inpainting (capturing long-range structural dependencies) through four stacked efficient Transformer blocks. Finally, it is upsampled through transposed convolution to output the inpainted high-resolution edge prediction map and high-resolution wireframe prediction map. After the high-resolution structure inpainting network generates the structure prediction, the first stage of loss calculation and optimization begins. Based on the predicted high-resolution edge prediction map and high-resolution wireframe prediction map, and the ground truth labels (high-resolution target edge map and high-resolution target wireframe map), a cross-entropy loss function is calculated. Specifically, the first loss value is the binary cross-entropy (BCE) loss between the high-resolution edge prediction map and the high-resolution target edge map, and the second loss value is the BCE loss between the high-resolution wireframe prediction map and the high-resolution target wireframe map, as shown in the following formula:

[0184]

[0185] in, Represents a high-resolution target wireframe diagram. Represents a high-resolution target edge map. This represents a high-resolution wireframe prediction plot. Represents a high-resolution edge prediction map. The first loss value, The second loss value is calculated by weighting and summing the second and first loss values ​​(e.g., by simple averaging). The first-stage loss value measures the difference between the predicted and actual structures and is used for backpropagation to update the parameters of the high-resolution structure repair network. This enhances the model's ability to reconstruct geometric structures, enabling the high-resolution structure repair network to learn to generate accurate geometric contours during training. Through backpropagation and an optimizer (e.g., Adam), only the parameters of the high-resolution structure repair network are updated, allowing the network to focus on the structure repair task and improving edge coherence and line rationality.

[0186] After completing the first stage of training, the second stage begins: the same training image is input into the entire high-resolution image inpainting model (including the pre-trained structural branches). Through structural visual fusion and structural enhancement texture inpainting networks, a high-resolution predicted image is generated. The second-stage loss value is calculated based on the high-resolution predicted image and the high-resolution target image. This second-stage loss typically includes pixel reconstruction loss, adversarial loss, feature matching loss, and high receptive field perception loss, used to optimize visual realism and detail fidelity. The parameters of the structural enhancement texture inpainting network are updated accordingly (the structural branches can be frozen or jointly fine-tuned). The entire training process iterates until the second-stage loss value converges to a preset threshold or the validation performance no longer improves. At this point, the trained high-resolution image inpainting model is obtained. The entire training process constructs a training set with structural labels and uses a two-stage loss function to optimize the high-resolution structural restoration network and the structural enhancement texture restoration network respectively. Edges and wireframes are explicitly introduced as intermediate supervision signals to guide the model to learn reliable geometric priors first, avoiding distortion caused by texture dominating structure in end-to-end training. At the same time, the phased optimization strategy can effectively decouple the structural reconstruction and texture generation tasks, allowing each sub-network (high-resolution structural restoration network and structural enhancement texture restoration network) to focus on its core functions. This enables the high-resolution image restoration model to have the ability to restore high-resolution images with complete structure, delicate texture, and consistent style.

[0187] In some embodiments, the step of determining the second-stage loss value of the high-resolution image inpainting model based on the high-resolution prediction image and the high-resolution target image includes:

[0188] Calculate the L1 norm loss between the high-resolution predicted image and the high-resolution target image to obtain the pixel reconstruction loss value;

[0189] The high-resolution predicted image and the high-resolution target image are input into the discriminator, and the discriminator outputs the corresponding true or false judgment result. The discriminator loss and generator loss are calculated based on the true or false judgment result, and a gradient penalty term is added to obtain the adversarial loss value.

[0190] Obtain the output feature maps of the discriminator on the high-resolution prediction image and the high-resolution target image at multiple intermediate activation layers, and calculate the L1 norm loss between the output feature maps of the corresponding layers as the feature matching loss value;

[0191] The high-resolution predicted image and the high-resolution target image are respectively input into a pre-trained residual network with a dilated convolutional structure to obtain the corresponding first high-level semantic feature map and the second high-level semantic feature map.

[0192] The feature similarity between the first high-level semantic feature map and the second high-level semantic feature map is calculated and used as the high receptive field perceptual loss value.

[0193] The second-stage loss value is obtained by weighted summation of the pixel reconstruction loss value, adversarial loss value, feature matching loss value, and high receptive field perception loss value.

[0194] Specifically, the pixel reconstruction loss is calculated using the L1 norm loss, reflecting the pixel-level similarity between the high-resolution predicted image and the target image, ensuring the basic visual accuracy of the restored image. The discriminator is a neural network (usually a convolutional neural network) used to determine whether the input image is "real" or "generated." In adversarial training, the discriminator and the generator (here, a SETR network) compete to improve the realism of the generated image. The adversarial loss is composed of the discriminator loss, generator loss, and gradient penalty term. This adversarial competition between the generator and discriminator enhances the realism of the predicted image, preventing it from appearing overly artificial. The feature matching loss is calculated by comparing the feature maps of the discriminator's intermediate activation layers, constraining the consistency of the predicted and target images in mid-level semantic features, avoiding pixel-level similarity but semantic disconnect. The high receptive field perceptual loss is calculated by comparing the similarity of high-level semantic feature maps, constraining the consistency of the predicted and target images in global semantics and overall structure, ensuring the global coherence of the restored image.

[0195] A multi-scale, multi-objective joint loss function system is used to comprehensively supervise the training process of the structure-enhanced texture inpainting network, ensuring that the inpainting results achieve high-quality standards in pixel-level accuracy, visual realism, semantic consistency, and global coherence. As an example, firstly, a high-resolution predicted image is calculated. With high-resolution target images The L1 norm loss (i.e., pixel reconstruction loss) between the two is used to obtain the pixel reconstruction loss value, and the formula is:

[0196]

[0197] in, The pixel reconstruction loss value ensures the fundamental accuracy of the repaired area in terms of color and brightness by minimizing the pixel-level absolute error. Particular focus is placed on preserving the integrity of known area information for accurate reconstruction of the masked area.

[0198] Secondly, high-resolution predicted images and high-resolution target images The data are input into the discriminator, which outputs a true / false judgment result. The discriminator loss is then calculated based on this result. and generator loss And add a gradient penalty term. Obtain the resistance loss value Specifically, it is expressed as:

[0199]

[0200] in, This is a gradient penalty term. , These are the weighting coefficients for the gradient penalty term, used to balance the strength of the gradient penalty. Generator loss. The goal is to make the predicted image be judged as real; the discriminator loss... To improve the discriminator's ability to distinguish between true and false data, a gradient penalty term is added. To enhance training stability and combat loss values Together, they drive the realism of the generated texture details.

[0201] Next, the output feature maps of the discriminator at multiple intermediate activation layers are obtained, and the high-resolution predicted images are compared. and high-resolution target images The feature difference in the intermediate layer is used to calculate the L1 norm loss between the corresponding feature maps of the predicted image and the target image as the feature matching loss value, enabling the generator to produce a feature representation similar to the real image. Specifically, this is expressed as follows:

[0202]

[0203] in, Discriminator The final convolutional layer; Indicates the first The number of elements in each activation layer For the discriminator Layer activation, The feature matching loss value is used. This loss value forces the inpainting result to align with the real image at an abstract level by comparing semantic features. Then, the high-resolution predicted image is... and high-resolution target images The pre-trained ResNet50 network with dilated convolutional structures is input to extract first and second high-level semantic feature maps containing rich contextual information. The feature similarity between the first and second high-level semantic feature maps is calculated as the high receptive field perceptual loss value, as shown in the formula:

[0204] in, This can represent a pre-trained segmentation ResNet50 network with dilated convolutional structures. The high receptive field perceptual loss value is used to determine the perceptual loss value. Improve the restoration model's ability to perceive global information and context, and maintain the overall coherence and rationality of the image.

[0205] Finally, the pixel reconstruction loss, adversarial loss, feature matching loss, and high receptive field perception loss are weighted according to a preset weight. , , , We perform a weighted summation to obtain the final second-stage loss value, which is defined as follows: .

[0206] By constructing a multi-scale, multi-objective composite loss function, the restoration results are comprehensively supervised from four dimensions: pixel, feature, semantic, and adversarial. This helps to solve common problems in high-resolution image restoration, such as blurring, artifacts, and content distortion.

[0207] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the high-resolution image restoration method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0208] This application also provides a high-resolution image restoration apparatus; please refer to... Figure 6 The high-resolution image restoration device includes:

[0209] The structure extraction module 601 is used to acquire a low-resolution damaged image and a corresponding binary mask image of the damaged image, extract structural features from the damaged image, and obtain the corresponding edge image and wireframe image.

[0210] The multi-channel cascade module 602 is used to perform masking processing on the damaged image, edge image and wireframe image respectively through the binary mask image, and to cascade the corresponding masking results with the binary mask image to form a multi-channel input tensor;

[0211] The structure repair module 603 is used to input multi-channel input tensors into a high-resolution structure repair network, and perform structure repair on the edge map and wireframe map through the high-resolution structure repair network to generate a high-resolution edge map and a high-resolution wireframe map.

[0212] The feature fusion module 604 is used to fuse the high-resolution edge map, high-resolution wireframe map, mask image and binary mask image to obtain the corresponding structural visual fusion feature map.

[0213] The texture restoration module 605 is used to input the structural visual fusion feature map into the structural enhancement texture restoration network, and perform feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture restoration network to generate a high-resolution enhanced feature map.

[0214] The feature decoding module 606 is used to generate a high-resolution repair image corresponding to the damaged image based on the high-resolution enhanced feature map.

[0215] The high-resolution image restoration apparatus provided in this application, employing the high-resolution image restoration method described in the above embodiments, can solve the technical problem in the prior art where it is difficult to maintain the overall structural integrity of an image during high-resolution image restoration. Compared with the prior art, the beneficial effects of the high-resolution image restoration apparatus provided in this application are the same as those of the high-resolution image restoration method provided in the above embodiments, and other technical features in the high-resolution image restoration apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0216] This application provides a high-resolution image restoration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the high-resolution image restoration method in the above embodiment 1.

[0217] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the high-resolution image restoration device of the embodiments of this application. The high-resolution image restoration device in the embodiments of this application may include, but is not limited to, mobile terminals such as laptops, tablets (Portable Application Description, PADs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The high-resolution image restoration device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0218] like Figure 7 As shown, the high-resolution image restoration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the high-resolution image restoration device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the high-resolution image restoration device to communicate wirelessly or wiredly with other devices to exchange data. Although a high-resolution image restoration device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0219] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0220] The high-resolution image restoration device provided in this application, employing the high-resolution image restoration method described in the above embodiments, can solve the technical problem of maintaining the overall structural integrity of an image during the high-resolution image restoration process in the prior art. Compared with the prior art, the beneficial effects of the high-resolution image restoration device provided in this application are the same as those of the high-resolution image restoration method provided in the above embodiments, and other technical features of this high-resolution image restoration device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0221] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0222] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0223] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the high-resolution image restoration method in the above embodiments.

[0224] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0225] The aforementioned computer-readable storage medium may be included in a high-resolution image restoration device; or it may exist independently and not be assembled into a high-resolution image restoration device.

[0226] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a high-resolution image restoration device, the high-resolution image restoration device performs the following actions: acquires a low-resolution damaged image and a corresponding binary mask image; extracts structural features from the damaged image to obtain corresponding edge and wireframe images; performs masking processing on the damaged image, edge image, and wireframe image using the binary mask image, and concatenates the corresponding masking results with the binary mask image to form a multi-channel input tensor; inputs the multi-channel input tensor into a high-resolution structural restoration network, performs structural restoration on the edge and wireframe images using the high-resolution structural restoration network, and generates high-resolution edge and wireframe images; fuses the high-resolution edge image, high-resolution wireframe image, mask image, and binary mask image to obtain a corresponding structural visual fusion feature map; inputs the structural visual fusion feature map into a structural enhancement texture restoration network, performs feature enhancement processing on the structural visual fusion feature map using the structural enhancement texture restoration network, and generates a high-resolution enhancement feature map; and generates a high-resolution restored image corresponding to the damaged image based on the high-resolution enhancement feature map.

[0227] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0228] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0229] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0230] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described high-resolution image restoration method. This solves the technical problem in the prior art where it is difficult to maintain the overall structural integrity of the image during high-resolution image restoration. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the high-resolution image restoration method provided in the above embodiments, and will not be repeated here.

[0231] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the high-resolution image restoration method described above.

[0232] The computer program product provided in this application can solve the technical problem of maintaining the overall structural integrity of an image during high-resolution image restoration in the prior art. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the high-resolution image restoration method provided in the above embodiments, and will not be repeated here.

[0233] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A high-resolution image restoration method, characterized in that, The high-resolution image restoration method includes: A low-resolution damaged image and a corresponding binary mask image are obtained. Structural features are extracted from the damaged image to obtain the corresponding edge image and wireframe image. The damaged image, the edge image, and the wireframe image are respectively masked using the binary mask image to obtain the corresponding mask image, mask edge image, and mask wireframe image; The mask image, the binary mask image, the mask edge image, and the mask wireframe image are concatenated along the channel dimension to obtain a multi-channel input tensor. The multi-channel input tensor is input into a high-resolution structure repair network, and the edge map and wireframe map are repaired by the high-resolution structure repair network to generate a high-resolution edge map and a high-resolution wireframe map. The high-resolution edge map, the high-resolution wireframe map, the mask image, and the binary mask image are fused to obtain the corresponding structural visual fusion feature map. The structural visual fusion feature map is input into the structural enhancement texture restoration network, and the structural visual fusion feature map is enhanced by the structural enhancement texture restoration network to generate a high-resolution enhanced feature map. A high-resolution repair image corresponding to the damaged image is generated based on the high-resolution enhanced feature map.

2. The high-resolution image restoration method as described in claim 1, characterized in that, The step of inputting the multi-channel input tensor into the high-resolution structure repair network, and performing structure repair on the edge map and wireframe map through the high-resolution structure repair network to generate high-resolution edge map and high-resolution wireframe map includes: The multi-channel input tensor is subjected to convolutional downsampling to obtain the corresponding first downsampled feature map; The first downsampled feature map is input into the efficient Transformer block in the high-resolution structure repair network. The efficient Transformer block performs high-order spatial interaction and feature enhancement on the first downsampled feature map to obtain the corresponding structure enhancement feature map. The efficient Transformer block is constructed based on recursive gated convolution and multi-head interactive attention mechanism. The structure enhancement feature map is subjected to transposed convolution upsampling processing to generate a corresponding high-resolution structure sketch space, which includes the high-resolution edge map and the high-resolution wireframe map.

3. The high-resolution image restoration method as described in claim 2, characterized in that, The step of inputting the first downsampled feature map into the efficient Transformer block in the high-resolution structure repair network, and obtaining the corresponding structure-enhanced feature map through the high-order space interaction and feature enhancement of the first downsampled feature map by the efficient Transformer block includes: The first downsampled feature map is normalized to obtain the corresponding first normalized feature map; Perform a recursive gated convolution operation on the first normalized feature map to obtain the corresponding spatial blending feature map; The spatial hybrid feature map is transformed by the first feedforward neural network to obtain the corresponding initial structure enhancement feature map. The initial structure enhancement feature map and the first downsampled feature map are then added to obtain the corresponding first structure fusion feature map. The first structure fusion feature map is input into the hybrid attention module in the efficient Transformer block, and the first structure fusion feature map is normalized by the normalization layer in the hybrid attention module to obtain the corresponding normalized structure fusion feature map. The normalized structure fusion feature map is subjected to structural feature enhancement processing through the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head interactive attention processing result; and the normalized structure fusion feature map is subjected to global semantic extraction through the full attention branch in the hybrid attention module to obtain the corresponding full attention processing result. The full attention processing result, the multi-head interactive attention processing result, and the first structural fusion feature map are added together to obtain the corresponding second structural fusion feature map. The second structure fusion feature map is transformed by the second feedforward neural network to obtain the corresponding intermediate structure enhancement feature map. The intermediate structure enhancement feature map and the second structure fusion feature map are then added together to obtain the structure enhancement feature map.

4. The high-resolution image restoration method as described in claim 3, characterized in that, The step of performing structural feature enhancement processing on the normalized structural fusion feature map through the multi-head interactive attention branch in the hybrid attention module to obtain the corresponding multi-head interactive attention processing result includes: The normalized structure fusion feature map is segmented along the channel dimension to obtain the sub-features corresponding to each attention head, and each sub-feature is linearly transformed to generate the corresponding query tensor, key tensor and value tensor. Based on the binary mask image, the query tensor, the key tensor, and the value tensor are respectively divided into known region features and masked region features; Calculate the first global feature vector corresponding to the known region feature and the second global feature vector corresponding to the masked region feature respectively, and multiply the first global feature vector and the second global feature vector element by element to obtain the known region enhancement feature and the masked region enhancement feature; Based on the known region enhancement features and the masked region enhancement features, the SENet network is used to dynamically predict the weights and obtain the corresponding weight coefficients. Based on the weighting coefficients, the enhanced features of the known region and the enhanced features of the masked region are weighted and fused to obtain the corresponding weighted fusion result; The known region features and the weighted fusion result are combined to obtain the output features of each attention head; The output features of each attention head are concatenated along the channel dimension to obtain the multi-head interactive attention processing result.

5. The high-resolution image restoration method as described in claim 1, characterized in that, The step of inputting the structural visual fusion feature map into the structural enhancement texture inpainting network, and performing feature enhancement processing on the structural visual fusion feature map through the structural enhancement texture inpainting network to generate a high-resolution enhanced feature map includes: The structural visual fusion feature map is subjected to convolutional downsampling processing to obtain the corresponding second downsampling feature map; The second downsampled feature map is input into the global-local Transformer block of the structure-enhanced texture restoration network. The global-local Transformer block performs global-local feature enhancement on the second downsampled feature map to obtain the corresponding texture enhancement feature map. The texture enhancement feature map is subjected to convolutional upsampling to generate the high-resolution enhancement feature map.

6. The high-resolution image restoration method as described in claim 5, characterized in that, The step of inputting the second downsampled feature map into the global-local Transformer block of the structure-enhanced texture inpainting network, and performing global-local feature enhancement on the second downsampled feature map through the global-local Transformer block to obtain the corresponding texture-enhanced feature map includes: The second downsampled feature map is normalized to obtain the corresponding second normalized feature map; A recursive gated convolution operation is performed on the second normalized feature map to obtain the corresponding initial texture enhancement feature map. The initial texture enhancement feature map and the second downsampled feature map are then added to obtain the corresponding first texture fusion feature map. The first texture fusion feature map is normalized to obtain the corresponding normalized texture fusion feature map; The normalized texture fusion feature map is input into the global-local fusion module of the global-local Transformer block. Global features are extracted through the global branch of the global-local fusion module and filtered by the gating unit. Local detail features are extracted through the local branch of the global-local fusion module. The filtered global features are combined with the local features to obtain the intermediate texture enhancement feature map. The intermediate texture enhancement feature map and the first texture fusion feature map are added together to obtain the corresponding second texture fusion feature map; The second texture fusion feature map is transformed by a third feedforward neural network, and the corresponding feature transformation result is added to the second texture fusion feature map to obtain the texture enhancement feature map.

7. The high-resolution image restoration method as described in claim 1, characterized in that, Before the step of extracting structural features from the damaged image to obtain the corresponding edge map and wireframe map, the method further includes: Obtain a training set, which includes multiple training samples and labels for each training sample. The labels for each training sample include a high-resolution target image corresponding to the training sample, a high-resolution target edge map corresponding to the training sample, and a high-resolution target wireframe map corresponding to the training sample. The training samples are low-resolution training images and training binary mask maps corresponding to the training images. The training image is input into the structure extraction network in the high-resolution image restoration model to be trained. The structure extraction network extracts structural features from the training image to obtain the corresponding training edge map and training wireframe map. The training image, the training edge map, and the training wireframe map are masked using the training binary mask map to obtain the corresponding training mask image, training mask edge map, and training mask wireframe map. The training mask image, training binary mask image, training mask edge image and training mask wireframe image are channel-concatenated to form a training multi-channel input tensor. The trained multi-channel input tensor is input into the high-resolution structural repair network in the high-resolution image repair model. The high-resolution structural repair network is used to repair the edge map and wireframe map, generating a high-resolution edge prediction map and a high-resolution wireframe prediction map. Based on the high-resolution edge prediction map and the high-resolution target edge map, calculate the first loss value corresponding to the cross-entropy loss function, and based on the high-resolution wireframe prediction map and the high-resolution target wireframe map, calculate the second loss value corresponding to the cross-entropy loss function. The first-stage loss value of the high-resolution image inpainting model is calculated based on the first loss value and the second loss value, and the parameters of the high-resolution structural inpainting network are updated based on the first-stage loss value. The training image is input into the high-resolution image restoration model, and the low-resolution training image is restored by inputting the training image into the high-resolution image restoration model to be trained, so as to obtain the high-resolution prediction image corresponding to the training image. Based on the high-resolution predicted image and the high-resolution target image, the second-stage loss value of the high-resolution image inpainting model is determined, and the parameters of the structure-enhanced texture inpainting network are updated based on the second-stage loss value. When the loss value in the second stage is less than or equal to a preset value, the high-resolution image restoration model that has been trained is obtained.

8. The high-resolution image restoration method as described in claim 7, characterized in that, The step of determining the second-stage loss value of the high-resolution image inpainting model based on the high-resolution predicted image and the high-resolution target image includes: Calculate the L1 norm loss between the high-resolution predicted image and the high-resolution target image to obtain the pixel reconstruction loss value; The high-resolution predicted image and the high-resolution target image are respectively input into the discriminator, and the discriminator outputs the corresponding true or false judgment result. Based on the true or false judgment result, the discriminator loss and generator loss are calculated, and a gradient penalty term is added to obtain the adversarial loss value. Obtain the output feature maps of the discriminator for the high-resolution predicted image and the high-resolution target image in multiple intermediate activation layers, and calculate the L1 norm loss between the output feature maps of the corresponding layers as the feature matching loss value; The high-resolution predicted image and the high-resolution target image are respectively input into a pre-trained residual network with an expanded convolutional structure to obtain the corresponding first high-level semantic feature map and second high-level semantic feature map. The feature similarity between the first high-level semantic feature map and the second high-level semantic feature map is calculated and used as the high receptive field perception loss value. The second-stage loss value is obtained by weighted summing of the pixel reconstruction loss value, the adversarial loss value, the feature matching loss value, and the high receptive field perception loss value.

9. A high-resolution image restoration device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the high-resolution image restoration method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the high-resolution image restoration method as described in any one of claims 1 to 8.

Images