An image inpainting method, system, device and medium

Abstract

The application discloses an image repairing method, system, device and medium, aiming at solving the problems of single-scale feature extraction imbalance, low global modeling efficiency, fixed repairing strategy and the like in the prior art. The method comprises the following steps: acquiring and preprocessing a to-be-repaired image and a mask; a multi-scale module extracts detailed, structural and global features to form a set; a state space model encodes the global features to model long-distance dependence with linear complexity; a dynamic mask guiding module generates an initial repairing image through mask encoding, structural complexity weight generation, dynamic feature fusion and up-sampling; and the final image is output after frequency domain texture and space domain edge optimization. The application balances texture details and global semantics, balances repairing semantic consistency and real-time performance, adapts to different regional repairing requirements, improves visual authenticity, and is suitable for old photo restoration and the like in multiple scenes.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to an image restoration method, system, device, and medium. Background Technology

[0002] Image restoration technology is an important direction in the field of computer vision and image processing. It is widely used in practical scenarios such as old photo restoration, satellite image defect filling, and medical image artifact elimination. The core requirement is to accurately fill in the missing areas in the image so that the restoration result has both semantic consistency and visual realism.

[0003] Current mainstream image restoration methods still face significant technical bottlenecks. In the feature extraction stage, most solutions employ single-scale feature extraction mechanisms, which struggle to simultaneously capture both detailed textures and global semantics. This leads to semantic breaks when restoring large areas of missing data, such as misaligned facial features in face restoration or structural distortions in architectural restoration. Regarding global dependency modeling, traditional convolutional neural networks are limited by their receptive field, making it difficult to effectively capture the relationships between distant pixels in an image. While Transformer-based methods can model long-distance dependencies, their computational complexity increases quadratically with image resolution, making them unsuitable for real-time restoration of high-resolution images.

[0004] Furthermore, existing restoration techniques often employ fixed fusion strategies, using uniform feature fusion rules for both densely textured, complex structural regions and smooth, solid-color regions. This results in restorations that are either lacking in detail or semantically contradictory to the surrounding scene, failing to meet the diverse restoration needs of different regions. These issues collectively limit the effectiveness of image restoration techniques, thus necessitating a method that balances detail preservation, semantic coherence, and processing efficiency. Summary of the Invention

[0005] The purpose of this application is to propose an image restoration method, system, device, and medium to solve the technical problems in existing image restoration technologies, such as the easy loss of details or global semantics in single-scale feature extraction, the difficulty in balancing efficiency and effectiveness in global dependency modeling, and the imbalance between semantics and details caused by fixed restoration strategies.

[0006] The technical solution of the present invention is as follows: an image restoration method, comprising the following steps: The image to be repaired and its corresponding mask are obtained, and the standardized image and multi-channel standardized mask are obtained through preprocessing. A multi-scale feature set containing detailed features, structural features, and global features is generated through a multi-scale feature extraction module. The global features are input into the state-space model-based encoding module, and global encoded features are generated through long-distance dependency modeling. Combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, the repair module generates an initial repair image by using dynamic masks; The initial repaired image is processed by a detail enhancement module that includes frequency domain texture optimization and spatial domain edge optimization, and the final repaired image is output. The dynamic mask-guided repair module includes a mask encoding unit, a structural complexity weight generation unit, a multi-source feature fusion unit, and an upsampling generation unit. The dynamic mask-guided repair module extracts mask features through the mask encoding unit, outputs adaptive fusion weights through the structural complexity weight generation unit, realizes dynamic ratio fusion of global encoded features, mask features, and detail features through the multi-source feature fusion unit, and finally restores the image resolution through the upsampling generation unit.

[0007] In one possible implementation, the preprocessing includes: Global mean-variance normalization and adaptive Gaussian filtering are performed on the image to be repaired. The kernel size of the adaptive Gaussian filter is fixed at 3×3, and the standard deviation is dynamically adjusted with the image variance. The mask is sequentially subjected to format conversion, 3×3 mean filtering edge smoothing, and channel expansion operations to generate a multi-channel standardized mask with the same number of channels as the image to be repaired.

[0008] In one possible implementation, the multi-scale feature extraction module includes a shallow extraction unit, a mid-level extraction unit, and a deep extraction unit; the shallow extraction unit extracts 128 channels of detail features from a 1 / 2 resolution normalized image, the mid-level extraction unit extracts 256 channels of structural features from a 1 / 4 resolution normalized image, and the deep extraction unit extracts 512 channels of global features from a 1 / 8 resolution normalized image; after upsampling the deep features to the mid-level and shallow-level resolutions through bilinear interpolation, they are fused element-wise with the corresponding level features to form the multi-scale feature set.

[0009] In one possible implementation, the processing procedure of the encoding module is as follows: The global features are serialized into a one-dimensional feature sequence according to the row priority rule, and coordinate position codes based on sine-cosine functions are added to the feature sequence. The feature sequence encoded by coordinate position is input into the state space model, and the global dependency is modeled through a linear state transition mechanism. The dimension of the state space model matches the number of global feature channels. The encoded feature sequence is deserialized into a two-dimensional feature map, which is then fused with the original global features through attention weighting to generate the global encoded features.

[0010] In one possible implementation, the processing procedure of the dynamic mask guidance repair module is as follows: Feature extraction is performed on the multi-channel standardized mask by the mask coding unit, and a 512-channel mask feature with the same number of channels as the global encoded feature is output. The structural complexity weight generation unit calculates the gradient magnitude of the mask coverage area based on the Sobel operator, normalizes it to the [0,1] interval after min-max normalization to generate adaptive fusion weights, and then smooths the edges of the weight map through 3×3 Gaussian filtering; The multi-source feature fusion unit achieves dynamic ratio fusion of multi-source features according to the following formula:

[0011] in, As a feature of fusion, W represents the global encoded features, and W represents the adaptive fusion weights. For mask features, For detailed features; The upsampling generation unit restores the fused features to 1 / 4 resolution, 1 / 2 resolution and original resolution through three levels of 2×2 transposed convolution. During each upsampling process, features of the corresponding scale are fused to finally generate the initial repaired image.

[0012] In one possible implementation, the processing procedure of the detail enhancement module is as follows: In the frequency domain optimization stage, a two-dimensional fast Fourier transform is performed on the initial repaired image to separate the low-frequency structural components and the high-frequency texture components. The high-frequency components are dynamically multiplied by an enhancement factor of 1.2-1.5 according to the image texture complexity, and the frequency domain enhanced image is obtained by inverse transformation. In the spatial domain optimization stage, the edge mask of the frequency domain enhanced image is extracted using the Canny operator, and Laplacian sharpening is performed only on the edge regions. The Laplacian sharpening formula is as follows:

[0013] in, These are the sharpened pixel values. To enhance the pixel values ​​of the image in the frequency domain, non-edge regions remain unchanged to output the final repaired image.

[0014] In one possible implementation, the model training process is also included, employing a combination of four loss functions for parameter optimization, with the total loss formula being:

[0015] in, To achieve region-based loss, the known regions of the image to be repaired are subjected to L1 loss, while the missing regions are subjected to L2 loss. Feature computation based on the conv3_3 and conv4_3 layers of the pre-trained VGG-19 network; Includes BCE loss for mask prediction and SSIM loss for edge regions; To repair the frequency domain Euclidean distance between the image and the original image.

[0016] An image restoration system, comprising: The image preprocessing module acquires the image to be repaired and its corresponding mask, and obtains a standardized image and a multi-channel standardized mask through preprocessing. The feature extraction module generates a multi-scale feature set containing detailed features, structural features, and global features through the multi-scale feature extraction module. The encoding feature generation module inputs the global features into the state-space model-based encoding module, and generates global encoding features through long-distance dependency modeling; The image restoration module, combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, guides the restoration module to generate an initial restored image through dynamic masks. The detail enhancement module processes the initial repaired image through frequency domain texture optimization and spatial domain edge optimization, outputting the final repaired image.

[0017] A computer device, including A memory that stores computer-readable instructions; A processor that, when executing the computer-readable instructions, implements the steps of the image restoration method.

[0018] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the image restoration method.

[0019] Compared with the prior art, this application has the following main advantages: This invention generates a multi-scale feature set encompassing details, structure, and global features through a multi-scale feature extraction module. This simultaneously captures image texture details and global scene semantics, effectively reducing semantic fragmentation issues in large-area missing image restoration and ensuring accurate positioning of core structures. By employing a state-space model-based encoding module to model the long-distance dependencies of global features, it overcomes the receptive field limitations of traditional convolutional networks and replaces the high computational cost of Transformers with linear computational complexity, achieving a balance between semantic consistency and real-time performance in high-resolution image restoration. The dynamic mask-guided restoration module outputs adaptive fusion weights through a structural complexity weight generation unit, working in conjunction with a multi-source feature fusion unit to dynamically match global encoded features with detail features. This allows the restoration strategy to accurately adapt to different regional characteristics, ensuring natural semantic integration between the restored area and the surrounding scene. Finally, a detail enhancement module, including frequency domain texture optimization and spatial domain edge optimization, specifically improves the texture clarity and edge sharpness of the restored image, further enhancing visual realism and meeting the requirements of high-precision applications. Attached Figure Description

[0020] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the method in Example 1; Figure 2 This is a flowchart of the dynamic mask-guided repair module in Example 1; Figure 3 Here is a flowchart of the state-space model encoding process for Example 1; Figure 4 This is a diagram of the image restoration system architecture for Example 2; Figure 5 This is a block diagram of the computer device structure in Example 3. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] Example 1 like Figures 1-3 As shown, this embodiment proposes an image restoration method, including the following steps: An image restoration method includes the following steps: S100. Obtain the image to be repaired and its corresponding mask, and obtain the standardized image and multi-channel standardized mask through preprocessing; Step S100 eliminates brightness and contrast differences in the image to be repaired through global mean-variance normalization, making the data distribution more consistent and providing a stable input basis for subsequent feature extraction and repair processes. The adaptive Gaussian filter adopts a design with a fixed 3×3 kernel and a standard deviation that dynamically adjusts with the image variance. This effectively suppresses image noise while avoiding the loss of details caused by excessive smoothing, ensuring the integrity of key image information. The format conversion, edge smoothing, and channel expansion operations performed on the mask not only eliminate the abruptness of the mask edges but also align the number of channels between the mask and the image to be repaired. This allows the mask to accurately guide the subsequent feature fusion and repair process, avoiding blurred repair boundaries or feature misalignment caused by mask format mismatch, and improving the boundary consistency of the repair results.

[0024] S200. Generate a multi-scale feature set containing detailed features, structural features, and global features through the multi-scale feature extraction module; The multi-scale feature extraction module in step S200 extracts detailed features, structural features, and global features for images of different resolutions through shallow, medium, and deep extraction units, respectively, thereby achieving hierarchical capture of image information.

[0025] The shallow 128-channel detail features preserve key local information such as texture and edges of the image; the mid-layer 256-channel structural features outline the core skeleton of the image; and the deep 512-channel global features grasp the overall layout and semantic relationships of the image. These three types of features complement each other, covering the information needs of different dimensions of the image. By upsampling the deep features through bilinear interpolation and fusing them element-by-element with the corresponding layer features, a complete multi-scale feature set is constructed. This effectively solves the problems of "loss of local details" or "imbalance of global structure" in single-scale feature extraction. This allows the subsequent restoration process to rely on global features to ensure overall consistency while using detail and structural features to restore local realism, avoiding the defects of fragmentation or over-smoothing in the restoration results.

[0026] S300. Input the global features into the encoding module based on the state space model, and generate global encoded features through long-distance dependency modeling; The state-space model-based encoding module in step S300 fully preserves the spatial location information of global features through feature serialization and sine-cosine coordinate position encoding, providing spatial dimension support for long-distance dependency modeling.

[0027] The state-space model employs a linear state transition mechanism, which, compared to the self-attention mechanism of the traditional Transformer, offers higher computational efficiency in modeling global dependencies and can adapt to the channel dimension of global features, achieving efficient and accurate long-distance semantic association capture. The encoded feature sequence is deserialized and then fused with the original global features using attention-weighted fusion. This enhances the consistency of global semantics while preserving the core structural information of the original global features, effectively addressing the technical challenges of inconsistent global structure and broken long-distance semantic associations in large-size image restoration. This results in restoration results that are more closely aligned with the characteristics of real images in terms of overall layout and semantic logic.

[0028] S400. Combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, the repair module generates an initial repair image by guiding the repair module through a dynamic mask. The dynamic mask-guided repair module in step S400 uses the mask features extracted by the mask encoding unit to enable the repair process to accurately identify the location and range of the area to be repaired, providing targeted guidance for feature fusion. The structural complexity weight generation unit calculates the gradient magnitude based on the Sobel operator and generates adaptive fusion weights, realizing the dynamic adjustment of fusion weights according to the structural complexity of the mask coverage area. In simple structural regions, it focuses on global encoding features to ensure overall consistency, while in complex structural regions, it focuses on detailed features to restore local realism, avoiding the problems of "insufficient details in complex regions" or "excessive redundancy in simple regions" caused by fixed weight fusion.

[0029] The dynamic matching and fusion of multi-source features achieves the organic combination of global coding features, mask features and detail features through a mathematical model, giving full play to the advantages of various features. The design of three-level transposed convolution upsampling and fusion of corresponding scale features ensures the continuity of feature information during resolution restoration, effectively suppresses the generation of artifacts during upsampling, and enables the initial restored image to restore resolution while taking into account global structural consistency and local detail integrity.

[0030] S500. The initial repaired image is processed by a detail enhancement module that includes frequency domain texture optimization and spatial domain edge optimization, and the final repaired image is output. The detail enhancement module in step S500 adopts a design that combines frequency domain and spatial domain optimization. In the frequency domain stage, the low-frequency structure component and the high-frequency texture component are separated by two-dimensional fast Fourier transform. An enhancement factor of 1.2-1.5 is dynamically applied to the high-frequency texture component according to the image texture complexity, thereby achieving precise enhancement of texture information and avoiding the noise amplification problem caused by global frequency domain enhancement.

[0031] In the spatial domain stage, edge masks are extracted using the Canny operator, and Laplacian sharpening is performed only on edge regions. This significantly improves edge clarity and sharpness while maintaining the natural texture of non-edge areas, effectively solving the problems of harsh images and increased noise caused by traditional global sharpening. The synergistic effect of the two-stage optimization results in richer texture details, clearer edge contours, and a visual effect closer to the real image, significantly improving the sensory quality and practical value of the restoration result.

[0032] Meanwhile, as the foundational guarantee for the entire process, S100 eliminates differences in image brightness, contrast, and noise through standardized processing. It also aligns the mask with the image format and channels, clearing data interference for S200 feature extraction, making the extracted details and structural features purer. It also provides precise guidance for S400 targeted repair, avoiding blurred boundaries or misaligned features, and ensuring the effectiveness of subsequent processes from the source.

[0033] S200's multi-scale feature extraction is the core feature supply link. The layered capture of detailed, structural, and global features complement each other, providing high-quality global feature raw materials for S300, giving it a solid foundation for long-distance dependency modeling, and providing multi-dimensional materials for S400's dynamic fusion. Detailed features support the repair of complex regions, structural features outline the core skeleton, and global features ensure overall coordination, solving the problem that a single feature cannot take into account both local and global aspects.

[0034] S300 efficiently models long-distance semantic associations through a state-space model, upgrading the static global features of S200 into encoded features with dynamic associations. This not only makes up for the limited receptive field of traditional convolutional networks, but also avoids high-complexity computation, providing key global constraints for S400. This ensures that the repair results do not deviate from the overall layout and semantic logic, allowing the repair of simple regions to fit the overall scene.

[0035] S400 is the core restoration hub, integrating the standardized mask of S100, the multi-scale features of S200, and the global encoding features of S300. Through dynamic weight adaptation to different regional complexities, it achieves on-demand feature fusion. Then, through three levels of transposed convolution, it restores resolution and suppresses artifacts, transforming abstract features into an initial restoration map that has both global consistency and local integrity, laying a solid foundation for the detailed optimization of S500.

[0036] The S500 focuses on improving the texture of details. Addressing the issues of texture blurring and edge blurring that may exist in the initial restoration image of the S400, it uses frequency domain dynamic texture enhancement and spatial domain directional edge sharpening to accurately fill the gaps without destroying the global structure. This results in a restored image with richer texture, sharper edges, and a visual effect that is closer to the real image.

[0037] Furthermore, the preprocessing step in step S100 includes: Global mean-variance normalization and adaptive Gaussian filtering are performed on the image to be repaired. The calculation process of global mean-variance normalization is as follows: first, the mean-variance of all images in the image to be repaired is calculated. mean of elements and variance Then through the formula Normalize each pixel.

[0038] in To prevent the use of tiny constants with a denominator of zero, the value range is 1e-5 to 1e-6.

[0039] The adaptive Gaussian filter has a fixed kernel size of 3×3, and its standard deviation is dynamically adjusted according to the image variance; specifically, the standard deviation of the adaptive Gaussian filter... Through formula calculate.

[0040] in The standard deviation is the image standard deviation, the magnitude of which matches the pixel value fluctuation (typically 10~50 for 8-bit images); k is an adjustment coefficient, ranging from 0.1 to 0.3, which can be adaptively fine-tuned according to the actual restoration scenario (e.g., low-noise images, high-noise images). =0.5; at the same time .

[0041] Then, the mask is sequentially subjected to format conversion, 3×3 mean filtering edge smoothing, and channel expansion operations to generate a multi-channel standardized mask with the same number of channels as the image to be repaired.

[0042] The format conversion process involves converting the original mask (usually a single-channel binary image with pixel values ​​of 0 or 255) into a floating-point tensor format, while normalizing the pixel values ​​to the [0,1] range, where 1 corresponds to the area to be repaired and 0 corresponds to the background area.

[0043] Channel expansion involves repeatedly copying a single-channel mask along the channel dimension until the number of channels matches the image to be repaired. If the image to be repaired is an RGB three-channel image, the single-channel mask is copied into a three-channel mask, ensuring that the mask information for each channel is completely identical, thus guaranteeing consistent guidance of image features for each channel.

[0044] Through the above preprocessing steps, global mean and variance normalization is performed on the repaired image, which can unify the brightness and contrast distribution of images from different sources and under different shooting conditions, eliminate the interference of data distribution differences on subsequent feature extraction and model training, enable the model to focus on learning the essential features of the image, and improve the convergence speed and stability of model training, avoiding training oscillations or performance fluctuations caused by discrete data distribution.

[0045] The adaptive Gaussian filter employs a fixed 3×3 core design. While ensuring filtering efficiency and controlling computational load, it achieves precise matching between noise reduction intensity and image noise level through logic that dynamically adjusts the standard deviation according to the image variance. When the image variance is large (more noise), the standard deviation is automatically increased to enhance the noise reduction effect and avoid noise interference with subsequent feature extraction. When the image variance is small (rich in detail), the standard deviation is automatically decreased to reduce the smoothing loss of image details. This achieves a dynamic balance between effectively suppressing noise and fully preserving key details such as image texture and edges, solving the problems of "over-noise reduction leading to loss of details" or "insufficient noise reduction leaving residual noise" in traditional fixed standard deviation Gaussian filters.

[0046] Finally, a format conversion operation is performed on the mask to ensure that it can adapt to the input requirements of subsequent models, thus guaranteeing the compatibility and effectiveness of data transmission.

[0047] Among them, the edge smoothing processing of 3×3 mean filtering can weaken the hard boundary between the mask and the image, avoiding the problem of abrupt transition between the repair area and the background area and obvious splicing marks caused by the abrupt edge of the mask during the subsequent repair process, making the repair boundary more naturally integrated; the channel expansion operation keeps the number of channels of the mask consistent with the number of channels of the image to be repaired, realizing the alignment of mask features and image features in the channel dimension, ensuring that the mask can accurately guide the image features of each channel during the subsequent feature fusion process, avoiding problems such as feature misalignment and guidance failure caused by channel mismatch, and improving the targeting and effectiveness of the mask in the repair process.

[0048] Furthermore, in step S200, the multi-scale feature extraction module includes a shallow extraction unit, a medium extraction unit, and a deep extraction unit; the shallow extraction unit extracts 128 channels of detail features from a 1 / 2 resolution normalized image, the medium extraction unit extracts 256 channels of structural features from a 1 / 4 resolution normalized image, and the deep extraction unit extracts 512 channels of global features from a 1 / 8 resolution normalized image.

[0049] The shallow, middle, and deep extraction units are all constructed using three convolutional layers with a kernel size of 3×3. Each convolutional layer is followed by a BatchNorm layer and a ReLU activation function. The shallow extraction unit downsamples the normalized image to half resolution using a first convolutional layer with a stride of 2. The middle extraction unit downsamples the half-resolution features to quarter resolution using a first convolutional layer with a stride of 2. The deep extraction unit downsamples the quarter-resolution features to eighth resolution using a first convolutional layer with a stride of 2. The number of output channels for each convolutional layer is the same as the number of feature channels for the corresponding unit.

[0050] Meanwhile, all convolutional layers in each extraction unit use "same padding" (padding size = (kernel size - 1) / 2) to ensure that the resolution of each layer's feature map is controlled only by the stride, avoiding feature map size distortion caused by improper padding and ensuring scale matching of subsequent upsampling fusion.

[0051] Then, the deep features are upsampled to the mid- and shallow resolutions using bilinear interpolation, and then fused element-by-element with the corresponding layer features to form the multi-scale feature set.

[0052] Specifically, deep features are upsampled to mid-level resolution via bilinear interpolation and then element-wise added to the structural features output by the mid-level extraction unit. Similarly, deep features upsampled to shallow-level resolution are element-wise added to the detail features output by the shallow-level extraction unit. During fusion, to balance the weights of features at different scales, the upsampled deep features are multiplied by a weighting coefficient of 0.6 to 0.8 before being fused with the corresponding level features, ensuring that the core information of the original level features is not overwritten. The bilinear interpolation process employs a pixel center alignment strategy, using the pixel center of the upsampled feature map as a reference to map to the original deep feature map for interpolation calculations. This avoids feature misalignment due to interpolation alignment deviations and ensures the accuracy of cross-scale fusion.

[0053] The multi-scale feature extraction module achieves layered and precise capture of image features by accurately binding shallow, medium, and deep extraction units with specific resolutions and channel numbers: the 1 / 2 resolution shallow extraction focuses on 128 channels of detailed features, which can completely preserve local fine information such as image texture and edges, providing sufficient local realism support for the restoration process; the 1 / 4 resolution medium extraction targets 256 channels of structural features, accurately outlining the core skeleton and morphological relationship of the image, avoiding structural distortion in the restoration result; the 1 / 8 resolution deep extraction uses 512 channels of global features to comprehensively capture the overall layout and semantic relationships of the image, ensuring the global consistency of the restoration result.

[0054] The design of increasing the number of channels with the extraction level enables deeper features to have richer semantic expression capabilities, while shallow features maintain higher detail fidelity, forming a complementary feature system of "detail-structure-global".

[0055] By upsampling deep features to the corresponding level through bilinear interpolation and fusing them element by element, the information barriers between features of different scales are broken down. This allows the fused multi-scale feature set to contain both global semantic guidance and local details and structural support. This effectively solves the problems of "fragmentation" or "oversmoothing" in restoration caused by isolated features and insufficient information transmission in traditional multi-scale extraction. It enables the subsequent restoration process to accurately restore local details under global structural constraints, significantly improving the overall coordination and realism of the restored image.

[0056] Further, the processing procedure of the encoding module in step S300 is as follows: S310. Serialize the global features into a one-dimensional feature sequence according to the row-major rule, and add coordinate position encoding generated based on sine-cosine functions to the feature sequence.

[0057] By serializing the global features according to the row-major rule, the original spatial arrangement relationship of the features can be retained to the greatest extent while converting the feature form to adapt to the input requirements of the state space model, which lays a foundation for subsequent deserialization to restore the two-dimensional structure and avoids spatial information disorder caused by improper serialization methods.

[0058] Specifically, the calculation process of the coordinate position encoding is as follows: for the encoding vector at the i-th position in the feature sequence , the calculation of its 2k-th dimension and 2k+1-th dimension follows the following formulas respectively:

[0059]

[0060] wherein, is the number of channels of the global feature, k is the encoding dimension index, and its value range is , i is the position index of the feature sequence, and its value range is 0≤i<H×W (H and W are respectively the height and width of the original global feature map).

[0061] The coordinate position encoding generated by the sine-cosine function can not only accurately embed the spatial position information of the feature sequence to ensure that the relative position relationship between pixels is not lost when modeling global dependencies, but also has generalization capability independent of the sequence length, which can adapt to the global feature processing of images with different resolutions, and solves the problems of insufficient generalization and high training cost of learnable position encoding in long sequence scenarios.

[0062] S320. Input the feature sequence added with the coordinate position encoding into the state space model, and model global dependencies through a linear state transition mechanism, wherein the dimension of the state space model matches the number of global feature channels; Specifically, the state space model adopts a structured state space model, its linear state transition mechanism is realized by diagonalizing the state transition matrix, and the state transition process follows the formula: ; wherein is the state vector at time t, A is a diagonalized state transition matrix (A=diag(α_1, α_2, ..., α_d), α_i is an attenuation coefficient with a value range of 0.9~0.99), B is an input projection matrix (the dimension is consistent with the number of global feature channels), The input feature vector at time t (i.e., the feature sequence elements after adding position encoding).

[0063] By employing a state-space model to model global dependencies using a linear state transition mechanism, linear computational efficiency is achieved compared to the quadratic computational complexity of traditional self-attention mechanisms. This effectively reduces computational overhead and shortens processing time when handling high-dimensional global features of large images, while avoiding convergence issues caused by excessive computational complexity. The state-space model maintains the same dimensionality as the number of global feature channels, ensuring accurate capture of feature channel-dimensional correlation information. This allows global dependency modeling to cover both long-distance spatial correlations and semantic correlations across channels, resolving the problem of fragmented or insufficient capture of spatial and channel correlations in traditional global dependency modeling.

[0064] S330. The encoded feature sequence is deserialized into a two-dimensional feature map, and then fused with the original global features through attention weighting to generate the global encoded features.

[0065] The attention-weighted fusion employs a scaled dot product attention mechanism, specifically as follows: the deserialized two-dimensional feature map is denoted as... The original global features are denoted as First, two 1×1 convolutions are used to... and Query vector Q, key vector K, and value vector V mapped to the same dimension (Q= K=V= ); Attention weight W is expressed by the formula calculate, in The dimension of the query vector (consistent with the number of global feature channels); the final global encoded features. Through formula (Residual connections) are generated to ensure gradient stability and feature information integrity during the fusion process.

[0066] Meanwhile, let the dimensions of the original global features be H×W×C (H is the feature map height, W is the width, and C is the number of channels). During serialization according to the row priority rule, the two-dimensional feature map of H×W×C is read row by row into a one-dimensional feature sequence of length L=H×W, with a sequence dimension of L×C. During deserialization, the encoded L×C one-dimensional feature sequence is rearranged into a two-dimensional feature map of H×W×C according to the original row priority order, ensuring that the feature space position is completely aligned with the original global features.

[0067] Through step S330, the encoded feature sequence is deserialized to recover the two-dimensional feature map, which can accurately restore the spatial structure of the global features and ensure the spatial consistency of subsequent feature fusion and image restoration processes. The attention-weighted fusion design with the original global features can adaptively highlight the weight of the encoded global dependency information and the key structural information in the original global features. This not only strengthens the guiding role of long-distance semantic association in the restoration process, but also avoids the loss of core structural information caused by excessive abstraction during the encoding process. This makes the generated global encoded features have both strong global consistency and high structural fidelity, providing better feature input for subsequent restoration modules and significantly improving the overall layout coordination and semantic logic rationality of the restored image.

[0068] Furthermore, the dynamic mask-guided repair module in step S400 includes a mask encoding unit, a structural complexity weight generation unit, a multi-source feature fusion unit, and an upsampling generation unit. The dynamic mask-guided repair module extracts mask features through the mask encoding unit, outputs adaptive fusion weights through the structural complexity weight generation unit, realizes dynamic ratio fusion of global encoded features, mask features, and detail features through the multi-source feature fusion unit, and finally restores the image resolution through the upsampling generation unit.

[0069] Specifically, the processing procedure of the dynamic mask-guided repair module is as follows: Feature extraction is performed on the multi-channel standardized mask by the mask coding unit, and a 512-channel mask feature with the same number of channels as the global encoded feature is output.

[0070] Specifically, the mask coding unit consists of four 3×3 convolutional layers, BatchNorm layers, and ReLU activation functions, alternating between them. The first convolutional layer maps the number of channels of the multi-channel normalized mask to 256 dimensions, and the number of channels in each subsequent convolutional layer increases to 512 dimensions. The last convolutional layer outputs 512-channel mask features. All convolutional layers use "same padding" with a stride of 1 to ensure the resolution of the mask feature map matches the global encoded features. Figure 1 To.

[0071] By performing targeted feature extraction on multi-channel standardized masks through mask coding units, 512-channel mask features with the same number of channels as global coding features are generated. This achieves precise alignment of mask features with global coding features and detail features in the channel dimension, breaking the limitation that traditional single-channel masks can only provide positional information. This allows the mask to participate deeply in the feature fusion process, providing richer semantic guidance for the repair module to the region to be repaired, and ensuring accurate positioning and targeted filling of the missing region during the repair process.

[0072] The structural complexity weight generation unit calculates the gradient magnitude of the mask coverage area based on the Sobel operator, normalizes it to the [0,1] interval after min-max normalization to generate adaptive fusion weights, and then smooths the edges of the weight map through 3×3 Gaussian filtering; The Sobel operator calculates the gradient of the mask-covered region using convolution kernels in the x and y directions, respectively. The convolution kernel in the x direction is... The convolution kernel in the y-direction is Gradient magnitude ,in This is the result of the convolution in the x-direction. The result of the convolution in the y-direction is calculated and only the gradient magnitude of the masked area (the area with a pixel value of 1) is retained, while the gradient magnitude of the background area (the area with a pixel value of 0) is set to 0.

[0073] The formula for calculating min-max normalization is:

[0074] in This is the original gradient magnitude map. This represents the minimum value of the gradient magnitude. The maximum value of the gradient magnitude is used to obtain the adaptive fusion weight map in the interval [0,1] after normalization.

[0075] The structural complexity weight generation unit calculates the gradient magnitude of the mask-covered region based on the Sobel operator. By quantifying the structural complexity of the region through gradient information, the generation of adaptive fusion weights has a clear physical meaning, avoiding the subjectivity and limitations of traditional fixed-weight fusion. Min-max normalization constrains the weights to the [0,1] interval, ensuring the interpretability of the weights and the stability of the fusion process. Meanwhile, 3×3 Gaussian filtering smooths the edges of the weight map, avoiding abrupt changes in weights that could cause discontinuities in the fusion features, resulting in a more natural transition between the repaired region and the background region. This design allows the fusion weights to dynamically adjust according to the structural complexity of the region to be repaired. In structurally simple regions, it strengthens the guidance of global encoding features to ensure overall consistency, while in structurally complex regions, it strengthens the contribution of detailed features to restore fine textures, achieving a dynamic fusion logic of "complexity adaptation."

[0076] The multi-source feature fusion unit achieves dynamic ratio fusion of multi-source features according to the following formula:

[0077] in, As a feature of fusion, W represents the global encoded features, and W represents the adaptive fusion weights. For mask features, For detailed features.

[0078] Through the multi-source feature fusion unit, the dynamic ratio fusion of global coding features, mask features and detail features is achieved by formula. It fully integrates the three major advantages of long-distance semantic association of global coding features, position-targeted guidance of mask features, and local realism support of detail features. It avoids the problems of information redundancy or missing key information in traditional feature fusion. The fused features have both global structural consistency and local detail fidelity and accurate position guidance, providing a high-quality feature foundation for the generation of the initial repair image.

[0079] The upsampling generation unit restores the fused features to 1 / 4 resolution, 1 / 2 resolution and original resolution through three levels of 2×2 transposed convolution. During each upsampling process, features of the corresponding scale are fused to finally generate the initial repaired image.

[0080] In the three-stage transposed convolution process of the upsampling generation unit, the first-stage transposed convolution restores the fused features from 1 / 8 resolution to 1 / 4 resolution, fusing 256-channel structural features from the multi-scale feature set; the second-stage transposed convolution restores the features from 1 / 4 resolution to 1 / 2 resolution, fusing 128-channel detail features from the multi-scale feature set; and the third-stage transposed convolution restores the features from 1 / 2 resolution to the original resolution, fusing preprocessed standardized image features. All fusion methods are element-wise addition operations.

[0081] The stride of each of the three 2×2 transposed convolutions is 2, and the padding is 1. Each transposed convolution is followed by a BatchNorm layer and a ReLU activation function. The first transposed convolution has 512 input channels and 256 output channels; the second transposed convolution has 256 input channels and 128 output channels; and the third transposed convolution has 128 input channels and the number of output channels is the same as the number of channels in the image to be repaired.

[0082] The upsampling generation unit adopts a three-level 2×2 transposed convolutional design to successively restore resolution. Compared with traditional single upsampling or skip upsampling, it effectively reduces the risk of artifact generation caused by resolution abrupt changes and ensures the continuity of image structure. Features of the corresponding scale are fused in each upsampling process, realizing the continuous complementarity of multi-scale features in the resolution restoration process. This allows the global semantic information in the low-resolution stage and the local detail information in the high-resolution stage to work together throughout the process, avoiding the loss or misalignment of feature information during upsampling. The final generated initial restored image achieves a good balance in resolution, global structure and local details, providing a high-quality input foundation for the subsequent detail enhancement module.

[0083] Furthermore, in step S500, the processing procedure of the detail enhancement module is as follows: S510. In the frequency domain optimization stage, a two-dimensional fast Fourier transform is performed on the initial repaired image to separate the low-frequency structural components and high-frequency texture components. Specifically, after the two-dimensional fast Fourier transform, the frequency domain spectrum is centered (the low-frequency components are moved to the center of the frequency domain image). A Gaussian low-pass filter is used to separate the high and low frequencies: the low-frequency structural components are the element-wise product of the frequency domain spectrum and the Gaussian low-pass filter, and the high-frequency texture components are the original frequency domain spectrum minus the low-frequency components. The kernel size of the Gaussian low-pass filter is consistent with the size of the initial repaired image, and the standard deviation σ = 0.05 × max(H, W) (H and W are the image height and width), ensuring that the separated high and low frequency components focus on structural and texture information respectively.

[0084] The high-frequency components are dynamically multiplied by an enhancement factor of 1.2-1.5 according to the image texture complexity, and the frequency domain enhanced image is obtained by inverse transformation; Specifically, after enhancing the high-frequency components, they are superimposed with the low-frequency components, and a two-dimensional inverse fast Fourier transform is performed to obtain image data in complex form. The real part of this data is taken as the basic data of the frequency-domain enhanced image, and then normalized using the min-max formula.

[0085] Map pixel values ​​to the range [0, 255] (where, To complete the mapping, the output frequency domain enhanced image pixel values ​​are represented by Re(·), where Re(·) denotes taking the real part. , These are the minimum and maximum values ​​of the real part, respectively, to avoid image distortion caused by pixel value overflow.

[0086] The image texture complexity is determined by the entropy value calculated using the Gray-Level Co-occurrence Matrix (GLCM). A higher entropy value indicates a more complex texture, while a lower entropy value indicates a simpler texture. The specific calculation process is as follows: For the grayscale image of the initial restoration image (or, if it's a color image, first convert it to the luminance channel in RGB space), construct a GLCM with a distance d=1 and an angle θ=0°. The formula for calculating the entropy value H is...

[0087] Where L is the number of gray levels (value 256). is the probability value at position (i,j) in the gray-level co-occurrence matrix.

[0088] The adjustment rule for the dynamic enhancement factor k is based on the texture complexity index mapping: when the texture complexity entropy value H∈[0,4] (or the gradient mean G∈[0,20]), k∈[1.2,1.3]; when H∈(4,8] (or G∈(20,50]), k∈(1.3,1.4]; when H∈(8,16] (or G∈(50,100]), k∈(1.4,1.5]; linear interpolation is used to achieve a smooth transition of factors in different complexity ranges, ensuring the continuity of the enhancement effect.

[0089] In the above steps, the frequency domain optimization stage decomposes the initial repair image into low-frequency structural components and high-frequency texture components through two-dimensional fast Fourier transform, achieving precise separation of image structure and texture information and avoiding the problem of mutual interference between structure and texture in traditional global detail enhancement. The low-frequency structural component carries the core shape and layout of the image, and maintaining its stability ensures that the overall structure of the repaired image is not distorted; the high-frequency texture component corresponds to the detailed texture and quality information of the image, and targeted enhancement processing is applied to it, which can accurately enhance the texture richness of the repaired area and solve the common problems of "texture blurring" and "texture loss" in repaired images. More importantly, the enhancement factor is dynamically adjusted in the range of 1.2-1.5 based on the image texture complexity, rather than using a fixed value, so that the texture enhancement intensity is adaptively matched with the texture density and fineness of the image itself. Complex texture areas (such as fabric texture and vegetation details) are enhanced with higher enhancement factors to enhance the detail level, while simple texture areas (such as solid color background and smooth surface) are enhanced with lower enhancement factors to avoid noise amplification caused by over-enhancement, realizing "tailor-made" texture optimization and breaking the limitations of traditional fixed intensity enhancement.

[0090] S520. In the spatial domain optimization stage, the edge mask of the frequency domain enhanced image is extracted using the Canny operator, and Laplacian sharpening is performed only on the edge regions. The Laplacian sharpening formula is as follows:

[0091] in, These are the sharpened pixel values. To enhance the pixel values ​​of the image in the frequency domain, non-edge regions remain unchanged to output the final repaired image.

[0092] When performing Laplacian sharpening on edge pixels (x=0, x=H-1 or y=0, y=W-1), missing neighboring pixels (such as x-1=-1) are filled with mirror images, i.e., I(x-1,y)=I(x+1,y) and I(x,y-1)=I(x,y+1), to ensure that the sharpening effect of edge pixels is consistent with that of internal pixels and to avoid black edges or blurring at the image boundaries.

[0093] In addition, the processing flow of the Canny operator is as follows: first, a 5×5 Gaussian filter (standard deviation σ=1.0) is applied to the frequency domain enhanced image to smooth the noise; then, the gradient magnitude and direction are calculated by the Sobel operator; the edges are refined by non-maximum suppression; finally, the edges are filtered by a double threshold method—the high threshold T1 and the low threshold T2 are generated by the Otsu adaptive threshold algorithm, satisfying T1=1.5×T2. The final edge mask is a binary image (the pixel value in the edge region is 1, and the pixel value in the non-edge region is 0).

[0094] In the above steps, the spatial domain optimization stage extracts edge masks using the Canny operator and performs Laplacian sharpening only on edge regions, completely overcoming the defects of harshness and increased noise in non-edge regions caused by traditional global sharpening. Non-edge regions retain the natural texture after frequency domain optimization, while edge regions, through a 5x center pixel weight design using the Laplacian operator, efficiently enhance edge contrast and sharpness, making image contours clearer and more distinct, solving the problem of "blurred" and "unclear" edges in the restored image. Frequency domain optimization and spatial domain optimization form a two-stage collaborative mechanism: the frequency domain stage supplements the "quantity" of texture details, while the spatial domain stage enhances the "sharpness" of edge contours. The two complement each other, ensuring both the texture realism and richness of the restored image and the clarity and three-dimensionality of the edges, effectively avoiding the one-sidedness of single frequency domain or spatial domain processing. This results in a high balance between detail fidelity, edge sharpness, and overall naturalness in the final restored image, with a visual effect closer to the real captured image.

[0095] Furthermore, it also includes the model training process, which uses a combination of four loss functions for parameter optimization. The total loss formula is:

[0096] in, To achieve region-based loss, the known regions of the image to be repaired are subjected to L1 loss, while the missing regions are subjected to L2 loss. Feature computation based on the conv3_3 and conv4_3 layers of the pre-trained VGG-19 network; Includes BCE loss for mask prediction and SSIM loss for edge regions; To repair the frequency domain Euclidean distance between the image and the original image.

[0097] The pre-trained VGG-19 network uses weights pre-trained on the ImageNet dataset. The input image size is adjusted to match the image to be repaired. During the forward propagation of the network, only the output features of the conv3_3 and conv4_3 layers are extracted, and they do not participate in the backpropagation parameter update (i.e., all network layer parameters are frozen) to ensure the stability and generalization of the perceptual loss.

[0098] Furthermore, let the original image of the image to be repaired be... The image was repaired. The repair mask is M (where M(x,y)=1 indicates a missing region, and M(x,y)=0 indicates a known region). The calculation formula is:

[0099] in, The total number of pixels in the known area. The total number of pixels in the missing region is used to ensure that the two loss weights are balanced by normalizing the number of pixels, thus avoiding loss bias caused by differences in region size.

[0100] The calculation process is as follows: The original image... With image restoration The pre-trained VGG-19 network with input freeze parameters is used to extract the output feature map of conv3_3 layer. , Compared with the output feature map of conv4_3 layer , The mean squared error (MSE) is used to calculate the distance between features in each layer, and then the features are fused according to fixed weights to obtain the final result. The formula is:

[0101] The input images for the pre-trained VGG-19 network need to be standardized first (pixel values ​​are normalized to [-1,1], consistent with the preprocessing during training), and the network is only used for feature extraction and does not participate in model parameter updates.

[0102] The loss is composed of the mask prediction BCE loss and the edge region SSIM loss, fused according to weights, as shown in the formula:

[0103] in, To repair the region mask prediction loss, the prediction mask is... Additional output from the model, via formula calculate; To compensate for the structural similarity loss in edge regions, the edge masks of the original and repaired images are extracted using the Canny operator. , Only for the edge region ( or The SSIM value is calculated using the following formula:

[0104] in, The SSIM calculation window size is 3×3, representing the total number of pixels in the edge region.

[0105] The calculation process in the L-frequency domain is as follows: First, the original image... With image restoration Perform two-dimensional Fast Fourier Transform (FFT2) to obtain the frequency domain spectrum. , Then, centering is performed on the frequency spectrum (low-frequency components are moved to the center of the frequency spectrum); then the real and imaginary parts of the frequency spectrum are separated, and the Euclidean distance is calculated separately; finally, the distances between the real and imaginary parts are combined to obtain the L-frequency domain, as shown in the formula: ,

[0106] ,

[0107]

[0108] Where Re(·) represents taking the real part of the frequency domain spectrum, Im(·) represents taking the imaginary part of the frequency domain spectrum, ||·||_2 represents the Euclidean distance, and H and W are the image height and width.

[0109] To avoid the impact of differences in the magnitude of different loss terms on training stability, each loss term needs to be min-max normalized before being substituted into the total loss formula, mapping its numerical range uniformly to the [0,1] interval. The normalization formula is as follows:

[0110] in, The original loss value, , The historical minimum and maximum values ​​of this loss term during training are updated in real time to adapt to the training dynamics.

[0111] In summary, the combination of four loss functions constructs a comprehensive constraint system from four dimensions: pixel level, semantic perception level, region guidance level, and frequency domain feature level. Through scientific weight allocation, the synergistic complementarity of each loss term is achieved, which not only ensures the basic fitting accuracy of the repair result, but also takes into account visual perception, region consistency and frequency domain feature matching, significantly improving the stability of model training and the comprehensiveness of repair effect.

[0112] As a regional loss, L1 loss is used for the known regions of the image to be repaired. Its low sensitivity to outliers can effectively maintain the smoothness and naturalness of the background pixels and avoid background distortion caused by noise interference. L2 loss is used for the missing regions. Its squared error characteristics make the model fit the pixels of the missing regions more accurately and converge faster. It solves the problem that a single loss function cannot take into account both "smoothness" and "accuracy" in the known and missing regions, and realizes differentiated optimization for different regions.

[0113] Based on the feature calculation of the pre-trained VGG-19 network's conv3_3 and conv4_3 layers, this method overcomes the limitation of pixel-level loss, which only focuses on local pixel matching. It can capture the mid-level structural features and deep semantic information of the image, ensuring that the restoration result is not only consistent with the original image at the pixel level, but also conforms to human visual perception and semantic logic. This avoids the pseudo-restoration problem of "pixel-level similarity but semantic distortion", making the overall style and structural hierarchy of the restored image highly consistent with the original image.

[0114] The fusion of BCE loss from mask prediction and SSIM loss from edge regions creates a dual constraint: BCE loss guides the model to accurately identify the boundary between the area to be repaired and the background area, avoiding offset or omission of the repair range and ensuring the targeting of the repair; SSIM loss calculates the structural similarity of the edge region, strengthens the consistency of texture and grayscale distribution in the edge region, effectively suppresses the problems of blurred, broken or abrupt transitions at the edges during the repair process, and improves the fusion between the repaired area and the background.

[0115] By calculating the frequency domain Euclidean distance between the restored image and the original image, the frequency domain feature distribution of the two is constrained to match, thus overcoming the shortcomings of traditional spatial domain loss in capturing texture and frequency information. Frequency domain constraints can ensure that the proportion of high and low frequency components in the restored image is consistent with that in the original image, avoiding visual incongruity caused by excessively smooth texture, loss of high-frequency details, or frequency domain distortion, making the texture and richness of the restored image closer to the real image.

[0116] The weight allocation of the four-fold loss fully reflects the logic of "prioritizing basic fitting, emphasizing perceptual quality, assisting with guidance constraints, and supplementing with frequency domain details": the L-structure, as the core loss, ensures pixel-level fitting accuracy; L-perception improves semantic visual quality; L-mask strengthens the consistency between regions and edges; and L-frequency domain optimizes detailed texture features. Each loss term supports each other and does not interfere with each other, effectively avoiding the one-sidedness of single or double loss constraints. This allows the model to optimize multiple key indicators simultaneously during training. The final repaired image achieves a high balance in pixel accuracy, semantic consistency, edge clarity, and texture realism, significantly outperforming the training effect of traditional single-dimensional loss functions.

[0117] This embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0118] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0120] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0121] Example 2 like Figure 4 As shown, this embodiment proposes an image restoration system, including: The image preprocessing module acquires the image to be repaired and its corresponding mask, and obtains a standardized image and a multi-channel standardized mask through preprocessing. The feature extraction module generates a multi-scale feature set containing detailed features, structural features, and global features through the multi-scale feature extraction module. The encoding feature generation module inputs the global features into the state-space model-based encoding module, and generates global encoding features through long-distance dependency modeling; The image restoration module, combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, guides the restoration module to generate an initial restored image through dynamic masks. The detail enhancement module processes the initial repaired image through frequency domain texture optimization and spatial domain edge optimization, outputting the final repaired image.

[0122] The image restoration system provided in this embodiment can realize all the processes of the image restoration method in Embodiment 1 above. The functions and technical effects of each module in the image restoration system are the same as those of the image restoration method in the above embodiment, and will not be repeated here.

[0123] Example 3 To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 5 .

[0124] The computer device 3 includes a memory 31, a processor 32, and a network interface 33 that are interconnected via a system bus. It should be noted that only the computer device 3 with components 31-33 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0125] The computer device 3 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 can interact with the user through a keyboard, mouse, remote control, touchpad, or voice control device.

[0126] The memory 31 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 31 may be an internal storage unit of the computer device 3, such as the hard disk or memory of the computer device 3. In other embodiments, the memory 31 may also be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 3. Of course, the memory 31 may include both the internal storage unit and its external storage device of the computer device 3. In this embodiment, the memory 31 is typically used to store the operating system and various application software installed on the computer device 3, such as computer-readable instructions for image restoration methods. In addition, the memory 31 can also be used to temporarily store various types of data that have been output or will be output.

[0127] In some embodiments, the processor 32 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 32 is typically used to control the overall operation of the computer device 3. In this embodiment, the processor 32 is used to execute computer-readable instructions stored in the memory 31 or to process data, for example, to execute computer-readable instructions for the image restoration method.

[0128] The network interface 33 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 3 and other electronic devices.

[0129] Example 4 This embodiment also provides another implementation method, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the image restoration method described above.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0131] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image restoration method, characterized in that, Includes the following steps: The image to be repaired and its corresponding mask are obtained, and the standardized image and multi-channel standardized mask are obtained through preprocessing. A multi-scale feature set containing detailed features, structural features, and global features is generated through a multi-scale feature extraction module. The global features are input into the state-space model-based encoding module, and global encoded features are generated through long-distance dependency modeling. Combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, the repair module generates an initial repair image by using dynamic masks; The initial repaired image is processed by a detail enhancement module that includes frequency domain texture optimization and spatial domain edge optimization, and the final repaired image is output. The dynamic mask-guided repair module includes a mask encoding unit, a structural complexity weight generation unit, a multi-source feature fusion unit, and an upsampling generation unit. The dynamic mask-guided repair module extracts mask features through the mask encoding unit, outputs adaptive fusion weights through the structural complexity weight generation unit, realizes dynamic ratio fusion of global encoded features, mask features, and detail features through the multi-source feature fusion unit, and finally restores the image resolution through the upsampling generation unit.

2. The image restoration method according to claim 1, characterized in that, The preprocessing includes: Global mean-variance normalization and adaptive Gaussian filtering are performed on the image to be repaired. The kernel size of the adaptive Gaussian filter is fixed at 3×3, and the standard deviation is dynamically adjusted with the image variance. The mask is sequentially subjected to format conversion, 3×3 mean filtering edge smoothing, and channel expansion operations to generate a multi-channel standardized mask with the same number of channels as the image to be repaired.

3. The image restoration method according to claim 1, characterized in that, The multi-scale feature extraction module includes a shallow extraction unit, a mid-level extraction unit, and a deep extraction unit. The shallow extraction unit extracts 128 channels of detail features from a 1 / 2 resolution normalized image, the mid-level extraction unit extracts 256 channels of structural features from a 1 / 4 resolution normalized image, and the deep extraction unit extracts 512 channels of global features from a 1 / 8 resolution normalized image. After upsampling the deep features to the mid-level and shallow-level resolutions using bilinear interpolation, they are fused element-wise with the corresponding level features to form the multi-scale feature set.

4. The image restoration method according to claim 1, characterized in that, The processing procedure of the encoding module is as follows: The global features are serialized into a one-dimensional feature sequence according to the row priority rule, and coordinate position codes based on sine-cosine functions are added to the feature sequence. The feature sequence encoded by coordinate position is input into the state space model, and the global dependency is modeled through a linear state transition mechanism. The dimension of the state space model matches the number of global feature channels. The encoded feature sequence is deserialized into a two-dimensional feature map, which is then fused with the original global features through attention weighting to generate the global encoded features.

5. The image restoration method according to claim 1, characterized in that, The processing procedure of the dynamic mask-guided repair module is as follows: Feature extraction is performed on the multi-channel standardized mask by the mask coding unit, and a 512-channel mask feature with the same number of channels as the global encoded feature is output. The structural complexity weight generation unit calculates the gradient magnitude of the mask coverage area based on the Sobel operator, normalizes it to the [0,1] interval after min-max normalization to generate adaptive fusion weights, and then smooths the edges of the weight map through 3×3 Gaussian filtering; The multi-source feature fusion unit achieves dynamic ratio fusion of multi-source features according to the following formula: ， in, As a feature of fusion, W represents the global encoded features, and W represents the adaptive fusion weights. For mask features, For detailed features; The upsampling generation unit restores the fused features to 1 / 4 resolution, 1 / 2 resolution and original resolution through three levels of 2×2 transposed convolution. During each upsampling process, features of the corresponding scale are fused to finally generate the initial repaired image.

6. The image restoration method according to claim 1, characterized in that, The processing procedure of the detail enhancement module is as follows: In the frequency domain optimization stage, a two-dimensional fast Fourier transform is performed on the initial repaired image to separate the low-frequency structural components and the high-frequency texture components. The high-frequency components are dynamically multiplied by an enhancement factor of 1.2-1.5 according to the image texture complexity, and the frequency domain enhanced image is obtained by inverse transformation. In the spatial domain optimization stage, the edge mask of the frequency domain enhanced image is extracted using the Canny operator, and Laplacian sharpening is performed only on the edge regions. The Laplacian sharpening formula is as follows: ， in, These are the pixel values ​​after sharpening. To enhance the pixel values ​​of the image in the frequency domain, non-edge regions remain unchanged to output the final repaired image.

7. The image restoration method according to claim 1, characterized in that, It also includes the model training process, which uses a combination of four loss functions for parameter optimization. The total loss formula is: ， in, To achieve region-based loss, the known regions of the image to be repaired are subjected to L1 loss, while the missing regions are subjected to L2 loss. Feature computation based on the conv3_3 and conv4_3 layers of the pre-trained VGG-19 network; Includes BCE loss for mask prediction and SSIM loss for edge regions; To repair the frequency domain Euclidean distance between the image and the original image.

8. An image restoration system, characterized in that, include: The image preprocessing module acquires the image to be repaired and its corresponding mask, and obtains a standardized image and a multi-channel standardized mask through preprocessing. The feature extraction module generates a multi-scale feature set containing detailed features, structural features, and global features through the multi-scale feature extraction module. The encoding feature generation module inputs the global features into the state-space model-based encoding module, and generates global encoding features through long-distance dependency modeling; The image restoration module, combining the global coding features, multi-scale feature sets, and multi-channel standardized masks, guides the restoration module to generate an initial restored image through dynamic masks. The detail enhancement module processes the initial repaired image through frequency domain texture optimization and spatial domain edge optimization, outputting the final repaired image.

9. A computer device, characterized in that, include A memory that stores computer-readable instructions; A processor that, when executing the computer-readable instructions, implements the steps of an image restoration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of an image restoration method as described in any one of claims 1 to 7.

Images