A high-precision image inpainting method

Abstract

The application discloses a high-precision image repairing method, comprising the following steps: obtaining a reference image and a to-be-repaired image, calculating an initial affine transformation, and generating a coarse alignment reference image; extracting reference features of the coarse alignment reference image and target features of the to-be-repaired image; modeling boundary uncertainty of the to-be-repaired image to obtain boundary uncertainty information; performing bidirectional feature mapping and consistency verification between the reference features and the target features to obtain a consistency verification result; performing semantic-guided local adaptive transformation to generate local adaptive transformation information; based on the three information, the reference features and the target features are cooperatively fused to generate fused features; and the to-be-repaired image is reconstructed by using the fused features. Through the multi-module cooperation of boundary probability modeling, bidirectional consistency verification and semantic adaptive transformation, the problems of harsh boundary processing, feature error propagation and local transformation mismatch in the prior art are solved, and the structural consistency and detail fidelity of the repaired image are improved.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing, and in particular to a high-precision image restoration method. Background Technology

[0002] Image restoration, which utilizes known information in an image or external prior information to reasonably infer and fill in missing areas, is a key technology in computer vision. High-fidelity, structurally consistent image restoration techniques have significant research and application value in many fields, including digital preservation of cultural relics, medical image analysis (such as lesion area completion), film and television post-production (such as wire rigging removal), and computational photography (such as de-occlusion).

[0003] To achieve image inpainting, existing technologies have evolved from traditional diffusion-based or patch-based methods (such as the PatchMatch algorithm) to generative methods centered on deep learning. In particular, inpainting models based on Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) can generate semantically plausible filling content for large-area defects. To further enhance the detail and texture realism of the inpainted images, reference image-based inpainting methods have become a research hotspot. These methods typically first estimate the global geometric transformation (such as affine transformation or homography matrix) through feature point matching (such as SIFT or ORB) to achieve coarse alignment between the reference image and the image to be inpainted. Subsequently, in the deep feature domain, effective feature information from the reference image is transferred to the defective area of ​​the image to be inpainted through feature fusion modules (such as spatial attention mechanisms, style transfer, or basic deformable convolution). Finally, the inpainted result is reconstructed through a decoder network.

[0004] However, existing reference image-based inpainting methods, especially those relying on global transformations and deterministic feature transfer, still face specific and challenging technical bottlenecks in practical applications. These problems mainly focus on handling alignment uncertainties, the reliability of feature transfer, and the rigid constraints of geometric transformations. Summary of the Invention

[0005] The purpose of this invention is to provide a high-precision image restoration method to solve the aforementioned problems existing in the prior art.

[0006] Technical solution: A high-precision image restoration method, comprising:

[0007] Obtain a reference image and the image to be repaired, calculate and apply an initial affine transformation matrix to the reference image, and generate a coarsely aligned reference image;

[0008] Extract reference features from the coarse alignment reference image and extract target features from the image to be repaired;

[0009] Model the boundary uncertainty of the image to be repaired to obtain boundary uncertainty information;

[0010] Perform bidirectional feature mapping and consistency verification between reference features and target features to obtain consistency verification results;

[0011] Based on the image to be repaired, a semantically guided local adaptive transformation is performed to generate local adaptive transformation information;

[0012] Based on boundary uncertainty information, consistency verification results, and local adaptive transformation information, reference features and target features are collaboratively fused to generate fused features;

[0013] The image to be repaired is reconstructed using fusion features to obtain the repaired image.

[0014] Beneficial effects: This invention solves the problems of rigid boundary processing, feature error propagation, and local transformation mismatch by using a multi-module collaboration of boundary probability modeling, bidirectional consistency verification, and semantic adaptive transformation, thereby improving the structural consistency and detail fidelity of the restored image. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of a high-precision image restoration method provided in an embodiment of this application.

[0016] Figure 2 A flowchart illustrating the steps for obtaining boundary uncertainty information provided in this application embodiment.

[0017] Figure 3 A flowchart illustrating the steps for calculating the multi-factor boundary confidence tensor provided in this application embodiment.

[0018] Figure 4 A flowchart illustrating the steps for obtaining consistency verification results provided in this application embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0021] The study found that existing methods generally treat the alignment of restoration boundaries as a deterministic problem. They tend to rely on a single, best-guessing offset field to guide feature sampling, ignoring the inherent high uncertainty in boundary regions with occlusion, weak texture, or significant viewpoint differences. This leads to the easy adoption of incorrect reference features in areas with difficult registration, resulting in persistent blurring, misalignment, or artifacts at the restoration boundaries. Furthermore, the feature transfer process is typically unidirectional and open-loop, meaning features only flow from the reference image to the image to be restored, lacking an effective mechanism to verify the correctness and reliability of the transferred features. When initial alignment deviates (e.g., the grass texture of the reference image is incorrectly mapped to the water surface area of ​​the image to be restored), this feature conflict cannot be explicitly detected and reconciled. Errors passively propagate and amplify along the decoding path, leading to severe semantic or structural mismatches in the restoration results. Moreover, most methods rely on a single global affine transformation to align the entire reference image. This one-size-fits-all transformation model cannot adapt to the different geometric constraints that different semantic regions (e.g., buildings and sky) in the image should follow. For example, building areas should conform more to rigid or similar transformations, while global affine transformations may introduce unnecessary shear; sky or vegetation areas may require more flexible non-rigid deformations. This globally uniform transformation model leads to structural distortions and unnatural deformations at the boundaries of different semantic regions.

[0022] For example, in image inpainting tasks based on reference images, the core challenge lies in the fact that there are often geometric and appearance differences between the reference image and the image to be inpainted, such as viewpoint, scale, and lighting. If they are not accurately aligned, the effective information from the reference image will not only fail to assist in the inpainting but will also introduce incorrect textures or structures, resulting in obvious artifacts or stitching marks in the inpainting result. A basic approach is to obtain the keypoint sets of the reference image and the image to be inpainted through feature point detection and matching algorithms (such as Scale Invariant Feature Transform (SIFT) or SuperPoint feature extraction). Robust estimation algorithms such as RANSAC (Random Sample Consensus) are then used to solve for the global affine transformation matrix from the matched keypoint pairs. This matrix is ​​then applied to the reference image to generate a coarsely aligned reference image that is roughly aligned with the image to be inpainted in the coordinate system. In the feature fusion stage, the conventional approach is to input the coarsely aligned reference image and the image to be inpainted (and its defect mask) together into a deep neural network, such as the Dual U-Net architecture. In this architecture, the encoder extracts multi-scale depth features, while the fusion module (e.g., deformable convolution or multi-head attention mechanism) attempts to align and transfer information at the feature level, and finally the decoder reconstructs the repaired image. However, this basic framework has several technical drawbacks: traditional alignment methods treat boundaries as deterministic geometric transformations, resulting in obvious stitching marks in boundary regions where registration is difficult or occlusion exists; feature fusion is usually unidirectional (from reference to target), lacking a reverse verification mechanism for alignment quality, making it impossible to detect and correct erroneous feature propagation; a single global affine transformation ignores the different local deformations that may exist in different semantic regions (e.g., sky, buildings, vegetation), limiting the accuracy of alignment. Figure 1 As shown, a high-precision image restoration method is proposed, including the following steps:

[0023] Obtain a reference image and the image to be repaired, calculate and apply an initial affine transformation matrix to the reference image, and generate a coarsely aligned reference image.

[0024] In other words, obtain the reference image and the image to be repaired, and calculate the initial affine transformation matrix; apply the initial affine transformation matrix to the reference image to generate a coarsely aligned reference image.

[0025] Specifically, load the original reference image R raw And the original image to be repaired T raw The image is then standardized (e.g., normalized to the [0, 1] interval) to obtain a standardized reference image R and a standardized image to be repaired T. Simultaneously, a repair mask M is generated to mark the areas to be repaired. Preferably, the mask M is processed through morphological operations (e.g., dilation) to obtain an extended mask MT. dilatedThis is used for subsequent boundary analysis. A feature point detector (e.g., the SuperPoint detector) is applied to the non-masked regions (labeled by 1-M) of the normalized reference image R and the normalized image to be repaired T. This detector extracts feature points at multiple scale levels, resulting in a reference feature point coordinate set {p}. R i} and its descriptor {d R i}, and the target feature point coordinate set {p T j} and descriptor {d T j Initial feature point correspondences are established through descriptor matching (e.g., nearest neighbor matching). Based on these correspondences, the RANSAC algorithm is used to iteratively estimate the affine transformation, and the transformation with the most interior points is selected as the initial affine transformation matrix A. init For example, to improve transformation accuracy, the transformation parameters are optimized using the least squares method using all feature point pairs identified as interiors by RANSAC, resulting in a refined affine transformation matrix A. refined In subsequent steps, the initial affine transformation matrix can preferably be a refined affine transformation matrix A. refined The refined affine transformation matrix A refined Apply to the normalized reference image R to generate a coarsely aligned reference image R'.

[0026] Extract reference features from the coarsely aligned reference image and extract target features from the image to be repaired.

[0027] In this embodiment, the coarse-aligned reference image R' and the normalized image to be repaired T (along with the mask M) are simultaneously input into an encoder with shared parameters, such as a variational autoencoder (VAE). This encoder extracts depth feature representations at different resolutions through multiple convolutional and downsampling operations, thereby obtaining the reference feature pyramid {F}. R l} (l=1 to L) and target feature pyramid {F T l} (l=1 to L), where L is the number of feature layers. Typically, low-level features (e.g., l=1, 2) preserve high-frequency texture details, mid-level features (e.g., l=3, 4) encode structural information, and high-level features (e.g., l=5, 6) contain abstract semantic information.

[0028] Model the boundary uncertainty of the image to be repaired to obtain boundary uncertainty information.

[0029] In this embodiment, boundary alignment is transformed from a deterministic problem into a probabilistic one. Specifically, a multi-factor boundary confidence tensor is constructed by analyzing the confidence levels across three dimensions: geometric, semantic, and structural. This tensor is then fused to generate a comprehensive uncertainty map, which quantifies the registration reliability of each boundary pixel. The comprehensive uncertainty map is combined with target features, and a probabilistic migration prediction network is used to generate distribution parameters (e.g., mean and covariance) of the probabilistic migration field. These parameters collectively constitute the boundary uncertainty information.

[0030] Perform bidirectional feature mapping and consistency verification between reference features and target features to obtain consistency verification results.

[0031] In this embodiment, a feature transfer verification mechanism is established. Specifically, sampling is performed using probability offset field parameters to achieve a forward mapping from reference features to target features (forward aligned features). Simultaneously, the inverse transformation of the initial affine transformation is calculated to achieve a reverse mapping from target features to reference features (reverse mapped features). By comparing the original features with the round-trip features after reverse-forward mapping, the forward consistency error is calculated; similarly, the reverse consistency error is calculated. Based on this bidirectional error, hierarchical conflict detection (e.g., texture, structural, and semantic conflicts) is performed, and adaptive fusion weights are generated. These together constitute the consistency verification result.

[0032] Based on the image to be repaired, a semantically guided local adaptive transformation is performed to generate local adaptive transformation information.

[0033] Specifically, semantic segmentation is performed on the image to be restored to obtain semantic segmentation maps that identify multiple semantic categories (such as sky, buildings, etc.). For each semantic category, specific semantic features are extracted, and unique local transformation correction parameters are predicted. These correction parameters are applied to the initial affine transformation matrix to obtain a category-specific transformation matrix. Interpolation is performed at the boundaries of different semantic categories using a smooth transition function to synthesize a spatially continuous adaptive transformation field, which is the local adaptive transformation information. This local adaptive transformation information overcomes the limitations of global transformation.

[0034] Based on boundary uncertainty information, consistency verification results, and local adaptive transformation information, reference features and target features are collaboratively fused to generate fused features.

[0035] Specifically, a comprehensive fusion weight is generated by integrating information on overall uncertainty (e.g., using its complement to represent confidence), consistency verification results (e.g., adaptive fusion weights), and transformation information (e.g., boundary transition weights). This weight is used to modulate the subsequent feature fusion process. For example, in an attention mechanism, the target feature is used as the query, and the reference features after secondary correction by the adaptive transformation field are used as the key and value. The calculated attention result is then multiplied by this comprehensive fusion weight to finally generate the fused feature.

[0036] The image to be repaired is reconstructed using fusion features to obtain the repaired image.

[0037] For example, the fusion feature F fused l The input is fed into the decoder. The decoder progressively reconstructs image details by combining features (or fused features) from the corresponding level of the encoder with layer-by-layer upsampling (e.g., transposed convolution) and skip connections. Preferably, post-processing operations can be performed before outputting the final image. For example, color statistics (mean and standard deviation) matching can be performed on the repaired region of the repaired image and the non-masked region of the original image to perform color uniformity correction. Another example is in the extended mask M... dilated Poisson fusion or gradient domain fusion is applied to the defined boundary regions to further eliminate potential stitching artifacts, ultimately outputting a high-quality inpainted image I. final .

[0038] like Figure 2 As shown, in an exemplary embodiment, obtaining boundary uncertainty information specifically includes:

[0039] For the boundary region of the image to be repaired, calculate the multi-factor boundary confidence tensor.

[0040] In this embodiment, the multi-factor boundary confidence tensor (denoted as B) preferably includes three dimensions, corresponding to geometric confidence, semantic confidence, and structural confidence, respectively.

[0041] like Figure 3 As shown, in a further embodiment, calculating the multi-factor boundary confidence tensor includes:

[0042] Evaluate the local registration error associated with the initial affine transformation matrix and determine the geometric confidence level.

[0043] Specifically, for each pixel (x, y) in the boundary region, find its K nearest matching feature point pairs in its neighborhood. Based on these matching point pairs, perform a refined affine transformation matrix A. refined The actual deviation under transformation is used to calculate the local registration error statistic ε. localFurthermore, spatial distance weighting can be applied to the registration error within the neighborhood (e.g., using a Gaussian kernel function exp(-||p-(x,y)||). 2 / σ spatial 2 ), where p is the position of a matching feature point pair in the neighborhood, (x, y) are the pixel coordinates of the current boundary region to be evaluated, and σ spatial (The scale parameter is used to weight the spatial distance), thus obtaining the spatial registration error ε of the pixel. spatial (x, y). Map this error value through a negative exponent (e.g., geometric confidence B). geo = exp(-ε spatial 2 / σ geo 2 ), where σ geo (The scale parameter used to map spatial registration errors to geometric confidence is converted to geometric confidence in the interval [0, 1].)

[0044] The similarity between reference features and target features in a predefined high-level semantic dimension is measured to obtain semantic confidence.

[0045] In this embodiment, high-level semantic features (e.g., F) are selected from the feature pyramid. R 5 F R 6 F T 5 F T 6 For each boundary pixel location, a multi-scale reference feature vector v is obtained through interpolation sampling. R and target feature vector v T Calculate the cosine similarity (e.g., cosine similarity) between them. sim 5 and cos sim 6 , representing the cosine similarity between the reference image feature vector and the target image feature vector in the 5th and 6th level feature pyramids. The similarities from different levels are weighted and fused (e.g., s). init =α5×cos sim 5 +α6×cos sim 6 ), where α5 and α6 are learnable weights. Preferably, to enhance spatial smoothness, the obtained similarity graph s can be... init Applying 3x3 median filtering, the semantic confidence map B is finally generated. sem .

[0046] The structural confidence is obtained by analyzing the gradient consistency between the coarsely aligned reference image and the image to be repaired.

[0047] Specifically, the gradients (e.g., Sobel gradients) of the coarse-aligned reference image R' and the normalized image to be repaired T are calculated respectively to obtain the gradient magnitude map (G). R_mag G T_mag ) and gradient pattern (G R_ori G T_ori ), where G R_mag G T_mag The gradient magnitude maps are shown for the coarsely aligned reference image R' and the normalized image to be repaired T, respectively; G R_ori G T_ori These are the gradient direction maps of the coarsely aligned reference image R' and the normalized image to be repaired T, respectively. Within the strip regions on both sides of the boundary, their gradient histograms are compared (e.g., using EMD, Earth Mover's Distance, to measure the gradient distribution distance d). grad ), and calculate the consistency of gradient directions (e.g., gradient direction consistency metric coherence = cos(G)). R_ori - G T_ori Through nonlinear functions (e.g., B) str =f struct (d grad coherence) = exp(-β1×d grad )×(1+β2×coherence) / 2, where f struct (The fusion function, with β1 and β2 as different weights), fuses these metrics to generate the structural confidence map B. str .

[0048] The geometric confidence, semantic confidence, and structural confidence are combined to form a multi-factor boundary confidence tensor.

[0049] Specifically, B geo B sem and B str The three confidence maps are stacked along the channel dimension to form the original three-dimensional confidence tensor B. raw Preferably, in order to balance the differences in numerical range and importance among the three dimensions, the three-dimensional confidence tensor B can be optimized. raw Each channel undergoes adaptive normalization, for example, z-score normalization, and an adaptive scaling factor γ is calculated based on the difference in confidence distribution between the boundary region and the non-boundary region. c By scaling and truncation operations, the final boundary confidence tensor B is output.

[0050] By fusing the multi-factor boundary confidence tensor, a comprehensive uncertainty graph is generated.

[0051] In this embodiment, the three-dimensional confidence tensor B is fused into a scalar comprehensive uncertainty map U (the value range of U is typically [0, 1], with larger values ​​indicating greater uncertainty, which is the opposite of confidence). Specifically, tensor B is input into a learnable fusion network, such as a multilayer perceptron (MLP). Preferably, to capture spatial correlations, the network can employ multi-scale convolution, channel attention (SE-Net), and spatial attention modules to enhance feature representation. Further, to ensure spatial smoothness, a pixel map can be constructed, edge weights can be defined using feature similarity, and uncertainty information can be propagated through random walks or graph Laplacian smoothing. Boundary distance-weighted smoothing can be applied (e.g., using a small kernel near the boundary and a large kernel for Gaussian filtering at the far boundary) to eliminate internal noise while maintaining boundary sharpness, outputting the final comprehensive uncertainty map U.

[0052] By combining the comprehensive uncertainty map with the target features, the distribution parameters of the generated probability offset field are predicted.

[0053] In this embodiment, based on the uncertainty U, the probability distribution P(ΔP|U, F) of the predicted pixel offset ΔP is calculated. T Specifically, from the target feature pyramid {F} T l Aggregating multi-scale contextual features in} (e.g., F T 2 F T 3 F T 4 Align and merge to obtain aggregated context features F context The integrated uncertainty graph U and the aggregated context features F will be combined. context The data is concatenated along the channel dimension and fed as input into the Probability Offset Prediction Network (ProbOffsetNet). This network can employ a Conditional Variational Autoencoder (C-VAE) structure, using an encoder-decoder architecture to predict the parameters of the offset distribution. It's important to note that since the pixel offset ΔP is a two-dimensional vector (containing offsets in both the x and y directions), its probability distribution (e.g., a Gaussian distribution) is described by a two-dimensional mean vector μ(x, y) and a 2x2 covariance matrix Σ(x, y). In the implementation, the network can directly output μ(x, y) and a parameterized form of the covariance matrix Σ(x, y) (e.g., outputting the logarithmic standard deviation of the diagonal elements, or the lower triangular elements of the Cholesky decomposition) to ensure the positive definiteness of the covariance matrix.

[0054] By combining the distribution parameters of the comprehensive uncertainty map and the probability offset field, boundary uncertainty information is generated.

[0055] Alternatively, the boundary uncertainty information can be described as including the distribution parameters {μ, Σ} of the comprehensive uncertainty map U and the probability offset field.

[0056] In yet another exemplary embodiment, it further includes:

[0057] An offset field instance is generated by sampling from the probability distribution defined by the distribution parameters of the probability offset field.

[0058] Specifically, during the training phase, reparameterization techniques are used for differentiable sampling, such as ΔP. sample =μ+ L×ε, where L is the Cholesky decomposition of the covariance matrix Σ (Σ= L×L) T , T (where ε is the transpose), ε is random noise sampled from the standard normal distribution N(0, I), ΔP sample This is a specific instance of a offset field sampled from the distribution of the probability offset field. During the inference phase, an adaptive sampling strategy is preferably employed. For example, based on the value of the comprehensive uncertainty map U, K is sampled in high uncertainty regions (e.g., U > 0.7). max Times (e.g., K) max =10) sampling, in low uncertainty regions (e.g., U < 0.3), directly use the mean μ (i.e., K=1 sampling). Generate one or more offset field instances ΔP k .

[0059] The offset field instance is applied to the reference feature to generate the forward aligned feature.

[0060] Specifically, the offset field instance ΔP k (Interpolation to the resolution of the corresponding feature layer l is required) Applied to the reference feature F R l This application process is preferably implemented through deformable convolution, where ΔP k Used to guide the sampling offset of deformable convolution kernels. When K samplings are performed (K>1), K sets of aligned features {F} are obtained. aligned l ,k}. At this point, statistical analysis can be performed on these K sets of results, for example, calculating their characteristic mean μ. feat and variance σ 2 feat The variance-weighted average (i.e., assigning higher weights to samples with high alignment stability) or the direct mean is used to fuse the samples into a single, robust forward alignment feature F. ref→target l .

[0061] The consistency verification result is generated based on the forward alignment feature. That is, the forward alignment feature will be used as one of the inputs for the consistency verification.

[0062] like Figure 4 As shown, according to one aspect of this application, obtaining a conformance verification result includes:

[0063] Perform forward mapping to align the reference features to the preset target coordinate space and generate forward-aligned features.

[0064] Specifically, the forward alignment feature is denoted as F. ref_to_target l Where l represents the feature level. Forward aligned feature F ref_to_target l The probability offset field instance ΔP can be applied. k Based on the original reference feature F R l Then, it is obtained through deformable convolution and multisampling fusion.

[0065] Calculate the inverse transformation matrix of the initial affine transformation matrix.

[0066] Specifically, obtain the refined affine transformation matrix A. refined This matrix is ​​typically a 3x3 homogeneous coordinate matrix or a 2x3 affine matrix. Its inverse transformation matrix A is obtained by performing the standard matrix inversion operation. inv During the calculation process, numerical stability checks are required, such as checking the refined affine transformation matrix A. refined If the determinant is close to zero, then a pseudo-inverse can be used instead of the standard inverse matrix to avoid numerical overflow.

[0067] The inverse transformation matrix is ​​applied to perform inverse mapping, aligning the target features to a preset reference coordinate space and generating inverse mapped features.

[0068] In this embodiment, the inverse transformation matrix A is... inv Applied to target feature pyramid {F T l A preferred implementation is based on the inverse transformation matrix A. inv The basic inverse transformation migration field ΔP is calculated at different feature levels l. inv l The inverse transformation offset field ΔP inv l As an offset, it is applied to the target feature F through deformable convolution. T l This generates the inverse mapping feature (denoted as F). target_to_ref l ).

[0069] Calculate the forward consistency error between the reference feature and the reverse-mapped feature.

[0070] In this embodiment, it is verified whether the original reference features can be recovered after mapping the target back to the reference space and then applying the original forward offset. Specifically, the probability offset field instance ΔP is used. k (or its fusion mean) is applied to the inverse mapping feature F target_to_ref l The round-trip feature F is obtained. roundtrip_R l Calculate the original reference feature F. R l With the round-trip feature F roundtrip_R l The difference between them is taken as the forward consistency error C. forward l C forward l = Distance(F R l F roundtrip_R l ); where Distance(·) can be a distance metric function, such as L2 norm (Euclidean distance), L1 norm (Manhattan distance), or 1 minus cosine similarity.

[0071] Calculate the reverse consistency error between the target feature and the forward alignment feature.

[0072] In this embodiment, it is verified whether the original target features can be recovered after forward mapping from the reference space to the target space and then applying a reverse offset. Specifically, the basic inverse transformation offset field ΔP is... inv l Applied to forward alignment feature F ref_to_target l The round-trip feature F is obtained. roundtrip_T l Calculate the original target features F. T l With the round-trip feature F roundtrip_T l The difference between them is taken as the reverse consistency error C. backward l C backward l = Distance(F T l F roundtrip_T l ).

[0073] Hierarchical conflict detection is performed based on forward consistency error and reverse consistency error.

[0074] Specifically, the forward consistency error C forward l and reverse consistency error C backwardl (or a combination of both, C) total l This is transformed into a judgment on the credibility of features.

[0075] In one optional implementation, performing hierarchical conflict detection includes:

[0076] Forward consistency error and backward consistency error are analyzed at the low-level feature scale to identify texture conflicts.

[0077] Specifically, low-level features (e.g., l=1, 2) mainly contain high-frequency texture information. When the low-level error C total 1 Or C total 2 Exceeding the preset texture threshold θ texture When this occurs, it is determined to be a texture conflict. A preferred analysis method is to perform frequency domain analysis: apply a two-dimensional discrete cosine transform (DCT) to local blocks (e.g., 8x8 pixels) of the low-level error map. Analyze its frequency domain coefficients; if the energy proportion of the high-frequency alternating current (AC) coefficients exceeds the high-frequency threshold θ... high_freq (For example, 0.4) indicates that there is a significant inconsistency between the reference feature and the target feature in terms of detail texture (i.e., texture conflict), and a texture conflict flag Conflict is generated. texture .

[0078] Analyze forward consistency error and backward consistency error at the mid-level feature scale to detect structural conflicts.

[0079] Specifically, mid-level features (e.g., l=3, 4) primarily encode contour and structural information. When C total 3 Or C total 4 Exceeding the structural threshold θ structure When this occurs, it is determined to be a structural conflict. A preferred analysis method is to perform edge response detection: [the detection is performed on the original reference feature F]. R 3 and forward alignment feature F ref_to_target 3 Applying an edge detector (e.g., the Canny operator) yields two edge maps E. R and E align Calculate the Hausdorff distance d between the two edge graphs. H If Hausdorf is far from d H Exceeding the distance threshold θ hausdorff Or, the difference in the number of structural elements such as lines and curves detected by Hough transform Δn struct Exceeding the difference threshold θ struct_diffThis indicates that the key structures of the image have been misaligned or deformed (i.e., structural conflict), generating a structural conflict flag. structure .

[0080] Analyze forward consistency error and backward consistency error at a high-level feature scale to determine semantic conflicts.

[0081] Specifically, high-level features (e.g., l=5, 6) primarily contain abstract semantic information. When C total 5 Or C total 6 Exceeding the semantic threshold θ semantic When this occurs, it is determined to be a semantic conflict. A preferred analysis method is to compare feature distributions: in the feature space, calculate the original reference feature F. R 5 Local probability distribution P R Alignment feature F with forward ref_to_target 5 Local probability distribution P align The KL divergence between them (D) KL If the KL divergence D KL Exceeding the divergence threshold θ KL (For example, 0.5) indicates that the semantic category of the reference image (e.g., tree) has been incorrectly aligned to another semantic category of the target (e.g., building) (i.e., semantic conflict), generating a semantic conflict flag Conflict. semantic Another alternative analysis approach is to use the original reference feature F R 5 and forward alignment feature F ref_to_target 5 If the predicted category labels are inconsistent when inputting into the pre-trained semantic classifier, it is considered a semantic conflict.

[0082] The results of hierarchical conflict detection include texture conflicts, structural conflicts, and semantic conflicts.

[0083] In other words, based on texture conflict, structural conflict, and semantic conflict, hierarchical conflict detection results are generated.

[0084] Specifically, the texture conflict flag will be... texture Structural conflict marker Conflict structure and semantic conflict flag Conflict semantic Integrate and generate a conflict type graph. mapPreferably, the integration process follows a priority determination rule: semantic conflicts have the highest priority, followed by structural conflicts, and finally texture conflicts. That is, if a pixel location is marked as both a semantic and structural conflict, then it is prioritized in the conflict type graph. map The final label in the code is recorded as a semantic conflict.

[0085] Based on the results of hierarchical conflict detection, a consistency verification result is generated.

[0086] In other words, the consistency verification results include the results of hierarchical conflict detection. Specifically, this includes the conflict type graph. map It is one of the outputs of the consistency verification results.

[0087] In a further embodiment, obtaining the consistency verification result further includes:

[0088] The basic consistency weight is calculated based on the forward consistency error and the reverse consistency error.

[0089] Specifically, the forward consistency error C forward l and reverse consistency error C backward l The total error C is obtained by weighted summation across all levels l. total The underlying consistency weight W is calculated using a negative exponential transformation. base :W base = exp(-λ* C total ); where λ is an adjustable scaling factor, and the total error C total The larger the value (the greater the error), the greater the basic consistency weight W. base The closer to 0, the lower the confidence level.

[0090] A differentiated weight adjustment strategy is applied to adjust the basic consistency weights and generate adaptive fusion weights. The weight adjustment strategy varies depending on whether the conflict type is texture conflict, structural conflict, or semantic conflict.

[0091] In this embodiment, the conflict type graph Conflict is used. map To refine the weights. Specifically, apply the following rule: if the conflict type graph is... map If a semantic conflict is marked at position (x, y), then the adaptive fusion weight W at that position is forced to be applied. consistency (x, y) is set to extremely low values ​​(e.g., 0 or 0.1) to indicate that the reference features are not trusted at all; if it is marked as a structural conflict, the fusion weight W will be used. consistency (x, y) is set to a medium value (e.g., 0.5) to indicate partial trust; if marked as texture conflict or no conflict, the fusion weight W is used.consistency (x, y) can be set to a high value (e.g., 0.8) or the base weight W can be used directly. base (x, y).

[0092] By combining the results of adaptive fusion weights and hierarchical conflict detection, a consistency verification result is generated.

[0093] In other words, the consistency verification result further includes adaptive fusion weights. That is, the adaptive fusion weights W consistency It is one of the final outputs of the consistency verification results and will be used for subsequent collaborative fusion.

[0094] In one exemplary embodiment, generating local adaptive transformation information includes:

[0095] Semantic segmentation is performed on the image to be repaired to obtain a semantic segmentation map, which identifies a predetermined number of semantic categories.

[0096] Specifically, the standardized image T to be repaired is input into a pre-trained deep learning semantic segmentation network. Such networks can be DeepLabV3+, SegFormer, or Mask R-CNN, etc. The deep learning semantic segmentation network outputs a pixel-level semantic label map S. map Each pixel is assigned a semantic category label, such as {0: sky, 1: building, 2: road, 3: vegetation, 4: water, 5: object}. Preferably, to improve the robustness of subsequent processing, the semantic label map S can be... map Post-processing: Generate a corresponding binary mask M for each category c. class_c By applying connected component analysis, isolated small regions (fragments) with an area smaller than a certain threshold (e.g., 100 pixels) are removed to obtain the cleaned semantic mask M. class_c_clean .

[0097] For each semantic category, specific local transformation correction parameters are predicted by combining the target features.

[0098] In this embodiment, the geometric transformation adjustment amount is customized for each semantic category learning.

[0099] In a preferred implementation, predicting specific local transformation correction parameters includes:

[0100] Using a semantic segmentation map as a mask, features corresponding to semantic categories are extracted and aggregated from the target features to form a semantically specific feature representation.

[0101] In this embodiment, the cleaned semantic mask M is preferably applied. class_c_clean For masking. Specifically, from the target feature pyramid, select one or more mid-to-high-level features (e.g., F... T4 For category c, use its semantic mask M. class_c_clean For middle and high-level characteristics F T 4 Perform operations to aggregate features. There are several aggregation methods, including: Method 1: Using masked average pooling, this category c is aggregated into the mid-to-high-level features F. T 4 The global category feature g is obtained by averaging all feature vectors on the vector. class_c Method 2: Retain all feature maps within the masked area. class_c The features are then encoded using a small convolutional network. A preferred approach is hybrid aggregation, where the global features g from method one are used... class_c The local features encoded using method two are concatenated to form a rich semantically specific feature representation F that contains both global statistical information and local spatial layout. semantic_rep (c)

[0102] Input semantically specific feature representations into a pre-configured transform prediction network to generate dedicated local transform correction parameters.

[0103] Specifically, a transform prediction network (SemanticTransformNet) is constructed, which can consist of multiple fully connected layers (MLP) or convolutional layers. The semantically specific feature representation F... semantic_rep (c) is fed into the transform prediction network as input. Optionally, the learnable class embedding vector corresponding to class c can also be input simultaneously. c This enhances class differentiation. The network output is the local transformation correction parameter ΔA specific to class c. c Since affine transformations have 6 degrees of freedom (2x3 matrix) in 2D space, the local transformation correction parameter ΔAc is a 6-dimensional vector: [Δa 11 , Δa 12 , Δa 21 , Δa 22 , Δt x , Δt y ], where Δa 11 To control the components of horizontal scaling and partial rotation, Δa 12 To control the components of horizontal shear and partial rotation, Δa 21 To control the components of vertical shear and partial rotation, Δa 22 To control the components of vertical scaling and partial rotation, Δt x To control the horizontal translation component, Δt yTo control the vertical translation component, further constraints can be added during network training or inference to ensure the physical rationality of the transformation. For example, for the sky category, the rotation and shear components (Δa12, Δa21) in its correction parameters should be constrained to a small range; for the building category, its vertical shear component should be strictly constrained to maintain the verticality of the lines.

[0104] By applying the local transformation correction parameters to the initial affine transformation matrix, a category-specific transformation matrix for the semantic category is obtained.

[0105] Specifically, obtain the refined affine transformation matrix A. refined For each semantic category c, its specific local transformation correction parameter ΔA is... c With the refined affine transformation matrix A refined Performing composition (e.g., matrix addition or multiplication, depending on A) refined and ΔA c (Definition method), to obtain the category-specific transformation matrix A of the semantic category. local (c). A local (c) =A refined + ΔA c This A local (c) is the local affine transformation that best suits the semantic category c.

[0106] Based on category-specific transformation matrices and semantic segmentation graphs, a spatially continuous adaptive transformation field is synthesized by performing smooth transitions at semantic boundaries.

[0107] In this embodiment, discrete (one for each category) category-specific transformation matrices A are used. local (c) , fused into a smooth, pixel-level transform field A adaptive_field This field defines its own transformation parameters for each pixel (x, y) in the image.

[0108] In a further implementation, the synthesized spatially continuous adaptive transformation field includes:

[0109] Extract semantic boundaries from semantic segmentation graphs.

[0110] Specifically, for semantic label graph S map By applying edge detection operators (such as the Sobel operator or morphological gradient operations), a semantic boundary map E is obtained, which identifies the boundaries between different semantic categories. semantic .

[0111] Calculate the distance from each pixel to the nearest semantic boundary and generate a distance map.

[0112] Specifically, for the semantic boundary graph E semanticThe Euclidean Distance Transform (EDT) algorithm is applied. This algorithm outputs a distance map D. distance_map The value D(x,y) of each pixel (x,y) represents the Euclidean distance from that pixel to the nearest semantic boundary.

[0113] A transition function is constructed based on the distance graph, and a transition weight graph is generated.

[0114] In this embodiment, weights are defined that are smooth at the boundaries and saturate within the region. Specifically, a Sigmoid function (or Gaussian function) is preferably used as the transition function: W transition (x, y) = 1.0 / (1.0 + exp( -(D(x, y) - d threshold ) / τ)); where D(x, y) are the values ​​from the distance map; d threshold It is a distance threshold that controls the center position of the transition zone (e.g., d). threshold =5 pixels); τ is a temperature coefficient that controls the smoothness of the transition (the smaller the τ, the sharper the transition). Transition weight W transition The value of (x, y) is close to 1 (when the pixel is far from the boundary) or smoothly transitions to 0.5 near the boundary. Alternatively, the transition function can be adaptive: a smaller τ value is used for boundaries with large semantic differences (e.g., sky and buildings) to make the transition sharp; a larger τ value is used for boundaries with small semantic differences (e.g., roads and sidewalks) to make the transition smooth.

[0115] By applying a transition weight map, weighted interpolation is performed on the category-specific transformation matrix to synthesize an adaptive transformation field.

[0116] Specifically, for each pixel (x, y) in the image, its final transformation parameter A adaptive_field (x, y) (a 6-dimensional vector), through a category-specific transformation matrix A of its class c. local (c) and the category-specific transformation matrix A of neighboring categories local (c') is obtained by weighted interpolation. An alternative implementation is to utilize semantic mask M. class_c_clean and transition weight map W transition The transformation parameters at each pixel (x, y) are calculated using a weighted average (e.g., using a Gaussian kernel or a bilateral filter). Preferably, the synthesized adaptive transformation field A can be... adaptive_field Laplacian smoothing or guided filtering is used to ensure that the transformation parameters within the semantic region are spatially continuous while maintaining the necessary abrupt changes at the semantic boundaries, thereby avoiding tearing.

[0117] Based on the adaptive transformation field, local adaptive transformation information is generated.

[0118] In other words, the local adaptive transformation information contains the adaptive transformation field. The spatially continuous adaptive transformation field A adaptive_field It will be used to perform secondary correction on the reference features.

[0119] According to one aspect of this application, generating fusion features includes:

[0120] By integrating the comprehensive uncertainty graph contained in the boundary uncertainty information and the adaptive fusion weight contained in the consistency verification results, a comprehensive fusion weight is generated.

[0121] In this embodiment, a pixel-level overall confidence map is generated to guide the final fusion strength. Specifically, a comprehensive uncertainty map U is obtained. Since the comprehensive uncertainty map U represents uncertainty (the larger the value, the less reliable), its complement (1-U) can be calculated to represent the probability confidence of the boundary region. Simultaneously, the adaptive fusion weight W is obtained. consistency This weight reflects the consistency confidence of the feature map under bidirectional validation. Optionally, a transition weight map W can also be obtained. transition This weight graph reflects the transition confidence of semantic boundaries. These multiple weight graphs (e.g., (1-U) and W) are then compared. consistency And optional W transition The initial fusion weight W is obtained by integrating the elements through element-wise multiplication. preliminary :W preliminary =(1 - U) * W consistency Preferably, in order to eliminate isolated noise points that may exist in the weight map while maintaining the weight changes at key structures (such as edges), the initial fusion weights W can be adjusted. preliminary Apply a smoothing filter that preserves the edges, such as a bilateral filter or a guided filter. The result after filtering is the comprehensive fusion weight W. final Comprehensive integration weight W final The higher the value of (x, y) (closer to 1), the higher the quality of the reference feature at pixel (x, y) (high probability confidence) and the more reliable the alignment with the target (high consistency confidence). Therefore, it should be given higher priority in subsequent fusion.

[0122] By applying comprehensive fusion weights and combining local adaptive transformation information, the reference features and target features are weighted and fused to generate fused features.

[0123] In this embodiment, an attention mechanism is preferably used to implement the weighted fusion. Specifically, weighted fusion is performed by applying comprehensive fusion weights, including:

[0124] Use target features as queries in the attention mechanism.

[0125] Specifically, at each scale level l of the feature pyramid, the target feature F is... T l As a query Q, i.e., Q=W Q * F T l W Q The learnable projection matrix for the query.

[0126] The adaptive transformation field contained in the local adaptive transformation information is applied to correct the reference features and determine the key and value of the attention mechanism.

[0127] Specifically, obtain the adaptive transformation field A smooth Adaptive transformation field A smooth Applied to reference features (e.g., forward alignment feature F) ref_to_target l Alternatively, this application process is a secondary fine-tuning correction, for example, achieved through deformable convolution, where A smooth Used to guide additional offsets. After correction, a finely calibrated reference feature F is obtained. aligned l The finely calibrated reference feature F aligned l It serves as both the key K and the value V. That is, K = W K * F aligned l V = W V * F aligned l W K and W V The learnable projection matrix for keys and values. Forward alignment feature F ref_to_target l It is based on probability offset geometric alignment, while the adaptive transform field A smooth It is a semantic-based local transformation, and the combination of the two can achieve dual alignment of geometry and semantics, thereby maximizing the accuracy of key K and value V.

[0128] The attention mechanism is executed to obtain the attention fusion result.

[0129] Specifically, in one implementation, a standard scaled dot product attention mechanism is used. The attention score matrix A is calculated. score A score = Softmax((Q·K T ) / sqrt(d k )); where K T It is the transpose of key K, d kThe feature dimension of key K (i.e., the number of channels after projection) is used. The Softmax function ensures that the sum of attention values ​​is 1. The attention fusion result F is obtained by weighted summation of the attention scores on the value V. attn :F attn = A score • V. In some preferred embodiments, a more complex attention mechanism can be employed to enhance the fusion effect: multi-head attention can be used, linearly projecting Q, K, and V into h different subspaces (i.e., h heads), each head having a dimension of d. h = d k / h. Perform the above scaled dot product attention calculation in parallel on h heads to obtain h outputs O. i . These h outputs O i The results are stitched together and passed through a final linear projection layer to obtain the multi-head attention fusion result F. mha Furthermore, building upon multi-head attention, the attention mechanism is combined with semantic information. For example, different attention heads can be guided to focus on different semantic combinations (e.g., heads 1-2 focus on foreground objects, heads 3-4 focus on background regions, and heads 5-6 focus on semantic boundaries). This can be achieved by applying the attention score (Q) before calculating the softmax. i ·K i T / sqrt(d h )) and from semantic label graph S map The generated semantic guidance mask M semantic i Add (e.g., M) semantic i In this context, the non-focused region is represented by negative infinity. Alternatively, bidirectional cross-attention can be used, in addition to calculating the forward attention F of the target query reference. mha In addition, the back attention F of the reference query target can be computed in parallel. rev_att The calculation method for reverse attention is: Q rev = W Q_rev * F aligned l K rev = W K_rev * F T l V rev = W V_rev * F T l ; where Q rev W is the query vector for reverse attention. Q_rev K is a learnable linear transformation matrix used to map reference features to query vectors. rev W is the key vector for reverse attention.K_rev V is a learnable linear transformation matrix used to map target features to key vectors. rev W is the value vector for reverse attention. V_rev Let F be a learnable linear transformation matrix used to map target features to a value vector. The forward attention F... mha and reverse attention F rev_att Features are fused through learnable gating mechanisms, such as bidirectional attention fusion feature representation F. bidirect = g· F mha + (1-g)· F rev_att , where g is the gating coefficient.

[0130] The attention fusion result is adjusted by applying comprehensive fusion weights to generate fusion features.

[0131] Specifically, the attention fusion result F attn (or its preferred option F) mha F bidirect ) and comprehensive integration weight W final Perform element-wise multiplication: F weighted_attn = F attn · W final Attention mechanisms solve the problem of which reference point to obtain information from, while W... final This addresses the question of the reliability of the acquired information. Preferably, to maintain the structural stability of the target features (especially the context of the region to be repaired), adaptive strength residual connections can be introduced. Specifically, the final fused feature F fused l It can be generated in the following way: F fused l = F weighted_attn + α res · F T l Among them, the residual weight α res It is adaptively adjusted. For example, α res Can be used with W final Related, such as α res = (1 - W final In a more preferred embodiment, α res The value can be determined based on the conflict type graph. map Dynamic setting: If there is a semantic conflict at (x, y), set α. res A high value (e.g., 0.9) indicates that the target feature F is almost entirely dependent on it. T l The context; if it is a structural conflict, set α. resSet the α value to the median (e.g., 0.7); if there is texture conflict or no conflict, set α. res The value is set to low (e.g., 0.3) to fully utilize the high-quality reference information. This generates fused features F at each scale l. fused l These features will be used for the final image reconstruction.

[0132] According to one aspect of this application, a high-precision image restoration system can be implemented as a computing device, such as, but not limited to, a dedicated image processing server, a personal computer, a workstation, a mobile smart terminal (such as a smartphone or tablet), or an embedded system (such as an in-vehicle computing unit or a security monitoring host). In terms of hardware architecture, the system typically includes one or more processors, memory, and one or more input / output (I / O) interfaces. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a dedicated AI acceleration chip (such as an NPU, TPU, or FPGA) for performing computationally intensive tasks. The memory can be volatile memory (such as RAM) or non-volatile memory (such as ROM, a hard disk drive (HDD), or a solid-state drive (SSD)) for storing program instructions and processing data (such as the image to be restored, a reference image, and intermediate feature maps). The I / O interfaces are used to communicate with external devices, such as a network interface (for receiving and sending image data) or a display controller (for presenting the restoration results). The memory stores computer program instructions (i.e., image restoration software or algorithm modules). When the processor executes these computer program instructions, it implements the following functional modules, specifically including:

[0133] The registration module acquires the reference image and the image to be repaired, calculates the initial affine transformation matrix, and applies the initial affine transformation matrix to the reference image to generate a coarsely aligned reference image. The specific implementation of the registration module includes feature point detection, RANSAC estimation, and least squares refinement.

[0134] The feature extraction module is used to extract reference features from the coarse alignment reference image and target features from the image to be repaired. The feature extraction module is implemented using a multi-scale convolutional encoder network with shared parameters.

[0135] A boundary modeling module is used to model the boundary uncertainty of the image to be repaired and obtain boundary uncertainty information. Preferably, the boundary modeling module includes a sub-module for calculating geometric, semantic, and structural confidence, an MLP network for fusing confidence maps, and a probability migration prediction network for predicting probability migration field parameters.

[0136] The consistency verification module is used to perform bidirectional feature mapping and consistency verification between reference features and target features to obtain consistency verification results. Preferably, the consistency verification module includes a submodule for calculating forward / backward consistency errors and a decision submodule for performing hierarchical conflict detection (e.g., based on frequency domain, edge response, and KL divergence).

[0137] The semantic transformation module is used to perform semantically guided local adaptive transformations and generate local adaptive transformation information. Preferably, the semantic transformation module includes a semantic segmentation network (such as DeepLabV3+), a transformation prediction network for predicting local transformation correction parameters, and a distance transformation and interpolation submodule for synthesizing the adaptive transformation field.

[0138] The collaborative fusion module is used to collaboratively fuse reference features and target features based on boundary uncertainty information, consistency verification results, and local adaptive transformation information to generate fused features. Preferably, the collaborative fusion module includes a multi-source weight integration submodule (used to generate W). final ) and attention fusion submodules (e.g., semantically guided multi-head attention or fusion modules with adaptive residual connections).

[0139] The reconstruction module is used to reconstruct the image to be repaired using fused features, thereby obtaining the repaired image. The reconstruction module is specifically implemented using a decoder network with layer-by-layer upsampling and skip connections. Optionally, the reconstruction module also includes a post-processing submodule for performing color uniformity correction and gradient domain fusion.

[0140] In actual operation, the processor calls and executes the above modules sequentially or in parallel to complete the entire process from inputting the image to be repaired and the reference image to outputting the final repaired image.

[0141] In another possible implementation, obtaining boundary uncertainty information can also be achieved by:

[0142] A pixel-level nominal offset distribution is constructed based on the reference features and the target features.

[0143] Specifically, the reference feature (denoted as F) R0 ) and target features (denoted as F) T The nominal offset distribution P at each pixel (x, y) is fed into the distribution prediction network. hat (ΔP|x). Nominal offset distribution P hat(ΔP|x) is a probabilistic estimate, rather than a deterministic value, of the most likely offset ΔP (a two-dimensional vector) at pixel (x, y). Preferably, this distribution is parameterized as a two-dimensional Gaussian distribution N(μ, ∑) or a low-order Gaussian mixture model (GMM). The network output is the parameters of this distribution, such as the two-dimensional mean field μ(x, y) and the covariance parameter field ∑. param (x, y). To ensure numerical stability, the covariance parameter field is preferably processed to ensure its physical validity. For example, the covariance matrix ∑(x, y) can be made positive definite by using the Softplus activation function or Cholesky reconstruction. Furthermore, a lower bound ε can be imposed on the eigenvalues ​​(or diagonal elements) of the covariance matrix. min and upper limit ε max This is to avoid the distribution degenerating into a deterministic distribution (zero variance) or becoming excessively divergent (infinite variance).

[0144] The boundary uncertainty of the image to be repaired is mapped to a risk budget.

[0145] Specifically, a boundary uncertainty map U(x, y) quantifies the difficulty of registration is obtained. This boundary uncertainty map can be obtained using a multi-factor confidence fusion method, or by analyzing the initial affine transformation A. init The reprojection error distribution and spatial consistency are used to estimate the uncertainty A of the initial transformation. unc This data is then fused with multi-source cues such as matching confidence and structural boundary strength, and normalized to obtain U(x, y). After obtaining U(x, y), a monotonically increasing mapping function (e.g., a linear function ρ(x) = ρ) is used. min +(ρ max -ρ min )·U(x,y) or a piecewise linear function, where ρ min For the minimum risk budget value, ρ max Mapping the uncertainty value U(x, y) to the risk budget ρ (for the maximum risk budget value) map (x, y). Risk budget ρ map (x, y) defines the offset distribution at pixel x that deviates from the nominal distribution P. hat The maximum distance.

[0146] Construct a sub-Bruker alignment target within a distribution sphere constrained by risk budget and Wasserstein distance.

[0147] In this embodiment, the nominal distribution P is no longer minimized. hat Instead of focusing on the expected loss under certain conditions, this aims to minimize the maximum expected loss arising from the most unfavorable distribution among all possible distributions Q* within the uncertainty set (i.e., the distribution sphere). Specifically, this involves the distribution sphere alignment objective (DR). objThis can be expressed as a min-max optimization problem: min θ E x [max Q*:Wp(Q*，Phat(Δ P|x))≤ρ(x) E Δ P~ Q* ψ θ [(x, ΔP)]; where θ is the model parameters to be optimized (e.g., parameters of the distribution prediction network and the alignment network); E x [·] indicates the expectation over all pixel positions x; max Q*:… W represents the search for the most unfavorable distribution among all possible probability distributions Q*; p (Q*, P) hat ) represents the distribution Q* and the nominal distribution P. hat The p-order Wasserstein distance between them, where p is preferably 1 or 2; ≤ρ(x) represents the distance between the most unfavorable distribution Q* and the nominal distribution P. hat The distance cannot exceed the risk budget ρ map (x); E Δ P~ Q* [·] indicates sampling the offset ΔP on the most unfavorable distribution Q* and calculating the expectation; ψ θ (x, ΔP) is the alignment loss function under offset ΔP (e.g., the reconstruction consistency loss between the reference feature and the target feature after offsetting by ΔP).

[0148] Solve for the target alignment of the sub-bars to obtain the adversarial sampling strategy.

[0149] Specifically, since the aforementioned min-max objective is difficult to optimize directly, in a preferred embodiment, its upper bound is derived through strong duality theory, or it is approximated using Conditional Value-at-Risk (CVaR). By introducing the dual variable ∧, the original problem can be transformed into a trainable optimization problem involving only minimization (min). Solving this dual problem yields the gradient for updating the model parameters θ, and the adversarial sampling strategy S used to guide sampling. rule .

[0150] Boundary uncertainty information is generated based on risk budgeting and adversarial sampling strategies. In other words, the boundary uncertainty information includes risk budgeting and adversarial sampling strategies.

[0151] Furthermore, it also includes:

[0152] An offset sample set is generated based on an adversarial sampling strategy and a nominal offset distribution.

[0153] Specifically, during training or inference, according to the adversarial sampling strategy S rule In P hat Centered on, ρ mapPreferential sampling is performed on the boundary of a Wasserstein sphere with radius (x), because the most unfavorable distribution usually appears at the edge of the distribution sphere. A set of offset samples S covering the most unfavorable cases is generated. Δ .

[0154] The offset sample set is applied to the reference feature, and risk-weighted aggregation is performed in conjunction with risk budgeting to generate robust reference features.

[0155] Specifically, for the offset sample set S Δ Each offset sample ΔP in i Apply it to the original reference feature F R0 (For example, offsetting can be achieved through deformable convolution) to obtain the aligned feature F. R0 (i). According to the adversarial sampling strategy S rule Or dual variables ∧ and risk budget ρ map Calculate the aligned features F in each sample. R0 (i) Risk weight w i Perform risk-weighted aggregation (e.g., weighted average) on all aligned features: F R_rob = ∑ i w i · F R0 (i). The aggregated result is the robust reference feature F. R_rob Robust reference features are a robust representation of all possible worst-case offsets and exhibit better robustness in high-uncertainty boundary regions compared to features aligned using only the nominal mean μ.

[0156] The adversarial sampling responses from the offset sample set are aggregated to generate offset evidence.

[0157] Specifically, during the sampling and convergence process, the response to adversarial sampling is statistically analyzed to generate offset evidence E. Δ . Shift evidence E Δ For a data structure containing rich diagnostic information, preferably, it includes one or more of the following fields: worst-case sample contribution percentage: in the aggregate robust reference feature F R_rob At that time, the one that contributes the most (i.e., the weight w) i The highest (statistical information of those offset samples); sampling diversity measure: generated S Δ Coverage in direction and magnitude; risk hotspot map (H risk ): Spatially identifies which regions have the strongest adversarial sampling response (i.e., the highest potential mismatch risk); Outlier Removal Record: The proportion of offset samples removed before aggregation due to causing excessive reconstruction error.

[0158] The collaborative fusion of reference features and target features is performed based on robust reference features and offset evidence.

[0159] In other words, in this embodiment, the output robust reference feature F R_rob And offset evidence E Δ This will serve as a key input for subsequent collaborative integration modules.

[0160] According to another aspect of this application, performing hierarchical conflict detection can also be:

[0161] By merging forward consistency error and backward consistency error, a closed-loop error is generated.

[0162] Specifically, the forward mapping residual R is calculated. fwd and the reverse mapping residual R bwd Forward residual R fwd It can be in the robust reference feature F R_rob Mapping to target feature F T Error calculated after domain. Reverse residual R bwd This involves using the target feature F T Mapping back to robust reference feature F R_rob Error in domain computation. The forward mapping residual R... fwd and the reverse mapping residual R bwd By fusing (e.g., adding or taking the maximum value) under the same dimensions (e.g., L2 norm), the original closed-loop error C is obtained. fb_raw Preferably, in order to eliminate the interference of isolated noise points on subsequent threshold calculations, it is necessary to adjust the original closed-loop error C. fb_raw Smoothing is then performed. This smoothing can be structure-sensitive, for example, using guided filtering or bilateral filtering, utilizing the semantic deformation field or the original image structure as a guide map. While maintaining clear structural and semantic boundaries, errors within the region are smoothed to obtain the final closed-loop error C. fb .

[0163] Regarding the distribution of closed-loop errors, the sample adaptive quantile method is used to dynamically calculate the set of error thresholds.

[0164] In this embodiment, a globally fixed hard threshold that may mismatch across different datasets is abandoned. Specifically, the closed-loop error C of the current input image (or images within a batch) is used instead. fb The actual statistical distribution is used to dynamically calculate its quantiles as thresholds. It can automatically adapt to different image content, lighting conditions, and noise levels, exhibiting stronger robustness.

[0165] The error threshold set contains corresponding thresholds for determining texture, structure, and semantic conflicts.

[0166] Specifically, to distinguish different types of conflicts, quantile calculation is preferably performed statistically on specific candidate regions: texture threshold θ texture : Identify texture candidate regions in an image (e.g., regions with high gradient magnitudes but small differences in eigenvalues ​​of the structure tensor), and only in these regions is the loop closure error C fb In terms of distribution, calculate its p1th percentile (e.g., p1=90%), and use it as θ. texture Structural threshold θ structure : Identify structural candidate regions (e.g., Canny edge bands or semantic boundary bands in an image), and only in these regions is the loop closure error C fb In terms of distribution, calculate its p2 percentile (e.g., p2 = 85%), and use it as θ. structure Semantic threshold θ semantic : Identify semantic candidate regions (e.g., regions where semantic categories change), and only in these regions is the loop closure error C fb In terms of distribution, calculate its p3 percentile (e.g., p3 = 80%) and use it as θ. semantic The final set of error thresholds is = {θ} texture θ structure θ semantic In a preferred embodiment, the threshold set Θ is also related to the risk budget ρ. map And offset evidence E Δ This involves coupled updates, forming a closed-loop feedback: for example, when the system detects a closed-loop error C in a certain region... fb It remains consistently above its adaptive threshold θ, and E Δ This also shows that when the proportion of worst-case samples in this area is high, the system can determine the risk budget ρ for this area. map The risk budget ρ for that area was set too low (overconfidence). Therefore, the system will automatically increase the local risk budget ρ for that area. map (But not exceeding global ρ) max This allows for the search for more robust alignments using a larger uncertainty ball in subsequent iterations or frames.

[0167] By comparing the closed-loop error with the set of error thresholds, the conflict type is determined to generate a conflict type map.

[0168] Specifically, for each pixel (x, y), its closed-loop error C is... fb (x, y) is compared with its corresponding dynamic threshold. To accurately determine the root cause of the conflict, this determination process should preferably also incorporate the pixel's type features. A type determination feature T is constructed for pixel (x, y). featFor example, texture intensity, structural gradient consistency, and semantic boundary proximity. The decision is made according to the order of semantic conflict taking precedence over structural conflict, and structural conflict taking precedence over texture conflict: IF C fb (x, y) > θ semantic AND T feat (x, y) indicates a semantic boundary; THEN determines a semantic conflict; ELSE IF C fb (x, y) > θ structure AND T feat (x, y) indicates a structural edge; THEN determines a structural conflict; ELSE IF C fb (x, y) > θ texture AND T feat (x, y) indicates a high-texture area; THEN indicates a texture conflict; ELSE indicates no conflict. The results of all pixel determinations constitute the conflict type map Q. type .

[0169] The results of hierarchical conflict detection include a conflict type graph. This graph serves as a crucial control signal, which is then fed into the subsequent collaborative fusion module.

[0170] In a further embodiment, the prediction-specific local transformation correction parameters are limited to local similarity transformation parameters.

[0171] Specifically, in predicting the local transformation correction parameter ΔA c In this process, the output of the transformation prediction network is no longer a general 6-dimensional affine correction vector, but is strictly limited to the parameters of the similarity transformation.

[0172] Local similarity transformation parameters characterize rotation, scaling, and translation transformations.

[0173] In this embodiment, the similarity transformation is a subset of the affine transformation, which only includes rotation, isotropic scaling (i.e., scaling by the same proportion in the x and y directions), and translation, but does not include shear or anisotropic scaling. Therefore, the parameters predicted by the network are limited to: rotation angle θ c Uniform scaling factor s c and the two-dimensional translation vector (tx) c ty c Compared to affine transformations with 6 degrees of freedom, similarity transformations with 4 degrees of freedom are less prone to non-physical distortions, tears, or folds during the transformation process (i.e., the Jacobian determinant of the transformation is always positive). This makes alignment within semantic regions more stable in engineering, especially when dealing with regions that require maintaining rigid structures, such as faces or intricate architecture, where the effect is more robust.

[0174] Smooth transitions at semantic boundaries are achieved by applying a monotonic boundary transition function.

[0175] Specifically, in the synthetic transformation field, the transition function (W) used for interpolation at semantic boundaries... transition In this embodiment, the transition function τ(d) is limited to being a monotonic function. The transition function τ(d) takes the distance d from the pixel to the nearest semantic boundary as input. Monotonicity ensures that as the pixel moves further away from the boundary (d increases), its transition weights (e.g., for local transformation A) change accordingly. local (c) The confidence level increases (or decreases) smoothly without ringing or non-physical swinging. For example, a truncated linear function or a sigmoid function τ(d) = 1.0 / (1.0 + exp(- (d - d0) / γ)) can be used as this monotonic transition function, where d is the distance from the current pixel to the nearest semantic boundary, d0 is the distance threshold or transition center position, and γ is the temperature coefficient or smoothness control parameter. This monotonic transition ensures that the transformation parameters of different semantic regions (e.g., buildings and sky) can be naturally and smoothly stitched together at the boundaries, avoiding the generation of new visual artifacts at the boundaries. This embodiment also generates a semantic deformation field Φ. sem and the transformed reference semantic features F R_sem Preferably, the generated deformation field Φ will also be... sem The quality is assessed (e.g., its Jacobian determinant and local smoothness are checked), and if a tendency to fold or abnormal stretching is found, this information is written back to the offset evidence E. Δ The deformation quality field is used as a reference for the subsequent fusion module when making decisions.

[0176] In one embodiment of this application, the specific steps of collaborative fusion and reconstruction are as follows:

[0177] Generate fusion weights W to dynamically harmonize robust reference features F R_rob Semantic reference feature F R_sem and features F from the target image itself T Specifically, the generation of fusion weights is not based on static rules, but is constrained by three key control signals: the conflict type graph (Q...). type ), offset evidence (E) Δ ) and semantic deformation field (Φ sem The preferred fusion decision strategy is as follows: when the conflict type graph Q... type When a semantic conflict is identified, it indicates a fundamental mismatch in category between the reference feature and the target feature (e.g., a building aligned to the sky). In this case, W is prioritized (e.g., given the highest weight) for using the target feature F. T Repairing contextual information (i.e., image-like rendering), robust reference feature F R_roband semantic reference features F R_sem The weights are forcibly reduced to near 0.1. When the conflict type graph Q... type When a structural conflict is identified, it indicates a misalignment of the geometric contours. In this case, it is preferable to enhance the semantic reference feature F. R_sem The proportion. Because semantic reference feature F R_sem It is generated based on a more rigid similarity transformation, and its geometric structure (such as lines and corners) typically has higher fidelity than that of F based on probability offset. R_rob When the conflict type graph Q type When a texture conflict is identified, it indicates inconsistencies in detail, but the structure may be correct. In this case, robust reference features F can be smoothly blended. R_rob and semantic reference features F R_sem The contribution, for example, each accounts for 50% of the weight, or slightly improves the robust reference feature F. R_rob The proportion of weight is adjusted to obtain a more robust texture. Above all the rules mentioned above, the offset evidence E... Δ (especially its risk hotspot map H) risk The field has veto power. If E Δ If the worst-case response percentage is extremely high for a certain region (regardless of its conflict type), indicating that any alignment attempt in that region is unreliable, then W for that region will also be forced to prioritize the use of the target feature F. T .

[0178] The original weights W may exhibit abrupt changes between pixels, which, if directly used for fusion, can easily produce visible blocky artifacts or checkerboard patterns in the restoration result. Therefore, it is preferable to apply spatial smoothing regularization to the original weights W to obtain regularized fusion weights W. reg One approach is to apply Gaussian filtering or bilateral filtering to W. Another preferred approach is to impose an L1 or L2 penalty on the spatial gradient of W during training, thus constraining its Lipschitz continuity so that it varies smoothly in space.

[0179] A confidence-gated mask, i.e., an independent control signal, is generated to control the degree to which the reconstruction decoder accepts reference information. Specifically, this confidence-gated mask M... conf The generation of the result incorporates the following information: closed-loop error C fb Error threshold set Θ, offset evidence E Δ Weight W of regularization reg For example, the mask generation rule is: when C fb (x, y) is much larger than Θ(x, y) (i.e., the self-calibration error exceeds the limit), and E Δ When (x, y) indicates a high risk, the confidence gating mask M will be used. confSetting (x, y) to a low value (e.g., 0.05) indicates a low confidence level. When C fb (x, y) is much smaller than Θ(x, y) and E Δ When (x, y) indicates low risk, the confidence gate mask M will be used. conf (x, y) is set to a high value (e.g., 1.0) to indicate high confidence. In the intermediate zone, a confidence-gated mask M is obtained by mapping through a smooth, gradual release curve (e.g., the Sigmoid function). conf The value of . Additionally, if a region has high boundary uncertainty U, but its deformation quality is assessed as good (i.e., Φ ). sem If the confidence level is stable, a gradual release strategy can be adopted, assigning a moderate (rather than the lowest) confidence value.

[0180] Using regularization to merge weights W reg For robust reference feature F R_rob and semantic reference features F R_sem Perform weighted fusion (e.g., weighted averaging) to obtain the final fused reference feature F. R_star_final .

[0181] The final fusion reference feature F R_star_final Confidence gate mask M conf Target features F T And the defect mask M, together input to the reconstruction decoder. Specifically, in the feature upsampling path of the decoder, when it is necessary to inject reference information (i.e., F) through skip connections. R_star_final When recovering details, the injection operation is subject to a confidence gating mask M. conf Gating regulation. For example, the injected characteristic = F R_star_final · M conf In the high confidence region (M) conf ≈ 1), the decoder fully utilizes F R_star_final Provides rich reference details; in the low confidence region (M conf ≈ 0), the reference information is truncated, and the decoder is forced to rely only on F. T The provided target context information (i.e., the execution class image drawing) avoids contamination from erroneous reference information. The decoder ultimately outputs the preliminary repaired image T. hat .

[0182] For the initial repaired image T hat Perform post-processing operations. Preferably, this includes: processing the initially repaired image T... hat With reference image R sem Perform local color statistics (e.g., mean and variance) matching to mitigate color differences across images. In the color-consistent image T... color Compared with the original target image T prepGradient domain fusion (e.g., Poisson fusion) is performed between them, especially when fusion weights W reg Larger areas (i.e., the restoration boundaries) are refined to obtain the final restored image T. out Optionally, the system also calculates the quality index S. metric (e.g., boundary consistency rate, based on C) fb and ρ map The worst-case risk metrics (PSNR / SSIM / LPIPS, etc.) are calculated and used for monitoring the training process and reporting during the validation period.

[0183] In a detailed embodiment, it is assumed that the focus is on a specific pixel p = (x, y) in the image to be repaired T that lies on the boundary of the repair mask M. The evaluation is performed using the refined affine matrix A. refined Spatial registration error ε caused in the neighborhood of this point spatial Suppose that by analyzing the actual deviations of the K nearest feature matching pairs in the neighborhood of point p, and then applying Gaussian kernel weighting, we can obtain the statistical value ε of the spatial registration error at that point. spatial (p) = 0.6 pixels. Assume the Gaussian kernel parameter σ for geometric confidence. geo 2 = 0.5. Geometric confidence level B geo = exp(-ε spatial 2 / σ geo 2 ) = exp(-0.6 2 / 0.5)≈0.4868. This measures high-level semantic features (e.g., F...). R 5 and F T 5 The cosine similarity at point p. Assume that in the high-level feature space (for simplicity, let's say 8 dimensions), the reference feature vector v is obtained by bilinear interpolation sampling at point p. R and target feature vector v T They are: v R = [0.1, 0.8, 0.0, 0.5, 0.2, 0.1, 0.0, 0.3]; v T = [0.1, 0.7, 0.1, 0.6, 0.3, 0.0, 0.1, 0.2]. Semantic confidence B sem = (v R ·v T ) / (||v R ||·||v T ||). Dot product v R · v T= (0.1 · 0.1) + (0.8 · 0.7) + … + (0.3 ·0.2) = 0.01 + 0.56 + 0.0 + 0.30 + 0.06 + 0.0 + 0.0 + 0.06 = 0.99; v R L2 norm ||v R || = sqrt{0.1 2 + 0.8 2 + … + 0.3 2} = sqrt{0.01 + 0.64 + 0.0 + 0.25 + 0.04 +0.01 + 0.0 + 0.09} = sqrt{1.04} ≈1.0198; v T L2 norm ||v T || = sqrt{0.1 2 + 0.7 2 +… + 0.2 2} = sqrt{0.01 + 0.49 + 0.01 + 0.36 + 0.09 + 0.0 + 0.01 + 0.04} =sqrt{1.01}≈1.0050. B sem = 0.99 / (1.0198 · ​​1.0050) = 0.99 / 1.0249 ≈ 0.9659. Analyze the gradient consistency between the coarsely aligned reference image R' and the target image T within the stripes on both sides of the boundary at point p. Assume that the gradient distribution distance d is calculated by comparing the gradient histograms. grad = 0.4. Assume the calculated gradient direction coherence is 0.8; the parameters of the fusion function are β1 = 1.0, β2 = 0.5. Structural confidence B str = exp(-β1×d grad )×(1+β2×coherence) / 2=0.6703 · 0.7≈0.4692. At pixel p=(x,y), the multi-factor boundary confidence tensor B(x,y) is the combination of these three components: B(x,y) = [B geo B sem B str ] = [0.4868, 0.9659, 0.4692]. This result (B sem Very high, but B geo and B str The low value indicates that at point p, the reference image and the target image are semantically matched (e.g., both are buildings), but there is some error in geometric registration, and local structures (such as window edges) are not aligned.

[0184] Input the confidence tensor B into a learnable fusion network (e.g., an MLP) to generate the comprehensive uncertainty U. U = 1.0 - comprehensive confidence. For simplicity, assume that the MLP, after adaptive normalization, learns a set of equivalent linear fusion weights: w g = 0.5 (geometric confidence level is the most important), w s = 0.2 (semantic confidence is the second highest), w t = 0.3 (structural confidence). Total uncertainty U(p) = 1.0 - (w g ·B geo +w s ·B sem + w t ·B str The overall confidence level = (0.5 * 0.4868) + (0.2 * 0.9659) + (0.3 * 0.4692) = 0.5774. The overall uncertainty U(p) = 1.0 - 0.5774 = 0.4226. In this numerical case, the overall uncertainty U(p) at pixel p = (x, y) is calculated to be 0.4226. This value (a moderate uncertainty) will serve as input for subsequent steps. For example, a U value of 0.4226 will cause the probability offset prediction network to predict a moderate variance, allowing the offset to explore to some extent around the mean μ; or, this U value will be mapped to a moderate risk budget ρ(p) used to define the radius of the Wasserstein distribution sphere.

[0185] In another embodiment of this application, the comprehensive loss function L total It is composed of a weighted sum of multiple components: L total = λ rec · L reconstruction + λ per · L perceptual + λ con · L consistency + λ sem · (L intra +L inter ) + λ unc · L uncertainty ; where λ rec , λ per , λ con , λ sem , λ uncAll are non-negative scalar weights used to balance the importance of each loss term. These weights can be set based on the experience of those skilled in the art or determined through hyperparameter optimization methods such as grid search. The specific definitions and functions of each loss component are as follows: Reconstruction Loss L reconstruction Used to measure the final restored image I repaired Compared with the original, undamaged true ground image I GT The pixel-level differences between them. Specifically, this loss preferably uses L1 loss (mean absolute error) or L2 loss (mean squared error), and its calculation range is limited to the area to be repaired identified by the repair mask M. For example, the L1 reconstruction loss is defined as: L reconstruction = (1 / N M )∑ x，y∈M |I repaired (x, y) - I GT (x, y)|, where N M It is the total number of pixels within the mask region M.

[0186] Perceived loss L perceptual This is used to enhance the visual realism of the restored image, making it closer to a real ground image in terms of structure and high-level texture. Specifically, the restored image I... repaired and ground real image I GT Simultaneously, input a pre-trained deep neural network (e.g., a VGG-16 or VGG-19 network pre-trained on ImageNet) and extract activation feature maps from one or more of its intermediate layers (e.g., ReLU33, ReLU43). perceptual This is defined as the L1 or L2 distance between these activated feature maps. The loss penalizes the difference in the feature space, rather than the difference in the pixel space, which helps generate details that better match human visual perception.

[0187] Consistency loss L consistency This is used for explicit supervision of bidirectional consistency verification. Specifically, the loss directly utilizes the forward consistency error C. forward l and reverse consistency error C backward l L consistency This is defined by some measure of these consistency errors, such as the average L1 or L2 norm calculated over all levels l and all pixels: L consistency = mean(|C forward | + |C backward |); By minimizing this loss, the network is forced to learn feature mappings with round-trip consistency, thereby improving the accuracy of conflict detection.

[0188] Semantic loss L semThe adaptive transformation used for supervising semantic guidance ensures that the generated transformation field (or its underlying probability offset field ΔP) is semantically plausible. This loss consists of two sub-items: semantic intra-congruence loss L. intra Its definition is L intra = ∑ c ∑ i，j∈c ||ΔP(i) -ΔP(j) || 2 / |N c |, where ΔP(i) is the predicted offset vector at pixel position i, N c Let L be the set of pixel pairs within semantic category c. This loss term acts as a smoothing regularizer, penalizing excessively large offset differences ΔP between two adjacent pixels i and j within the same semantic category c. It encourages the transform field to remain smooth and continuous within the semantic region (e.g., a patch of sky or the walls of a building). Cross-semantic adaptive loss L inter Its definition is L inter =∑ c1≠c2 g(d semantic (c1, c2))·||ΔP c1 -ΔP c2 || boundary This loss term applies to the boundaries between different semantic categories c1 and c2. g(·) is the loss term related to the semantic distance d. semantic Functions, such as g(d) = exp(-d / τ) semantic ). ΔP c1 τ is the average offset vector or representative offset value of semantic category c1 at the boundary; semantic denoted as temperature coefficient. If two semantic categories c1 and c2 are semantically similar (e.g., road and sidewalk, d small, g(d) large), the loss term penalizes the offset discontinuity at their boundaries; conversely, if they are semantically significantly different (e.g., sky and building, d large, g(d) small), the loss term allows for large offset jumps at the boundaries.

[0189] Uncertainty Loss L uncertainty Used to supervise the training of the Probability Offset Prediction Network (ProbOffsetNet). Specifically, the loss is defined as the negative log-likelihood (NLL) of the probability offset: Boundary uncertainty modeling module = -∑log(P(ΔP) GT | U)), where P is the probability distribution function; ΔP GT This is the true ground offset (obtained through other means such as optical flow estimation). By minimizing this loss, the probabilistic offset prediction network is trained to predict the ground offset ΔP that maximizes the true ground offset given the uncertainty U. TThe distribution of the probability of occurrence (i.e., mean μ and covariance Σ). In the preferred implementation employing a conditional variational autoencoder (C-VAE) architecture, this uncertainty loss L uncertainty It will also include a KL divergence term to regularize the latent spatial distribution of the C-VAE, making it close to the standard normal distribution, thereby giving the generated probability shift good distribution characteristics.

[0190] During training, the system receives a batch of training data (including a normalized reference image R, a normalized image to be repaired T, a repair mask M, and the corresponding ground truth image I). GT and ΔP GT The system performs forward propagation, calculating all intermediate results (such as I). repaired U, C forward / backward (e.g., ΔP, etc.). Calculate L based on the above definitions. total L is calculated using the backpropagation algorithm. total The gradients are calculated relative to the gradients of all trainable parameters in the system (e.g., the weights of the encoder, decoder, ProbOffsetNet, SemanticTransformNet, etc.), and these parameters are updated using an optimizer (e.g., the Adam optimizer or SGD). This process is performed iteratively until the model converges.

[0191] According to one aspect of this application, generating adaptive fusion weights can also involve: generating K offset field instances based on the mean parameter map μ and the standard deviation parameter map σ using reparameterization techniques; applying deformable convolution to the reference features at each scale; performing spatial transformation using the offset field instances to obtain K sets of forward alignment features; evaluating the alignment stability at each position through analysis of variance to generate forward alignment confidence; calculating the inverse matrix of the refined affine matrix to obtain the inverse transformation matrix; applying it to the target features at each scale to generate the reverse reference offset; obtaining the reverse mapping feature through deformable convolution; performing a forward transformation on this feature again through the offset field instances to obtain the round-trip feature; calculating the L2 distance between the reference feature and the round-trip feature as the forward consistency error; and similarly calculating the reverse consistency error.

[0192] Layered conflict determination is performed based on consistency errors at different scales: when the error of the lower layer (l≤2) exceeds the threshold θ texture The time marker is used as a texture conflict flag; the error in the middle layer (3≤l≤4) exceeds θ. structure When setting a structural conflict flag, if the error of a high-rise building (l≥5) exceeds θ... semantic The semantic conflict flag is activated in real time, generating a conflict type graph that records the dominant conflict type at each spatial location. Specifically, from the forward consistency error C... forward 1 and C forward 2Low-level texture errors are extracted, and a two-dimensional discrete cosine transform (DCT) is applied to each 8×8 local block to obtain a frequency domain error representation. The frequency domain coefficients are divided into three groups: low frequency (DC and the first 3 AC coefficients), mid frequency (4-15 AC coefficients), and high frequency (the remaining AC coefficients), and the energy ratio is calculated for each group. low ratio mid ratio high When the proportion of high-frequency energy exceeds the threshold θ high_freq When the value is 0.4, it is marked as inconsistent texture details. A sliding window is used to traverse the entire error map to generate an initial texture conflict map. Morphological opening is performed on the initial map to remove isolated noise points, and then Gaussian smoothing is applied to obtain continuous texture conflict markers, the values ​​of which represent the intensity of the texture conflict. The forward consistency error C is then extracted. forward 3 and C forward 4 As the mid-level structure error input, the Canny edge detector is used to measure the error in the reference feature F. R 3 and alignment feature F ref→target 3 Edge detection is performed to obtain a reference edge map and an aligned edge map. The Hausdorff distance of the edge is calculated to measure the degree of edge offset. At the same time, straight and curved structures are detected by Hough transform, and the difference Δn in the number of detected structural elements is compared. struct =|n R -n align |, when d H >θ hausdorff or Δn struct >θ struct_diff At that time, the structural conflict marker is activated, and the edge conflict is diffused to the neighboring region through distance transformation. The range of influence is exp(-d / σ). struct ) controls and generates spatially continuous structural conflict markers. Where n R Let n be the number of structuring elements detected by the Hough transform in the reference feature map. align d represents the number of structuring elements detected in the alignment feature map. H Let θ be the Hausdorff distance. hausdorff θ is the threshold for the Hausdorff distance. struct_diff σ is the threshold for the difference in the number of structural elements; struct Parameters used to control the extent of structural conflict propagation.

[0193] From the forward consistency error C forward 5 and C forward 6 Extract high-level semantic error information and apply it to the reference feature F.R 5 and alignment feature F ref→target 5 For each spatial location, extract a c-dimensional feature vector and calculate its k nearest neighbors in the feature space, using the KL divergence D... KL (P R ||P align ) measures the difference in the distribution of characteristics, where P R and P align The probability distributions of the reference and alignment features are respectively, when D KL Exceeding the threshold θ KL When the value is 0.5, a semantic conflict is initially identified. Further, a pre-trained semantic classifier is used to predict the category of the feature, and the consistency of the predicted categories is compared. If the categories are different, a semantic conflict is confirmed, and the conflict intensity is quantified into a semantic conflict flag (conf). semantic =D KL ×(1-P same_class ), where P same_class This refers to the probability that the reference feature and the alignment feature belong to the same category in the semantic classifier's prediction. Integrating texture conflict markers, structural conflict markers, and semantic conflict markers, for each spatial location (x, y), the dominant conflict type is determined by ranking the conflict intensity, with the priority rule being: semantic conflict > structural conflict > texture conflict. When multiple conflicts coexist, a weighting function is used, including the priority weight w for semantic conflicts. priority (semantic)=1.0, priority weight w for structural conflicts priority (structure)=0.6, texture conflict priority weight w priority The score is calculated by weighting the values ​​by (texture) = 0.3. total =Σw priority ×Conflict type Conflict type The conflict intensity value can be a numerical marker of texture conflict, structural conflict, or semantic conflict. A unique conflict category label is assigned to each location based on a score threshold, generating a conflict type map. Simultaneously, secondary conflict types are recorded to form a secondary conflict map for subsequent fine-tuning. Based on the weighted sum of forward and backward consistency errors, a basic consistency weight is calculated using a negative exponential transformation. The weighting strategy is adjusted according to the conflict type map: semantic conflict regions are forcibly given a reduced reference weight below 0.1, structural conflict regions are given a medium weight of 0.5, and texture conflict regions are maintained with a high weight above 0.8. Finally, adaptive fusion weights are generated for subsequent feature fusion.

[0194] This invention departs from relying on a single, deterministic offset field, instead transforming boundary alignment into a probabilistic modeling problem. Specifically, it computes a multi-factor boundary confidence tensor (evaluating the confidence levels of geometry, semantics, and structure) and fuses them into a comprehensive uncertainty map U. This uncertainty map U guides the prediction of the distribution parameters (mean μ and covariance Σ) of the probabilistic offset field, or employs Divided Bruker Bar Optimization (WDRO) to find the most unfavorable offset distribution within the Wasserstein sphere. This enables robust probabilistic or adversarial sampling in regions with high uncertainty (such as occluded boundaries), resolving boundary artifacts caused by deterministic sampling. By computing the inverse matrix of the initial transformation, bidirectional feature mapping is performed from reference to target (forward) and from target to reference (backward). Based on this, the reliability of feature transfer is quantified by calculating the round-trip consistency error. Furthermore, through hierarchical conflict detection (texture, structure, semantics) or adaptive quantile methods, the type and location of feature conflicts can be dynamically identified and reconciled through subsequent adaptive fusion weights, solving the problem of blind propagation of erroneous features. Semantic segmentation networks are used to identify different categories (such as buildings, sky, and vegetation) in images. For each semantic category, specific local transformation correction parameters (or constrained local similarity transformation parameters) are predicted. By smoothly transitioning at the boundaries of different semantic categories (e.g., based on distance transformation), a spatially continuous adaptive transformation field is synthesized. This allows rigid building regions and non-rigid sky regions to adopt their respective most suitable geometric transformations, solving the problem of local structural distortion caused by a one-size-fits-all approach to global transformations.

[0195] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A high-precision image restoration method, characterized in that, include: Obtain a reference image and the image to be repaired, calculate and apply an initial affine transformation matrix to the reference image, and generate a coarsely aligned reference image; Extract reference features from the coarse alignment reference image and extract target features from the image to be repaired; Model the boundary uncertainty of the image to be repaired to obtain boundary uncertainty information; Perform bidirectional feature mapping and consistency verification between reference features and target features to obtain consistency verification results; Based on the image to be repaired, a semantically guided local adaptive transformation is performed to generate local adaptive transformation information; Based on boundary uncertainty information, consistency verification results, and local adaptive transformation information, reference features and target features are collaboratively fused to generate fused features; The image to be repaired is reconstructed using fusion features to obtain the repaired image; Obtain information on boundary uncertainties, including: For the boundary region of the image to be repaired, calculate the multi-factor boundary confidence tensor; By fusing multi-factor boundary confidence tensors, a comprehensive uncertainty graph is generated. By combining the comprehensive uncertainty map with target features, the distribution parameters of the generated probability offset field are predicted; By combining the distribution parameters of the comprehensive uncertainty map and the probability offset field, boundary uncertainty information is generated; Calculating the multi-factor boundary confidence tensor includes: Evaluate the local registration error associated with the initial affine transformation matrix and determine the geometric confidence level; The similarity between reference features and target features in a predefined high-level semantic dimension is measured to obtain semantic confidence. The gradient consistency between the coarse alignment reference image and the image to be repaired is analyzed to obtain the structural confidence. The geometric confidence, semantic confidence, and structural confidence are combined to form a multi-factor boundary confidence tensor; Obtaining consistency verification results includes: Perform forward mapping to align the reference features to the preset target coordinate space and generate forward-aligned features; Calculate the inverse transformation matrix of the initial affine transformation matrix; The inverse transformation matrix is ​​applied to perform inverse mapping, aligning the target features to a preset reference coordinate space and generating inverse mapped features; Calculate the forward consistency error between the reference feature and the reverse-mapped feature; Calculate the reverse consistency error between the target feature and the forward alignment feature; Hierarchical conflict detection is performed based on forward consistency error and reverse consistency error. Based on the results of hierarchical conflict detection, a consistency verification result is generated. Obtaining consistency verification results also includes: Calculate the basic consistency weight based on the forward consistency error and the backward consistency error; Apply differentiated weight adjustment strategies to adjust the basic consistency weights and generate adaptive fusion weights; By combining the results of adaptive fusion weights and hierarchical conflict detection, a consistency verification result is generated; Generate fusion features, including: Integrate the comprehensive uncertainty graph contained in the boundary uncertainty information and the adaptive fusion weight contained in the consistency verification results to generate a comprehensive fusion weight; By applying comprehensive fusion weights and combining local adaptive transformation information, the reference features and target features are weighted and fused to generate fused features.

2. The method according to claim 1, characterized in that, Also includes: Sample from the probability distribution defined by the distribution parameters of the probability migration field to generate migration field instances; Apply the offset field instance to the reference feature to generate the forward aligned feature; The consistency verification results are generated based on forward alignment features.

3. The method according to claim 1, characterized in that, Generate local adaptive transformation information, including: Semantic segmentation is performed on the image to be repaired to obtain a semantic segmentation map, which identifies a predetermined number of semantic categories; For each semantic category, combined with the target features, predict specific local transformation correction parameters; By applying the local transformation correction parameters to the initial affine transformation matrix, a category-specific transformation matrix for the semantic category is obtained; Based on category-specific transformation matrices and semantic segmentation maps, a spatially continuous adaptive transformation field is synthesized by performing smooth transitions at semantic boundaries. Based on the adaptive transformation field, local adaptive transformation information is generated.

4. The method according to claim 3, characterized in that, Predict specific local transformation correction parameters, including: Using semantic segmentation maps as masks, features corresponding to semantic categories are extracted and aggregated from target features to form semantically specific feature representations; Input semantically specific feature representations into a pre-configured transform prediction network to generate dedicated local transform correction parameters.

5. The method according to claim 3, characterized in that, Synthesized spatially continuous adaptive transformation fields include: Extracting semantic boundaries from semantic segmentation maps; Calculate the distance from each pixel to the nearest semantic boundary and generate a distance map; A transition function is constructed based on the distance graph, and a transition weight graph is generated. By applying a transition weight map, weighted interpolation is performed on the category-specific transformation matrix to synthesize an adaptive transformation field.

Images