An image restoration method

Abstract

This invention discloses an image restoration method, comprising: acquiring a target image, a set of reference images, and a boundary intensity map; extracting target features, a list of reference features, and boundary features; based on the extracted features, performing similarity modeling using a learnable spiral path to generate a similarity model result; based on the similarity model result, performing geometric alignment and dual-domain fusion on the list of reference features to generate initial fusion features and alignment parameters; calculating an error pattern map based on the initial fusion features; performing closed-loop feedback according to the error pattern map to iteratively update and generate final fusion features; and decoding the final fusion features to obtain the restored image. This invention improves the structural consistency and detail fidelity of the restored image under non-rigid deformation through boundary-aware spiral sampling, non-folded alignment, and closed-loop feedback.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular, it is an image restoration method. Background Technology

[0002] Reference-based image inpainting technology aims to fill and repair missing or editable areas in a target image using one or more reference images containing complete, high-quality content. It holds significant research and application value in fields such as virtual try-on, digital artifact restoration, film and television post-production, and fashion e-commerce. The core challenge of this technology lies in accurately borrowing appropriate texture and structural information from the reference image and seamlessly and realistically integrating it into the missing areas of the target image to achieve a high-fidelity and structurally consistent restoration effect.

[0003] Most existing reference image inpainting methods employ deep feature matching-based strategies. A mainstream approach involves first extracting deep feature maps of the target and reference images using a parameter-sharing encoder (e.g., a variational autoencoder, VAE). Then, the feature maps are divided into dense, non-overlapping rectangular image patches, which are projected into a low-dimensional representation space through a shared linear mapping layer. In this space, a dense similarity matrix is ​​constructed by calculating the cosine similarity (or attention score) between the target image patch and all reference image patches. Using this similarity matrix as weights, the features of the reference image patches are weighted and summed to obtain the inpainting features, which are then fed into a decoder (e.g., a dual-UNet architecture) to reconstruct the inpainted result.

[0004] However, the aforementioned existing technologies still have limitations in similarity sampling, geometric alignment, and fusion strategies, especially in handling complex boundaries, large pose deformations, and texture-structure consistency. Boundary blurring or texture leakage may occur, leading to local folding or overstretching, resulting in detail distortion or structural misalignment. Summary of the Invention

[0005] The purpose of this invention is to provide an image restoration method to solve the aforementioned problems existing in the prior art.

[0006] Technical solution: An image restoration method, comprising:

[0007] Acquire the target image, a set of reference images, and a boundary intensity map;

[0008] Extract target features from the target image, a list of reference features from the reference image set, and boundary features from the boundary intensity map;

[0009] Based on target features, a list of reference features, and boundary features, similarity modeling is performed using a learnable spiral path to generate similarity model results.

[0010] Based on the similarity model results, target features, and boundary features, geometric alignment and dual-domain fusion are performed on the reference feature list to generate initial fused features and alignment parameters;

[0011] Based on the initial fusion features, target image, boundary features, alignment parameters and similarity model results, the error pattern map is calculated and closed-loop feedback is executed to iteratively update and generate the final fusion features;

[0012] The final fusion features are decoded to obtain the repaired image.

[0013] Beneficial effects: This invention improves the structural consistency and detail fidelity of repaired images under non-rigid deformation through boundary-aware spiral sampling, non-folded alignment, and closed-loop feedback. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of an image restoration method provided in an embodiment of this application.

[0015] Figure 2 A flowchart illustrating the steps of calculating the error pattern diagram provided in the embodiments of this application.

[0016] Figure 3 A flowchart illustrating the steps for generating similarity model results provided in this application embodiment.

[0017] Figure 4 A flowchart illustrating the steps for generating initial fusion features provided in this application embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0020] The study found that the sampling method based on rigid rectangular patches lacks flexibility. When the rectangular patch crosses the critical structural boundaries of the target (e.g., the outline of an object), it simultaneously samples semantically irrelevant regions (e.g., the background), introducing contamination information into the fused features, resulting in blurred boundaries or texture leakage. When dealing with significant non-rigid deformations between the reference and target images (e.g., differences in facial pose or clothing folds), existing alignment modules (such as conventional optical flow or unconstrained deformation fields) struggle to guarantee topological consistency. This can lead to local folding or overstretching during alignment, generating geometrically unreasonable artifacts and damaging the structure of the repaired area. Furthermore, the fusion process is typically open-loop. The system lacks an evaluation and correction mechanism for the quality of the initial fused features. Once there is a deviation in the initial similarity estimation or geometric alignment, the error will directly propagate to the final result, especially when balancing spatial (position) and frequency (texture) information, easily leading to a loss of focus on one over the other, resulting in detail distortion or structural misalignment.

[0021] In one possible implementation, an image inpainting method based on multi-reference image local similarity weighting and dual UNet attention interaction includes: acquiring at least one reference image (e.g., R1 and R2) and one image to be inpainted (denoted as T). The reference image and the image to be inpainted are respectively input into a VAE (Variational Autoencoder) encoder with shared parameters to extract their respective depth feature maps. For example, the size of the feature map can be (B, C, H, W), where B is the batch size, C is the number of channels (e.g., 3 or 512), and H and W are the height and width of the feature map, respectively (e.g., 64 or 32). Based on this, each feature map is divided into N non-overlapping image patches of a fixed size (e.g., 16x16). To facilitate subsequent computation, the features of each image patch are input into a shared linear mapping layer, mapping the channel dimension C to a lower dimension d (e.g., d=128), and optionally performing L2 normalization. The similarity between the image patches of the image to be inpainted and the image patches of the reference image is calculated. Specifically, for each image patch vector t of the image to be inpainted... i (i is the index of the image patch, i = 1, ..., N), and is associated with the vectors r of all image patches in reference images R1 and R2 respectively. j (j is the index of the reference image patch) Calculate the cosine similarity s (i，j) The calculation process can be expressed as: s (i，j) = (t i • r j ) / (||t i || × ||r j ||); where • represents the vector dot product; ||t i || represents vector t i L2 norm; ||r j|| represents vector r j The L2 norm is used to calculate the similarity score. After obtaining the similarity score, it is normalized using the Softmax function to obtain the weight distribution α. (i，j) α (i，j) = exp(s (i，j) ) / (Σ k exp(s (i，k) )); where exp() represents the exponential function; Σ k This represents summing over all reference image patches k. In some implementations, to prevent low-similarity image patches from completely losing information, a minimum weight threshold w can be set on the result after Softmax. min Then, it is re-normalized. Using the obtained weights, the image patch features of the reference image are weighted and summed to obtain the reference features f aligned with the target image patch. i (R1) and f i (R2) :f i (R1) = ∑ j α (i，j) (R1) × r j (R1) ;f i (R2) = ∑ j α (i，j) (R2) × r j (R2) To determine the overall contribution of the two reference images at the current location, a gating mechanism can be introduced to calculate the global weights g1 and g2. For example, this can be achieved using the sigmoid function σ: g1 = σ(W1) T [t i ;f i (R1) g2 = 1 - g1; where W1 is the learnable weight matrix; T represents matrix transpose; [;] represents feature concatenation; σ is the Sigmoid activation function. The fused reference feature f i (fusion) It can be represented as: f i (fusion) = g1 × f i (R1) + g2 × f i (R2) ; The fusion feature f of all image patches i (fusion)The original feature map is restored to a fused reference feature map of the same size as the input feature map. A dual UNet architecture is used for feature interaction and decoding. The first UNet (UNet1) takes the fused reference feature map as input, and the second UNet (UNet2) takes the feature map of the image to be repaired, T, as input. In the decoding stage, a multi-head attention mechanism across UNets is introduced. The calculation process of this attention mechanism can be schematically represented as: Attn(Q, K, V) = Softmax((QK) = Softmax ... T ) / sqrt{d k}) V; where query Q comes from decoder features of UNet2; key K and value V come from corresponding layer features of UNet1; d k The dimension of key K is used to scale the dot product result. In this way, UNet2 can dynamically focus on relevant information in the reference image when repairing the target image. The output of UNet2 is then decoded to generate the repaired image.

[0022] like Figure 1 As shown, an image restoration method is proposed, including:

[0023] Obtain the target image, a set of reference images, and a boundary intensity map.

[0024] Specifically, the target image T target The image to be repaired; reference image set R refs Includes one or more images used to provide repair information; Boundary intensity map B map It can be used to indicate structural boundaries or occlusion boundaries in a target image, and can be generated by conventional edge detection operators (such as Canny, Sobel) or pre-trained boundary detection networks.

[0025] Extract target features from the target image, a list of reference features from the reference image set, and boundary features from the boundary intensity map.

[0026] For example, a depth encoder (such as a VAE encoder or the encoding part of UNet) can be used to extract features from each image in the target image and reference image sets, respectively, to obtain the target features F. target (e.g., 32x32x512) and reference feature list F refs Simultaneously, boundary features B can be extracted using boundary subnetworks (such as lightweight convolutional networks). feat (e.g., 32x32xC) b C b (Number of boundary feature channels).

[0027] Based on target features, a list of reference features, and boundary features, similarity modeling is performed using a learnable spiral path to generate similarity model results.

[0028] In this embodiment, conventional rectangular image patch matching is no longer used; instead, a similarity modeling approach is introduced. For example, a geometric parameter subnet can be used to predict parameters (e.g., initial radius, radius growth rate, etc.) for defining the Archimedean spiral path based on local target features, reference features, and boundary features. Target features and reference features are sampled along these spiral paths generated in the spatial coordinate domain, and path-level similarity is calculated. The result of this similarity model can include the generated spiral path parameters. params The calculated path-level similarity S spiral And the initial reference fusion weight W after normalization by the temperature parameter τ0. ref_0 .

[0029] Based on the similarity model results, target features, and boundary features, geometric alignment and dual-domain fusion are performed on the reference feature list to generate initial fused features and alignment parameters.

[0030] In this embodiment, the geometric deformation between the reference image and the target image is addressed. Specifically, based on the initial fusion weight W... ref_0 It can solve for the deformation field of a thin plate spline (TPS), and the parameters of this deformation field are the alignment parameters θ. tps The solution process preferably includes boundary regularization terms and Jacobian non-folding constraints to ensure smooth deformation and avoid topological errors. This alignment parameter θ is then used. tps For reference feature list F refs Differentiable resampling is performed to obtain aligned reference features F. refs_warped In parallel, dual-domain fusion is also performed. Besides the spatial domain spiral path similarity S... spiral Furthermore, the bandpass similarity S of the path sampling sequences is calculated in the frequency domain. fourier The similarity S of the spiral paths spiral Bandpass similarity S fourier The learnable proportion is fused into a two-domain similarity S. dual Based on dual-domain similarity S dual Update the fusion weights to obtain W ref_1 And for the alignment reference feature F refs_warped We perform weighted fusion to obtain the initial fusion feature F. fused_0 .

[0031] Based on the initial fusion features, target image, boundary features, alignment parameters, and similarity model results, the error pattern map is calculated and closed-loop feedback is executed to iteratively update and generate the final fusion features.

[0032] In other words, based on the initial fusion features, target image, boundary features, alignment parameters and similarity model results, an error pattern map is calculated, and closed-loop feedback is performed according to the error pattern map to iteratively update and generate the final fusion features.

[0033] Specifically, to further optimize the fusion results, an error-driven closed-loop feedback mechanism is introduced. This involves comparing the initial fusion features F. fused_0 With target image T target (or its characteristics), calculate the differences in three dimensions—spatial domain, frequency domain, and boundary gradient—to form the error pattern diagram E. pattern The error mode diagram E pattern It is used to drive three feedback branches: error backfeeding to the sampling grid, generating offset residual Δ O The coordinate grid is used to fine-tune the sampling of the spiral path; the error is back-fed to the alignment parameters to generate the control point residual Δ. CP Used to correct TPS control points and alignment parameter θ tps Errors are fed back into the attention layer to generate gating signals, which are used to modulate the cross-network attention strength between the target and the reference in the feature extractor (such as an encoder) pyramid. Simultaneously, based on the error pattern diagram E... pattern The local statistics are used to adaptively adjust the temperature parameter τ and the helix radius growth rate b (i.e., temperature-radius coupled control). This closed-loop feedback process will iterate until convergence (e.g., the error is below the threshold or the maximum number of iterations is reached), outputting a stable final fused feature F. fused .

[0034] The final fusion features are decoded to obtain the repaired image.

[0035] The final fusion feature F after closed-loop feedback optimization fused The input is fed into a decoder (such as a VAE decoder or the decoding part of UNet) to reconstruct the final repaired image I. recon .

[0036] like Figure 3 As shown, in one possible implementation, a learnable spiral path is used for similarity modeling to generate similarity model results, including:

[0037] The target features, reference feature list, and boundary features are aggregated to form local context features.

[0038] Specifically, local context features feat At each spatial location of the feature map, the target feature F is aggregated. target Reference Feature List F refs Statistical information within a local window (e.g., 3x3 or 5x5), and boundary features B feat Intensity information and maskM It is formed based on observable information. Preferably, an effective reference filtering process can be performed simultaneously with the formation of local contextual features. This process is based on a mask. M Known region and boundary features B feat Based on the confidence level, reference images that are significantly inconsistent with the current repair location in content or structure are filtered out, generating a valid reference index. valid_refs This reduces interference in subsequent similarity calculations.

[0039] Local contextual features are input into a pre-configured geometric parameter subnet to regress and predict geometric parameters; the geometric parameters define the learnable spiral path; wherein the learnable spiral path is an Archimedean spiral.

[0040] In this embodiment, the geometric parameter subnetwork can be a lightweight neural network, such as one composed of stacked pointwise convolutions, depthwise separable convolutions, and local self-attention modules. The geometric parameter subnetwork utilizes local context features. feat As input, the learnable spiral path is the Archimedean spiral. Its geometric definition can be expressed as: r = a + bθ; where r is the polar radius; θ is the polar angle; a is the initial radius; and b is the radius growth rate. The geometric parameters of the subnet regression prediction are the spiral. params At least include the initial radius a, the radius growth rate b, and the angle step size θ. step and path length or maximum angle θ max In some implementations, to ensure the physical feasibility and stability of the generated paths, constraints need to be imposed on the regressed original geometric parameters. For example, nonnegative projections (e.g., through ReLU or Sigmoid activation functions) can be applied to the initial radius *a* and the radius growth rate *b*; and reasonable upper and lower bounds can be set for the angle step size and path length to avoid sampling being too sparse or too dense. Furthermore, as a preferred regularization method, multi-path equilibrium constraints can be introduced. When the geometric parameter subnet predicts multiple (k > 1) spiral paths, to avoid all paths collapsing to similar geometric shapes, an entropy-based equilibrium regularization term can be introduced to encourage parameter diversity, or a neighboring path difference penalty term can be introduced to ensure the discriminability between paths.

[0041] Similarity model results are generated based on geometric parameters.

[0042] In other words, geometric parameters constitute a part of the similarity model results.

[0043] In a further implementation, generating similarity model results also includes:

[0044] Generate a path coordinate sequence based on geometric parameters.

[0045] Specifically, based on geometric parameters a, b, θ step θ max In the initial sampling grid grid_0 For each center point of a 32x32 feature space coordinate system (e.g., θ=0, θ=1), generate a discrete sequence of angles (e.g., θ=0, θ=1). step ,2θ step , …, θ max The corresponding polar radius r is calculated using Archimedes' spiral formula, and then converted to Cartesian coordinates (x, y) to form the path coordinate sequence spiral. coords_raw It should be noted that this also includes out-of-bounds clipping and interpolation backoff. For those calculated coordinate points that exceed the feature map boundary (e.g., a 32x32 range), clipping is performed; or when approaching the boundary, interpolation backoff is used to ensure that all sampled points fall within the effective feature region.

[0046] The system detects the intersection of the path coordinate sequence with the preset high-intensity boundary based on boundary features; it then corrects or trims path segments that cross the high-intensity boundary in the path coordinate sequence to generate path coordinates with consistent boundaries.

[0047] In this embodiment, it is ensured that spiral sampling does not traverse semantically irrelevant regions. Specifically, boundary feature B is utilized. feat As an indicator, detect the path coordinate sequence spiral coords_raw Does the path cross from a low-boundary-intensity region to a high-boundary-intensity region (e.g., an object outline)? For example, correction or clipping can include several methods: Clipping: Once a path point is detected crossing a high-intensity boundary, discard that point and all subsequent path points; Micro-offset: Fine-tune the path point attempting to cross the boundary along the normal direction of the boundary to fit it but not cross the boundary; Reconnecting from breakpoint: Treat crossing the boundary as a path interruption, stop the current path extension at that location, and possibly restart a new path segment on the other side of the boundary (if needed). Output boundary-consistent path coordinates (spiral). coords_aligned .

[0048] Based on geometric parameters and boundary-consistent path coordinates, a similarity model result is generated. The boundary-consistent path coordinates are used for feature sampling in similarity modeling.

[0049] In a further implementation, generating similarity model results also includes:

[0050] On the boundary-consistent path coordinates, target path feature sequences and reference path feature sequences are extracted from the target feature list and the reference feature list, respectively.

[0051] Specifically, using boundary-consistent path coordinates spiral coords_alignedBy using differentiable sampling methods such as bilinear interpolation, the target feature F target Upsampling yields the target path feature sequence target seq And in the reference feature list F refs Upsampling of each feature map yields a reference path feature sequence. seq Preferably, before calculating similarity, the target path feature sequence is... seq and reference path feature sequence seq Perform channel normalization and scale alignment. Different reference images may have different feature distributions (i.e., distribution offsets). Normalization (such as L2 normalization or LayerNormalization) can reduce the interference caused by these offsets and make similarity comparisons more equitable.

[0052] Gaussian decay weights are constructed based on the distance from the boundary-consistent path coordinates to the path center.

[0053] In this embodiment, sampling points closer to the path center point (i.e., the current repair position) are given higher weight in the similarity calculation. For example, the Gaussian decay weight w(d) can be expressed as: w(d) = exp(-d 2 / (2σ 2 )); where d is the Euclidean distance from the current path coordinate point to the path center point (e.g., (0, 0)); σ is the variance of the Gaussian kernel. In some implementations, σ can be a fixed hyperparameter, or, preferably, σ can also be derived from local context features. feat It is obtained through dynamic prediction.

[0054] The path-level similarity is calculated by applying Gaussian decay weights to the feature sequences of the target path and the feature sequences of the reference path.

[0055] Specifically, for each valid reference image i (from the valid reference index index) valid_refs ), along the boundary consistent path coordinates spiral coords_aligned For each point p, calculate the cosine similarity (cos) between the normalized target feature and the reference feature at that point. sim (p), then multiplied by the Gaussian decay weight w(d(p)), and finally integrated (summation in the discrete case): Path-level similarity S spiral [i] = ∑ p ( w(d(p)) × cos sim (p) ) / Z; where Z is the normalization factor (e.g., ∑ p w(d(p))). If multiple spiral paths are generated, the path-level similarity S of the multiple paths also needs to be calculated.spiral The results are aggregated. In a preferred embodiment, this aggregation can be performed using learnable path weights for weighted summation to obtain the final spatial domain similarity S. spiral_agg [i].

[0056] Based on geometric parameters, boundary-consistent path coordinates, and path-level similarity, a similarity model result is generated.

[0057] In other words, path-level similarity also constitutes part of the similarity model results.

[0058] In a further implementation, the similarity model result also includes: the initial fusion weight W. ref_0 Specifically, based on spatial domain similarity S spiral_agg The degree of dispersion (e.g., variance or entropy) in the reference dimension (i.e., between different reference images), and the effective reference index. valid_refs The number of reference images is used to adaptively estimate the temperature parameter τ. When local ambiguity is high (e.g., multiple reference images provide moderate similarity), the temperature parameter τ should be increased to smooth the Softmax distribution, suppressing weight sharpening and achieving soft selection; when ambiguity is low (one reference image is significantly better than others), the temperature parameter τ should be decreased to highlight high-confidence references. The spatial domain similarity S... spiral_agg Temperature-based Softmax normalization is performed based on the temperature parameter τ to obtain the initial fusion weights W. ref_0 For valid reference indexes valid_refs Invalid references other than those specified are assigned weights that are set to zero and renormalized. The initial fusion weight W... ref_0 This will be used as input for geometric alignment.

[0059] Understandably, when generating a spiral path on a finite spatial grid (e.g., 32x32), if the radius is too large, a large number of sampling points may exceed the boundary. Optionally, the effectiveness of the sampling path can be ensured through several mechanisms: parameter constraints and regularization: a geometric parameter subnet regressing geometric parameters a (initial radius) and b (radius growth rate), whose output is passed through a non-negative projection (e.g., ReLU activation) such that a ≥ 0 and b ≥ 0. Further, through a learnable regularization term, the maximum sampling radius (e.g., a + bθ) is preferably selected. max Implicit constraints are placed within a reasonable proportion of the spatial dimension (e.g., 32) (e.g., the maximum radius preferably does not exceed 16) to avoid excessive out-of-bounds errors. Explicit clipping and rollback: The generated path coordinate sequence is then processed by the spiral... coords_rawPerform explicit out-of-bounds clipping and interpolation rollback. That is, for coordinate points exceeding the range (0, 31), clip them to the boundary; or, during path calculation, if the next step is detected to be out of bounds, terminate the path early or roll back to the previous valid point. Boundary consistency correction: through boundary feature B. feat The detection process involves actively correcting the path when it crosses strong boundaries (e.g., cropping or micro-offset), further ensuring that the sampled points fall within the same meaningful semantic region. This embodiment does not blindly generate large-radius spirals, but rather employs a three-layer mechanism of parameter constraints, explicit cropping, and boundary correction to ensure that most points sampled on a small-sized feature map remain valid and meaningful.

[0060] In one exemplary embodiment, generating alignment parameters includes:

[0061] Perform geometric alignment to generate alignment parameters. Specifically, this includes: constructing an objective function for solving the alignment parameters; the objective function includes a boundary regularization term calculated based on boundary features, and Jacobi non-folding constraints.

[0062] In this embodiment, the geometric alignment preferably employs a thin-plate spline deformation model. Specifically, a model is constructed to solve for the TPS parameters (i.e., the alignment parameter θ). tps The objective function is to estimate the control points (e.g., on a 16x16 regular grid G). regular The control point CP set above 16x16 The target displacement. Optionally, within the neighborhood of each control point, based on the initial fusion weight W. ref_0 The offset responses of each reference image at this location are weighted to obtain the target displacement estimate CP of the control point. disp_0 The objective function includes a data consistency term (to make the control points CP...). 16x16 Try to shift to CP disp_0 The boundary regularization term is implemented through a smoothing term. Specifically, based on boundary feature B... feat Calculate the boundary-guided smoothing weights reg weight The smoothing weight reg weight High weights are applied near high-intensity boundaries (e.g., object contours) to impose stronger smoothing constraints and limit drastic distortion of the deformation field across the boundaries. Solving the objective function of the TPS involves a radial basis function (RBF) kernel. Preferably, the RBF kernel uses U(r) = r 2 log(r 2 The form is given by ), where U(r) is the radial basis function and r is the Euclidean distance. The solution process can be schematically represented as solving a system of linear equations: (K tps + λ smooth I) w + P poly a = Vctrl ;P poly T w = 0; where K tps The TPS kernel matrix is ​​represented by w, which represents the radial basis weights; P is the weight of the radial basis. poly V is a polynomial matrix; a is the affine coefficient; V ctrl CP estimation from control point target displacement disp_0 The result is obtained by superposition; λ smooth The smoothing coefficients are spatially distributed by the smoothing weights reg. weight Scheduling, T I is the transpose of I, and I is the identity matrix.

[0063] The implementation of the Jacobian non-folding constraint includes: introducing a logarithmic barrier term to the local Jacobian determinant induced by geometric alignment (e.g., TPS deformation) to construct a barriered objective function; employing a differentiable numerical approximation to handle the logarithmic barrier term to maintain the gradient path openness in the barriered objective function; and generating a solvable expression for solving the alignment parameters based on the barriered objective function. In other words, to prevent TPS deformation from causing spatial folding (i.e., the Jacobian determinant is negative or zero), leading to topological errors, a penalty term based on the Jacobian determinant is added to the objective function. Preferably, this penalty term takes the form of a logarithmic barrier, which rapidly increases to infinity when the Jacobian determinant approaches zero, thereby preventing the optimization process from entering the folded region. To make the barrier function differentiable during backpropagation, a differentiable numerical approximation method is used. To ensure numerical stability when solving the barrier-bound objective function, a block-based solution strategy can be adopted (first solving the kernel matrix and polynomial subspace, then simultaneously solving w and a), and the Cholesky decomposition or QR decomposition can be automatically selected based on the condition number. Simultaneously, lower bound pruning or diagonal lifting can be applied to the singular radius to suppress numerical instability. The stable and non-folded TPS deformation parameter θ is obtained by solving. tps This refers to the alignment parameter.

[0064] like Figure 4 As shown, in another exemplary embodiment, generating initial fusion features includes:

[0065] Apply the alignment parameters to the reference feature list to obtain the alignment reference features.

[0066] Specifically, using the alignment parameter θ tps and rule-controlled grid G regular Generate TPS sampling grid in the feature resolution coordinate domain (e.g., 32x32). grid_tps Using TPS sampling grid grid_tps Through differentiable resampling (e.g., differentiable resampling grid) sample(operation), on the reference feature list F refs Geometric alignment is performed on each feature map in the dataset to obtain the alignment reference feature F. refs_warped Preferably, during this process, an alignment validity marker warp is generated simultaneously. valid_mask This is used to identify invalid regions that are mapped outside the feature map boundary (out of bounds) or produce holes during the resampling process.

[0067] Along the learnable spiral path, the alignment reference features and target features are sampled separately to calculate the spatial domain similarity re-evaluation.

[0068] Specifically, in the boundary-consistent path coordinates spiral coords_aligned Above, realign the reference feature F. refs_warped and target features F target Sampling is performed, and the cosine similarity of the Gaussian decay weighted integral is calculated to obtain the spatial domain similarity re-evaluation S. spiral_refined [i]. The revalued S spiral_refined Spatial domain similarity S spiral _ agg It is more accurate because it has taken into account macroscopic geometric deformation.

[0069] On the learnable spiral path, a bandpass filter is applied to the sampling sequences of the aligned reference features and target features, and the frequency domain similarity is calculated.

[0070] In this embodiment, calculating similarity solely in the spatial domain is insufficient, especially in texture-rich regions; consistency in the frequency domain is equally important. Specifically, in the boundary-consistent path coordinates spiral coords_aligned Above, extract target features F target Path sequence and alignment reference feature F refs_warped The path sequences are then processed. A Discrete Fourier Transform (DFT) is performed on these sequences. A bandpass filter is applied to suppress excessively low-frequency bands (e.g., DC components, representing overall brightness) and excessively high-frequency bands (e.g., noise). In some implementations, the window (cutoff frequency) of this bandpass filter can be determined based on local context features. feat The texture complexity is adaptively set. Similarity is calculated on the filtered frequency domain sequence, for example, by comparing phase consistency or energy alignment, and cross-path aggregation (e.g., using w(d) weighting) is performed to obtain the frequency domain similarity S. fourier [i].

[0071] Based on local contextual features, spatial domain similarity re-evaluation and frequency domain similarity are fused in a learnable proportion to obtain dual-domain similarity.

[0072] In this embodiment, spatial domain information and frequency domain information are fused. Specifically, this is achieved through a small network (with context). feat (As input) Predict the fusion scaling factor β, which is spatially variable. Two-domain similarity S dual [i] =β × S spiral_refined [i] + (1 - β) × S fourier [i]; The fusion scaling factor β is learnable and has physical meaning. For example, in regions with clear structures and well-defined edges, the context... feat It predicts a higher β, making spatial domain similarity dominant; while in regions with rich texture and dense details, it predicts a lower β, increasing the proportion of frequency domain similarity.

[0073] Initial fusion features are generated based on dual-domain similarity.

[0074] In other words, the two-domain similarity is used to generate the initial fused features.

[0075] Specifically, the dual-domain similarity S dual [i], using the estimated temperature parameter τ(τ), perform temperature-based softmax to obtain the updated reference fusion weights W. ref_1 During the normalization process, the alignment validity marker warp must be used. valid_mask For positions deemed invalid (out of bounds or with holes) in the TPS alignment, the weights are reset to zero and renormalized. The updated reference fusion weights W are then used. ref_1 For alignment reference feature F refs_warped We perform weighted summation (fusion) to obtain the initial fusion feature F. fused_0 The initial fusion feature F fused_0 Alignment parameter θ tps Updated reference fusion weights W ref_1 These will be sent as products to the next closed-loop feedback stage.

[0076] like Figure 2 As shown, according to one aspect of this application, calculating the error pattern diagram includes:

[0077] Based on the initial fusion features and the target image, the spatial residual and structural similarity residual are evaluated to obtain the spatial domain error components.

[0078] In this embodiment, the initial fusion feature is F fused_0 The target image is T target In the initial fusion feature F fused_0 The observable region (e.g., defined by a mask) M (Instruction) Calculate its relationship with the target image T targetThe difference between them. Spatial residuals can be L1 or L2 norm distances, while structural similarity residuals can be the difference between SSIM (Structural Similarity Index) and MS-SSIM. The combination of the two constitutes the spatial domain error component E. spatial .

[0079] A frequency domain transformation is performed on the initial fused features and the target image to calculate the bandpass energy difference and obtain the frequency domain error components.

[0080] In this embodiment, texture mismatches that are difficult to describe by spatial domain residuals are captured. Specifically, for the initial fused feature F fused_0 and target image T target Perform a Fourier transform (DFT) or wavelet transform on (or its characteristics), and compare the energy spectrum or coefficient differences between the two within a specific mid-to-high frequency bandpass window to obtain the frequency domain error component E. freq .

[0081] The continuity of the boundary gradient is evaluated based on the boundary characteristics, and boundary error components are formed.

[0082] Specifically, using boundary feature B feat Evaluation of initial fusion characteristics F fused_0 In boundary feature B feat Whether the gradient within the indicated strong boundary band is continuous and consistent with the target forms the boundary error component E. boundary .

[0083] The error pattern diagram is formed by combining the spatial domain error components, frequency domain error components, and boundary error components.

[0084] In one implementation, combination refers to combining the spatial domain error components E spatial Frequency domain error component E freq and boundary error component E boundary The three are concatenated along the channel dimension to form a multi-channel error pattern diagram E. pattern Error pattern diagram E pattern It fully characterizes the current fusion results in four dimensions: space, structure, frequency, and boundary.

[0085] In a further embodiment, closed-loop feedback is performed, including:

[0086] The first feedback branch (error → offset field): analyzes the error pattern diagram and generates offset residuals, which are used to update the sampling coordinates used in similarity modeling.

[0087] Specifically, the error pattern diagram E pattern Input a small residual network (e.g., several convolutional layers), and generate the offset residual field Δ via regression.O Used for sampling grids (e.g., sampling) grid_0 and sampling grid_tps Perform a first-order incremental update. In other words, if the error pattern map E pattern If a region shows poor fusion, the branch will fine-tune (offset) the coordinates of the sampling points used to calculate the spiral path similarity, attempting to find a better matching source in the neighborhood. Preferably, to ensure stable updates, this update rule adopts a momentum and pruning strategy. That is, the offset of the current step will partially accumulate the momentum of the previous step, and the total offset amplitude will be pruned within a preset maximum value (e.g., 1-2 pixels) to prevent oscillations. In some implementations, sampling can also be... grid_0 (Free sampling grid) updates and sampling grid_tps The update of the (TPS deformable mesh) is performed with consistency correction to prevent contradictions, and a rollback strategy is executed when the update causes the total error to increase.

[0088] The second feedback branch (error → TPS control point): The control point residual is derived from the error pattern diagram, and the control point residual is used to correct the alignment parameters.

[0089] In this embodiment, the second feedback branch is used to correct the macroscopic geometric alignment. Specifically, the error pattern diagram E pattern Used at TPS control point CP 16x16 The residual Δ of generating control points in the neighborhood CP The derivation process can be implemented in several ways. As one option, a graph convolutional network (GCN) can be used, where the TPS control points serve as graph nodes, based on the error pattern graph E. pattern Regressing Δ using statistics within its neighborhood CP As an alternative, locally weighted regression can be used, based on the error pattern plot E. pattern The distribution of the control points is used to calculate the required displacement. Preferably, the generated Δ CP It will undergo boundary regularization, for example, in strong boundaries (from boundary feature B). feat The residual Δ of the control points near the control points is... CP A greater smoothing constraint will be applied to prevent control points from undergoing cross-boundary displacement.

[0090] In one possible implementation, the control point residuals are used to correct alignment parameters, including:

[0091] Read the current target displacement of the control point, update the target displacement of the control point based on the control point residual, and obtain the corrected target displacement.

[0092] In other words, the target displacement of the geometrically aligned control points is updated based on the control point residuals.

[0093] Specifically, the control point residual Δ CP CP superimposed on the target displacement estimation of the control point disp_0 The corrected displacement target CP is obtained above. disp_corr .

[0094] Based on the corrected displacement target, while preserving the Jacobian non-folding constraint, the alignment parameters are re-solved, and the corrected alignment parameters are output.

[0095] In this embodiment, the control point residual Δ is not directly used. CP Instead of treating the deformation increment as a new optimization objective, the TPS equations are resolved. Furthermore, during the resolution process, the Jacobian non-folding constraints (e.g., logarithmic barrier terms) must be maintained to ensure the corrected alignment parameter θ... tps_updated It remains smooth and without topological folds.

[0096] The third feedback branch (error → cross-network attention): generates a cross-network attention gating from the error pattern graph, which is used to modulate the attention intensity during dual-domain fusion execution.

[0097] In this embodiment, the third feedback branch is used to adjust the information flow at the feature extraction level. Specifically, the error pattern diagram E pattern Used to generate one or more cross-network attention gating A cross Cross-network attention preferably operates on the encoder pyramid (e.g., UNet or FPN architecture). Gated A cross This is used to modulate the attention intensity between the target coding pyramid and the reference coding pyramid at different scales. In the error pattern diagram E... pattern This gate displays areas with high errors, suppressing reference feature channels that cause inconsistencies while amplifying the attention intensity of channels that align with the target features.

[0098] Furthermore, while executing the three feedback branches mentioned above, a parallel stabilization control loop is also executed. That is, executing closed-loop feedback also includes: calculating the local error statistics of the error pattern diagram; and, based on the local error statistics, adaptively updating the temperature parameters used for normalizing the similarity model results and the radius growth rate used for generating the learnable spiral path.

[0099] Specifically, in error mode diagram E pattern The mean and variance of local errors are calculated within a sliding window to generate local error statistics. local Based on local error statistics (stats) local Adjust the temperature parameter τ. When the mean or variance of local errors increases, increase the temperature parameter τ to adjust the Softmax weights (e.g., W).ref_iter In regions of high uncertainty, the algorithm becomes smoother, reducing weight sharpness and avoiding overconfidence in erroneous reference sources. This is based on local error statistics. local Adjust the radius growth rate b of the Archimedean spiral r = a + bθ. When local errors increase, decrease the radius growth rate b to make the spiral path sample more densely in the nearest region (because r grows more slowly for the same angle change in θ). Conversely, when errors decrease, the radius growth rate b can be appropriately increased to improve far-field exploration. Preferably, apply synchronous smoothing and boundary band constraint to the updates of the temperature parameter τ and the radius growth rate b to prevent rapid oscillation of the parameters near the boundary.

[0100] Understandably, the closed-loop feedback process is executed cyclically. In one iteration period k, based on the updated sampling grid (from the first feedback branch) and the updated alignment parameter θ... tps_updated (From the second feedback branch), the updated attention (from the third feedback branch), and the updated temperature parameter τ. updated and radius growth rate b updated (From stabilization control), the similarity calculation and fusion process is re-executed (i.e., the bi-domain similarity S is re-estimated). dual_new ), to obtain the intermediate fusion weight W ref_iter and intermediate fusion feature F fused_iter Preferably, to ensure convergence, the intermediate fusion weight W is... ref_iter and intermediate fusion feature F fused_iter Applying momentum smoothing and weight clipping, we obtain the smoothed W. ref_smooth and F fused_smooth The smoothed intermediate fusion feature F fused_smooth As the input for the next round k+1, it is fed back to recalculate the error pattern diagram E. pattern During the process, the convergence criteria for loop termination include: error pattern diagram E. pattern The overall energy decrease is below a certain threshold; the smoothed intermediate fusion weight W ref_smooth The change in value is below a certain threshold; the preset iteration limit (e.g., 5 or 10 times) is reached. When the loop terminates, the output W... ref_smooth and F fused_smooth That is, the final fusion weight W reffinal and the final fusion feature F fused .

[0101] In some complex restoration scenarios, fixed-density sampling may not be able to adapt to drastic changes in local texture. A preferred implementation provides a dynamic adaptive adjustment mechanism for the spiral sampling density. Specifically, the system adjusts the density based on local error statistics (stats). localAdaptively schedule the spiral radius growth rate b. When the error pattern diagram E... pattern When the current region error increases (indicating poor fusion performance and high uncertainty), the system will automatically reduce the radius growth rate b. At the same angular step size θ... step With the same path length, the radius r = a + bθ grows more slowly, making the spiral path denser in the nearest neighbor region (small radius range). This improves the algorithm's robustness and detail capture ability in difficult regions (such as fine textures or complex boundaries).

[0102] In one embodiment of this application, the final fusion feature is decoded to obtain the repaired image. Specifically, the final fusion feature F output after iterative convergence is... fused The input is fed into a decoder network (such as a VAE decoder or a UNet-structured decoder), which reconstructs the image space from the feature space back into the image space, resulting in the final repaired image I. recon .

[0103] In this embodiment, a distinction is made between training (or self-supervised) scenarios and inference (or deployment) scenarios: In training or self-supervised scenarios, the system needs to calculate the overall energy function (loss function) E. total And all learnable parameters (including encoder, decoder, geometric parameter subnet, offset field residual network, control point residual network, attention gating network, etc.) are updated through backpropagation. Preferably, the overall energy function E total It can be represented as: E total = E data + γ E smooth + η E boundary ; where E data This is a data fidelity item used to measure the final restored image I. recon Differences from real images. E data It can be L1 loss, L2 loss, perceptual loss (e.g., VGG feature distance), or a weighted combination thereof. γ is the weight coefficient of the smoothing term; E smooth For smoothing regularization terms, such as for the final repaired image I recon Apply total variational constraints to suppress artifacts. η is the boundary term weight coefficient; E boundary This is a boundary consistency term used to penalize boundary features B. feat The gradient generated within the indicated boundary band is discontinuous or ambiguous. Furthermore, in the training scenario, to ensure the reproducibility and analyzability of the experiment, the system should also record the change trajectories of the temperature parameter τ and the radius growth rate b regulated by the stabilization control loop. In the inference or deployment scenario, the system performs forward computation and outputs the repaired image I. final (i.e. I recon ).

[0104] In another embodiment of this application, it further includes:

[0105] While outputting the repaired image, a summary of key intermediate values ​​is sent back.

[0106] The key intermediate quantities summary includes: the final fusion weights derived from the similarity model results after closed-loop feedback, the final geometric parameters defining the learnable spiral path, and the final alignment parameters after iterative updates following closed-loop feedback.

[0107] Specifically, the key intermediate quantity summary contains three key intermediate tensors: the final fusion weight W reffinal This tensor (or list of tensors) shows which reference images, and with what weight, contribute to the final fusion result at each spatial location; the final geometric parameters spiral params This tensor (or list of tensors) shows the Archimedean spirals (a, b, θ) learned by the system for sampling at each spatial location. step The specific form of (etc.); the final alignment parameter θ tps_final This tensor (or list of tensors) defines the TPS geometric deformation field ultimately adopted by the system. Feeding back these intermediate quantities enables traceability and verifiability on the engineering side. For example, when ghosting or patchwork marks appear in a certain area of ​​the repair result, the final fusion weight W can be visualized. reffinal To determine which reference image caused the contamination, the final alignment parameter θ is visualized. tps_final To determine if unreasonable geometric distortion has occurred, or by visualizing the final geometric parameters of the spiral params This determines whether the sampling range is too large or too small. It provides crucial internal state information for algorithm debugging, iterative optimization, and comparative experiments.

[0108] In a preferred implementation, the encoder (e.g., a VAE encoder) receives the target image T. target Having (B, 3, H) in W in The dimensions of the target feature are given by (B, 3, 256, 256). The output target feature F... target It has dimensions (B, C, H, W), for example (B, 512, 32, 32). Dimension C (e.g., 512) is the channel dimension or feature dimension. Cosine similarity calculations are performed in this 512-dimensional feature vector space. Dimensions H and W (e.g., 32x32) are spatial dimensions. All spatial operations in this application are performed in this (H, W) spatial coordinate system, including: spiral path coordinates. coords Generation of TPS sampling grids; grid_tps Generation and application of offset residual ΔO Applications.

[0109] In a detailed embodiment, it is assumed that at p0 = (10, 10), the geometric parameter subnet regresses the Archimedean spiral path, whose parameters are spiral params The initial radius is a = 1.0 (pixels); the radius growth rate is b = 0.2 (pixels / radians); and the angle step size is θ. step=π / 4 (radians); Number of path points L = 5 (i.e., k=0, 1, 2, 3, 4). Based on the above parameters, calculate the (x, y) offsets of the 5 sampling points in the local coordinate system of p0: k=0: θ= 0×π / 4 = 0; r = 1.0 + 0.2 × 0 =1.0; Δx = 1.0 × cos(0) = 1.0; Δy = 1.0 × sin(0) = 0.0; Absolute coordinates: (10+1.0, 10+0.0) = (11.0, 10.0); Path distance d0 = 1.0. k=1: θ= 1 × π / 4 ≈0.785 radians; r = 1.0 +0.2 × 0.785 ≈1.157; Δx = 1.157 × cos(π / 4) ≈ 1.157 × 0.707 ≈ 0.818; Δy= 1.157 × sin(π / 4) ≈ 1.157 × 0.707 ≈ 0.818; Absolute coordinates: (10.818, 10.818); Path distance d1 = 1.157. k=2: θ= 2 ×π / 4 = π / 2 ≈1.571 radians; r = 1.0 + 0.2 ×1.571 ≈1.314; Δx= 1.314 × cos(π / 2) = 0.0; Δy = 1.314 × sin(π / 2) = 1.314; Absolute coordinates: (10.0, 11.314); Path distance d2 = 1.314. k=3: θ= 3π / 4 ≈ 2.356 radians; r = 1.0 + 0.2 ×2.356 ≈1.471; Δx = 1.471 × cos(3π / 4) ≈ 1.471 × -0.707 ≈ -1.040; Δy =1.471 × sin(3π / 4) ≈ 1.471 × 0.707 ≈ 1.040; Absolute coordinates: (8.960, 11.040); Path distance d3= 1.471. k=4: θ= 4π / 4 = π≈3.142 radians; r = 1.0 + 0.2 ×3.142 ≈1.628; Δx =1.628 × cos(π) = -1.628; Δy= 1.628 × sin(π) = 0.0; Absolute coordinates: (8.372, 10.0); Path distance d4 = 1.628. Assume that none of the above 5 points have exceeded the boundary and no boundary correction has been triggered. Assume the Gaussian decay weight σ=1.5. Calculate w(d) for the 5 points. k ) = exp(-d k 2 / (2×1.5 2 )) = exp(-d k2 / 4.5): w(d0) = exp(-1.0 2 / 4.5) ≈ 0.799; w(d1) = exp(-1.157 2 / 4.5) ≈ 0.742; w(d2) = exp(-1.314 2 / 4.5)≈ 0.681; w(d3) = exp(-1.471 2 / 4.5) ≈ 0.618; w(d4) = exp(-1.628 2 / 4.5) ≈0.555; weights and ∑w(d k ) ≈ 3.395. Assume that on the reference image R1, samples are taken at the above 5 coordinate points using bilinear interpolation, and compared with the target feature F. target Calculate the cosine similarity of the 512-dimensional feature vectors at the corresponding points to obtain the cosine similarity. sim The (k) sequence is: [0.9, 0.8, 0.95, 0.7, 0.6]. Calculate the path-level similarity S. spiral [R1]:S spiral [R1] = ∑ k ( w(d k ) ×cos sim (k) ) / ∑ w(d k The spatial similarity S is calculated as follows: S = (0.799 × 0.9 + 0.742 × 0.8 + 0.681 × 0.95 + 0.618 × 0.7 + 0.555 × 0.6) / 3.395 = (0.719 + 0.594 + 0.647 + 0.433 + 0.333) / 3.395 ≈ 0.796. Assuming there are three reference images R1, R2, and R3, the calculated spatial similarity S... spiral_agg The values ​​are [0.796, 0.700, 0.300]. Assuming the temperature parameter τ = 0.1, calculate the initial fusion weight W. ref_0 :W ref_0 = Softmax(S spiral_agg / τ) = Softmax([7.96, 7.00, 3.00]) ≈ [0.701, 0.269, 0.030]. This indicates that R1 is selected as the primary reference source at p0=(10, 10). TPS alignment is based on the initial fusion weights W. ref_0 ≈ [0.7, 0.27, 0.03], Control point displacement estimation CP disp_0 It will be mainly determined by the offset of R1. Assume that the solved θ tpsGeometric alignment was performed on R1, R2, and R3, generating the alignment reference feature F. refs_warped In alignment reference feature F refs_warped On [R1], the similarity is recalculated using 5 coordinate points. Assuming that the alignment is more accurate, cos... sim The (k) sequence is boosted to [0.95, 0.9, 0.96, 0.8, 0.7]. The spatial domain similarity reassessment value S... spiral_refined [R1] ≈ 0.871. The feature sequences obtained from the above 5 sampling points are subjected to DFT and bandpass filtering. Assuming the calculated frequency domain similarity S... fourier [R1] = 0.750. Assuming p0 = (10, 10) is in a region where texture and structure are equally important, the context... feat The predicted fusion ratio β = 0.6. Bi-domain similarity S dual [R1] = β ×S spiral_refined [R1] + (1 - β) × S fourier [R1] = 0.6 × 0.871 + 0.4 × 0.750 = 0.523 + 0.300 = 0.823. Assume the bi-domain similarity S of R2 and R3 is... dual The values ​​are 0.720 and 0.310 respectively. The updated reference fusion weights W ref_1 = Softmax( [0.823, 0.720, 0.310] / τ ) = Softmax( [8.23, 7.20, 3.10] )≈[0.706, 0.252, 0.042]. Updated reference fusion weights W ref_1 The initial fusion feature F will be used to generate the initial fusion feature. fused_0 And send it into the closed-loop feedback.

[0110] According to one aspect of this application, an image restoration apparatus includes: a processor; and a memory storing computer instructions; when executed by the processor, the computer instructions implement the image restoration method described in any of the above embodiments.

[0111] In one possible implementation, generating the similarity model results involves: reading the target features, reference feature list, boundary features, mask, and initial sampling grid; aggregating local windows, boundary strength, and observability at each spatial location to form a local context tensor; filtering out references significantly inconsistent with that location based on the confidence levels of the mask and boundary features to obtain a valid reference index. Using the local context tensor as input, predicting the spiral family parameters for each spatial location, including the initial radius, radius growth rate, angle step size, and path length of each spiral; and outputting the upper limit of the sampling angle. The geometric definition uses an Archimedean spiral: r = a + b·θ, where a represents the initial radius, b represents the radius growth rate, and θ is the angle variable. A learnable regularization is applied to the parameters to restrict the initial radius and radius growth rate to be non-negative, and upper and lower bounds are set for the angle step size and path length to avoid sparse or overly dense sampling. Specifically, the local context tensor, target features, and boundary features are read, aligned spatially, and concatenated by channel to form the geometric regression input. This input is then fed into a lightweight subnet (pointwise convolution + depthwise separable convolution + local self-attention) to regress the starting radius, radius growth rate, angle step, and path length of each path, yielding unconstrained geometric parameters. The unconstrained geometric parameters and boundary features are then read, and a non-negative projection is applied to the starting radius and radius growth rate, while upper and lower bound clipping is applied to the angle step and path length. Local boundary density is calculated using the boundary features to restrict the starting radius from crossing high-intensity boundary bands, resulting in projected geometric parameters. Finally, the projected geometric parameters and local context tensor are read to construct multi-path equilibrium constraints, preventing all paths from collapsing at the same radius or angle step. Entropy-based equilibrium regularization and neighboring path differential penalties are introduced into the parameters of each path, yielding regularized weights and regularized geometric parameters. Read the regularized geometric parameters and valid reference indices, adaptively set the upper limit of angle and the sampling point budget according to the local texture complexity and the number of valid references (to ensure that the amount of computation is controllable and covers a sufficient neighborhood), and output the upper limit of angle, sampling point budget, and spiral family parameters.

[0112] Multiple discrete path coordinate sequences are generated using an initial sampling grid and spiral family parameters. Coordinate correction and clipping are performed at high-intensity boundaries based on boundary characteristics to ensure the paths do not cross obvious fracture boundaries. Gaussian decaying base weights are constructed based on the distance to the path center, where the weight form is w(d) = exp(-d 2 / (2σ 2σ is predicted by the local context tensor. Specifically, the initial sampling grid, spiral family parameters and angle upper limits, and sampling point budget are read. A discrete angle sequence of the Archimedean spiral is constructed at each spatial location to generate a path coordinate sequence. Points that exceed the boundary and outside the image are cropped and interpolated back to obtain the cropped path coordinates. The cropped path coordinates and boundary features are read to detect the intersection of the path with high-intensity boundaries. Micro-offsets and breakpoint reconnection are performed on potential boundary-crossing path segments to ensure that path segments are in the same semantic region, resulting in boundary-consistent paths and path validity labels. The boundary-consistent paths and local context tensors are read, and Gaussian decay weights are calculated based on the distance to the path center. Combined with the path validity labels, nullification and renormalization are performed to generate basic weights. Kernel smoothing is performed on dense segments to avoid a few outliers dominating aggregation, and the path coordinate sequence and basic weights are output. On the path coordinate sequence, corresponding multi-point sequences are extracted from the target feature and reference feature lists to obtain the target path feature sequence and reference path feature sequence. Channel normalization and scale alignment are performed on both types of sequences to obtain normalized target path features and normalized reference path features, thereby reducing the distribution offset of different references. For each reference image, cosine similarity is calculated point-by-point on each spiral path, and weighted integral is performed using the basic weights to obtain path-level similarity. Multiple spirals are aggregated using learnable path weights to form spatial domain similarity. Based on the dispersion of path-level similarity in the reference dimension (such as variance or entropy) and the number of effective reference indices, a temperature parameter is estimated: the temperature is increased when local ambiguity is large to suppress weight sharpening, and the temperature is decreased when ambiguity is small to highlight high-confidence references. The spatial domain similarity is then temperature-modulated using Softmax normalization according to the temperature parameter to obtain the initial fusion weights; invalid references are set to zero according to the effective reference indices and renormalized.

[0113] In another possible embodiment, the final fusion feature is generated as follows: The initial fusion feature, target image, and mask are read; spatial residuals and structural similarity residuals are calculated in the observable region to obtain spatial domain error components; frequency domain transformation is performed on the initial fusion feature and target image to calculate the bandpass energy difference, obtaining frequency domain error components; gradient continuity within the boundary band is evaluated based on boundary features to form boundary error components. These three are then stitched together to form an error pattern map. Using the error pattern map as input, offset residuals are generated to perform small updates on the initial sampling grid and TPS sampling grid; the update rule employs a stabilization strategy of momentum and clipping to ensure gradual convergence of the update within a controllable range. The updated sampling grid is output. Specifically, the error pattern map is read, and the error pattern is mapped to offset residuals (containing horizontal and vertical components) through a small residual network; the offset residuals are used to perform a first-order incremental update on the initial sampling grid and TPS sampling grid to obtain a temporary sampling grid. The temporary sampling grid and the previous round's sampling grid state are read, and momentum smoothing and amplitude clipping are used to suppress oscillations and large step sizes, resulting in the updated sampling grid. Read the updated sampling grid and TPS sampling grid, perform consistency correction on both (to prevent contradictions between free offset and TPS deformation), and output the unified sampling grid and unified TPS sampling grid. Read the unified sampling grid, unified TPS sampling grid, and convergence monitoring metrics. If an out-of-bounds error or energy rise exceeding a threshold is detected, revert to the previous stable state; output the final updated sampling grid and final updated TPS sampling grid.

[0114] Using the error pattern map as input, control point residuals are generated. The control point set is corrected, and the TPS deformation parameters are resolved. During the solution process, Jacobian non-folding constraints and boundary regularization are added to ensure smooth geometric transformations without folding. The updated control points, updated deformation parameters, and updated TPS sampling mesh are output. Specifically, the error pattern map, control point set, and boundary features are read, and control point residuals are generated in the neighborhood of the control points through graph convolution or local weighted regression. Smoothing regularization is increased near strong boundaries to prevent control points from crossing boundary displacements, and regularized control point residuals are output. The regularized control point residuals and the current target displacement estimate of the control points are read to update the corrected control point displacements. While maintaining the Jacobian barrier and the original smooth structure, the solution is quickly resolved to obtain updated deformation parameters and a resolution cost cache (used for convergence criteria). The updated deformation parameters and re-resolution cost cache are read, and the Jacobian determinant of the deformation field is evaluated to determine if it is positive and above a threshold. If not, the magnitude of the regularization control point residual is reduced and the solution is re-resolved, or the process reverts to the stable TPS deformation parameters of the previous round. The final deformation parameters are output. A cross-network attention gating is generated using the error pattern graph to modulate the cross-network attention intensity between the target and reference encoding pyramids: inconsistent channels are suppressed in high-error regions, while channels consistent with the target are amplified. The output cross-network attention gating, along with the similarity results, determines the weight updates during the fusion process.

[0115] Calculate local error statistics (e.g., mean and variance of the error), and adjust the temperature parameter and the spiral radius growth rate simultaneously based on the statistics: increase the temperature to reduce weight sharpness and decrease the radius growth rate to densify nearest neighbor sampling when the error increases; adjust in the opposite way when the error decreases to improve far-field exploration. Output the updated temperature parameter and the updated radius growth rate. Specifically, read the error pattern diagram, calculate the mean and variance within the sliding window to form local error statistics. Read the local error statistics and the previous round of temperature parameters, and adaptively adjust the temperature according to the mean and variance of the error to make the reference weight smoother in the high uncertainty region and more concentrated in the low uncertainty region, thus obtaining the updated temperature parameter. Read the local error statistics and the previous round of radius growth rate b, decrease the radius growth rate in the error-increasing region to densify nearest neighbor sampling, and increase the radius growth rate in the error-decreasing region to improve far-field exploration, thus obtaining the updated radius growth rate. Read the updated temperature parameter, the updated radius growth rate, and boundary features, and apply synchronous smoothing and boundary band variation constraints to both to prevent the parameters from oscillating rapidly near the boundary; output the synchronized temperature parameter and the synchronized radius growth rate. Based on the updated sampling grid and the updated TPS sampling grid, the reference feature list is geometrically aligned again to obtain aligned reference features. A new path coordinate sequence is generated using the updated helical radius growth rate and helical family parameters, and sampling is performed on the target feature and aligned reference features to calculate a new spatial domain similarity. Bandpass similarity is calculated on the path sequence in the frequency domain to obtain frequency domain similarity. The two are then fused into a bi-domain similarity using a learnable ratio. Temperature-based softmax is performed based on the updated temperature parameters to obtain intermediate fusion weights, which are then used to generate intermediate fusion features. Momentum smoothing and weight clipping are applied to the intermediate fusion weights and intermediate fusion features to obtain smoothed fusion weights and smoothed fusion features. If the overall energy decrease of the error pattern map is insufficient, or the change in the smoothed fusion weights is below a threshold, or the iteration limit is reached, convergence is determined, and the final fusion features and final reference weights are output. Otherwise, the smoothed fusion result is fed back to continue the next round of updates.

[0116] In another possible embodiment, initial fusion features are generated, specifically by: reading target features, a list of reference features, boundary features, a regular control grid, a set of control points, initial fusion weights, spatial domain similarity, and a mask; performing a local correlation search within the neighborhood of each control point; and weighting the offset responses of each reference according to the initial fusion weights to obtain the target displacement estimate of the control points (representing the expected displacement of each control point in the target coordinate system). Based on the boundary features, boundary-guided smoothing weights are calculated to form regularization weights (increasing the smoothing weights at strong boundaries to limit cross-boundary distortion). The TPS kernel matrix and polynomial matrix of the thin-plate spline are constructed. The target displacement estimate of the control points, the set of control points, the TPS kernel matrix, the polynomial matrix, and the regularization weights are read to establish an objective function with boundary regularization and Jacobian non-folding constraints. The TPS deformation parameters are obtained by adding diagonal regularization and block solving. The data consistency and smoothing term is: (K tps + λ smooth I) w+ P poly a = V ctrl ;P poly T w = 0; where w is the radial basis weight, a is the affine coefficient, and V ctrl The result is obtained by superimposing the target displacement estimate of the control points with the set of control points; λ smooth The regularization weights are statistically scheduled. The non-folding constraint is to add a logarithmic barrier term to the Jacobian determinant during optimization to avoid local folding. A numerically stable decomposition method (such as Cholesky or QR) is used, and the singular radius is lower-bounded to obtain the TPS deformation parameters. Specifically, the target displacement estimates of control points, the control point set, the TPS kernel matrix, and the polynomial matrix are read. The target displacement estimates of control points are mapped to control point displacement vectors, constructing an objective function that includes data consistency and smoothing regularization. A spatially adaptive distribution of the smoothing coefficients is set according to the regularization weights to obtain the optimization problem definition. The optimization problem definition and boundary features are read, and a logarithmic barrier term is introduced to the local Jacobian determinant induced by deformation to construct a barriered objective function. The gradient path is kept open through a differentiable numerical approximation to obtain a solvable expression. The solveable expression is read, and a block-based solution strategy is adopted: first, the kernel matrix and polynomial subspace are decomposed, and then the radial basis weights and affine coefficients are solved simultaneously; Cholesky or QR decomposition is selected according to the condition number, and diagonal lifting is performed when necessary to obtain the TPS deformation parameters and numerical diagnostic indices. The TPS deformation parameters and numerical diagnostic indices are read, and the smoothing regularization is dynamically adjusted according to the diagnostic indices, and a lower bound is set for the near-zero radius term to suppress numerical instability; the stabilized TPS deformation parameters are output.

[0117] The TPS deformation parameters and rule control grid are read, and a TPS sampling grid is generated in the feature resolution coordinate domain. This sampling grid is used to perform differentiable resampling on each reference feature list to obtain aligned reference features, while simultaneously generating alignment validity markers for out-of-bounds and holes. The aligned reference features, target features, path coordinate sequences, and basic weights are read. Path feature sequences of the target and reference are extracted on each spiral path, and a Discrete Fourier Transform is performed with bandpass filtering (suppressing excessively low and high frequency bands) to obtain a frequency domain path representation sequence. Path-level frequency domain similarity is calculated based on the basic weights and aggregated across paths to output the frequency domain similarity. Specifically, the aligned reference features, target features, path coordinate sequences, and basic weights are read, and channel vector sequences of the target and reference are extracted on each path to form a target path sequence and a reference path sequence. A Discrete Fourier Transform is performed on both types of sequences to obtain the target frequency domain sequence and the reference frequency domain sequence. Read the target frequency domain sequence, reference frequency domain sequence, and local context tensor. Adaptively set the bandpass window based on local texture complexity and boundary density (suppressing excessively low and high frequency bands) to obtain the target frequency domain sequence and the reference frequency domain sequence. Read the target frequency domain sequence, the reference frequency domain sequence, and the basic weights. Calculate phase consistency and energy alignment in the frequency domain to obtain single-path frequency domain similarity. Aggregate across paths according to the basic weights to output the frequency domain similarity. Read the aligned reference features, target features, path coordinate sequence, and basic weights. Resample the path and integrate the cosine similarity on the aligned reference to obtain the spatial domain similarity re-evaluation (compared to the spatial domain similarity, this result already considers geometric alignment and hole clipping). Read the spatial domain similarity re-evaluation, frequency domain similarity, and local context tensor. Merge the two types of similarity into a dual-domain similarity using a context-conditional scaling factor. The scaling factor is predicted by the local context tensor, appropriately increasing the frequency domain proportion in texture-rich regions and increasing the spatial domain proportion in structurally clear regions. Output the dual-domain similarity. Read the bi-domain similarity and temperature parameters, perform temperature-based softmax on the reference dimension to obtain the reference fusion weights; simultaneously, use alignment validity markers to nullify and renormalize unusable locations. Read the alignment reference features, reference fusion weights, and alignment validity markers, perform weighted fusion on the reference dimension to obtain the initial fusion features; package the current geometry and similarity products to form an alignment and fusion product package (containing TPS sampling grid, TPS deformation parameters, bi-domain similarity, reference fusion weights, and initial fusion features).

[0118] This invention employs a learnable spiral path sampling method and introduces boundary consistency constraints. Specifically, it actively detects whether a path crosses a strong boundary using a boundary strength map, and corrects or prunes such paths. This ensures that feature sampling always occurs within a single semantic region, thus avoiding feature contamination and improving the boundary clarity of the repaired region. Through geometric alignment technology, when solving for alignment parameters (such as TPS parameters), both boundary regularization terms and Jacobi non-folding constraints are introduced. The former limits drastic deformation at the object's contour, while the latter mathematically guarantees the topological consistency of the deformation field through a logarithmic barrier term. This allows for the generation of smooth and structurally sound alignment results even under complex deformations. By constructing a closed-loop feedback mechanism, an error pattern map containing spatial, frequency, and boundary components is calculated to evaluate the quality of the initial fused features. The error map is used to drive three parallel correction branches: iteratively fine-tuning the coordinates of the spiral sampling, correcting the control points of geometric alignment, and modulating feature attention. This allows initial deviations to be continuously corrected, thereby improving the accuracy and fidelity of the final fused features.

[0119] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. An image inpainting method characterized by, The method comprises the following steps: obtaining a target image, a reference image set and a boundary intensity map; extracting a target feature of the target image, a reference feature list of the reference image set and a boundary feature of the boundary intensity map; based on the target feature, the reference feature list and the boundary feature, using a learnable spiral path to model similarity, generating a similarity model result; based on the similarity model result, the target feature and the boundary feature, performing geometric alignment and dual-domain fusion on the reference feature list to generate initial fusion features and alignment parameters; based on the initial fusion features, the target image, the boundary feature, the alignment parameters and the similarity model result, calculating an error mode map and performing closed-loop feedback to iteratively update and generate final fusion features; decoding the final fusion features to obtain a repaired image; performing closed-loop feedback, comprising: analyzing the error mode map to generate an offset residual, which is used to update the sampling coordinates used in the similarity modeling; deriving a control point residual from the error mode map, which is used to correct the alignment parameters; generating a cross-network attention gate from the error mode map, which is used to modulate the attention strength when performing dual-domain fusion; generating a similarity model result, comprising: aggregating the target feature, the reference feature list and the boundary feature to form a local context feature; inputting the local context feature into a pre-configured geometric parameter subnetwork to regress and predict geometric parameters; the geometric parameters define a learnable spiral path; wherein the learnable spiral path is an Archimedes spiral; generating a similarity model result based on the geometric parameters; generating a similarity model result, further comprising: generating a path coordinate sequence based on the geometric parameters; detecting the intersection of the path coordinate sequence and the preset high-intensity boundary according to the boundary feature; correcting or cropping the path segment in the path coordinate sequence that crosses the high-intensity boundary to generate a boundary-consistent path coordinate; generating a similarity model result based on the geometric parameters and the boundary-consistent path coordinate.

2. The method of claim 1, wherein, calculating an error mode map, comprising: based on the initial fusion features and the target image, evaluating spatial residuals and structural similarity residuals to obtain spatial domain error components; performing frequency domain transformation on the initial fusion features and the target image to calculate bandpass energy differences and obtain frequency domain error components; evaluating boundary gradient continuity based on the boundary feature to form boundary error components; combining the spatial domain error components, the frequency domain error components and the boundary error components to form an error mode map.

3. The method of claim 1, wherein, performing closed-loop feedback, further comprising: calculating local error statistics of the error mode map; based on the local error statistics, simultaneously and adaptively updating a temperature parameter used for normalization of the similarity model result and a radius growth rate used for generation of the learnable spiral path.

4. The method of claim 1, wherein, The control point residual is used to correct the alignment parameters, comprising: reading the current control point target displacement, updating the control point target displacement based on the control point residual to obtain the corrected displacement target; based on the corrected displacement target, re-solving the alignment parameters under the condition of maintaining the Jacobian non-fold constraint to output the corrected alignment parameters.

5. The method of claim 1, wherein, generating a similarity model result, further comprising: extracting a target path feature sequence and a reference path feature sequence from the target feature and the reference feature list respectively on the boundary-consistent path coordinate; constructing a Gaussian decay weight based on a distance from a boundary consistent path coordinate to a path center; applying the Gaussian decay weight to a weighted integration of a target path feature sequence and a reference path feature sequence to calculate a path level similarity; generating a similarity model result based on the geometric parameter, the boundary consistent path coordinate, and the path level similarity.

6. The method of claim 1, wherein, generating an initial fused feature, including: applying the alignment parameter to the reference feature list to obtain an aligned reference feature; sampling the aligned reference feature and the target feature on the learnable spiral path to calculate a spatial domain similarity re-estimation; applying a band-pass filter to the sampling sequence of the aligned reference feature and the target feature on the learnable spiral path to calculate a frequency domain similarity; fusing the spatial domain similarity re-estimation and the frequency domain similarity in a learnable proportion according to the local context feature to obtain a dual domain similarity; generating the initial fused feature based on the dual domain similarity.

7. The method of claim 1, wherein, generating the alignment parameter, including: constructing an objective function for solving the alignment parameter; the objective function includes a boundary regularization term calculated based on a boundary feature, and a Jacobian non-fold constraint.

Images