An image interpolation method based on two-stage perturbation and DR split expansion

Abstract

The application belongs to the technical field of image interpolation, and specifically discloses an image interpolation method based on double-stage disturbance and DR split expansion. The method takes a linear interpolation operator as an initialization module of a network to perform initial interpolation, maps the linear interpolation operator to a graph adjacency matrix through an interpolation operator theorem, and ensures the stability of the training process and the reliability of the interpolation result. In addition, the application introduces a double-stage optimization model of a directed graph disturbance matrix and an undirected graph disturbance matrix, and expands the model into a learnable neural network structure by using a DR split iteration algorithm, so that high-precision and interpretable image interpolation is realized. The directed graph adaptively models and corrects the connection relationship between observed pixels and pixels to be interpolated, and the undirected graph locally refines the smoothness between the pixels to be interpolated, effectively enhances the high-frequency details and overall smoothness of the image while maintaining the original structure information, and ensures the accuracy of the image interpolation result.

Description

Technical Field

[0001] This invention belongs to the field of image interpolation and relates to an image interpolation method based on two-stage perturbation and DR splitting expansion. Background Technology

[0002] Image interpolation is a fundamental research area in image processing and computer vision. Its main purpose is to estimate and reconstruct missing or low-resolution pixels in an image using information from some known pixels, thereby obtaining a more complete and higher-resolution image. Traditional image interpolation methods, such as bilinear interpolation, bicubic interpolation, and spline interpolation, have the advantages of being computationally simple and easy to implement. However, because these interpolation methods are usually based on local weighted averaging or polynomial fitting, their interpolation results often exhibit blurring and structural distortion in complex textures, edges, and detailed regions. Furthermore, traditional interpolation methods cannot adaptively adjust the interpolation weights, making it difficult to achieve ideal reconstruction results simultaneously in high-frequency changing regions and low-texture regions.

[0003] With the development of deep learning, image interpolation and reconstruction methods based on Convolutional Neural Networks (CNNs) and Visual Transformers have gradually become a research hotspot. These methods achieve complex feature transformations from low to high resolution through end-to-end nonlinear mapping, achieving significant improvements in objective evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). However, these data-driven models often lack interpretability, have a large parameter scale, rely on a large amount of labeled data for training, have high demands on computational resources and GPU memory, and exhibit poor stability in reconstruction results when input noise or observation gaps are significant. Furthermore, traditional deep networks often neglect the topological relationships between pixels in interpolation tasks, resulting in insufficient modeling of local structure and global consistency. Therefore, how to construct an efficient interpolation method that is both mathematically interpretable and adaptively captures image structural features has become a key research problem.

[0004] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0005] To address the problems of insufficient interpretability, fixed structure, and limited interpolation accuracy in existing technologies, this invention proposes an image interpolation method based on two-stage perturbation and DR split expansion. The traditional linear interpolation operator is mapped to an initial adjacency matrix. By introducing a two-stage optimization model of directed and undirected graph perturbation matrices, and using the DR split iterative algorithm to expand it into a learnable neural network structure, high-precision and interpretable image interpolation can be achieved.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An image interpolation method based on two-stage perturbation and DR splitting expansion includes the following steps:

[0008] Step 1. Obtain the raw image data and divide it into multiple image blocks;

[0009] Step 2. Initialize the relationship between known pixels and pixels to be interpolated in each image patch, and establish an initial adjacency matrix. And obtain the initial interpolated image;

[0010] Step 3. Construct the first perturbation matrix using the initial interpolated image. ,right Enhance the perturbation;

[0011] Step 4. Based on the new adjacency matrix after perturbation enhancement, establish a MAP optimization model based on GSV, and solve the optimization model through the expanded BICG iterative algorithm to obtain the interpolated image of the first stage;

[0012] Step 5. Introduce the second perturbation matrix. The interpolated image from the first stage is smoothed and refined.

[0013] Step 6. Based on and Establish a joint optimization model that includes a graph displacement variation term and a graph Laplace regularization term;

[0014] The joint optimization model is solved by splitting the DR and the DR iterative optimization process is transformed into an end-to-end trainable network by unfolding, which contains multiple iterative layers, each of which corresponds to one iteration of DR.

[0015] The following procedure is performed in each iteration until the iteration ends;

[0016] The input image is constructed using graph learning. and In the first iteration layer, the interpolated image from the first stage is used as input, while in other iteration layers, the output image from the previous iteration layer is used as input.

[0017] Subproblems are constructed for the graph displacement variation term and the graph Laplace regularization term, and the first and second intermediate variables are obtained by solving them using the expanded BICG iterative algorithm, respectively.

[0018] The pixel to be interpolated is updated using the iterative update formula combined with the first and second intermediate variables, and then stitched and fused with the original known pixel to obtain the output image of the current iterative layer;

[0019] Step 7. Stitch and merge all the optimized image patches to construct a complete interpolated image.

[0020] Furthermore, based on the aforementioned image interpolation method based on two-stage perturbation and DR split unrolling, this invention also proposes a computer device comprising a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the aforementioned image interpolation method based on two-stage perturbation and DR split unrolling.

[0021] Furthermore, based on the above-mentioned image interpolation method based on two-stage perturbation and DR split unrolling, the present invention also proposes a computer-readable storage medium storing a program thereon, which, when executed by a processor, is used to implement the steps of the above-mentioned image interpolation method based on two-stage perturbation and DR split unrolling.

[0022] The present invention has the following advantages:

[0023] As described above, this invention discloses an image interpolation method based on two-stage perturbation and DR splitting expansion. This method uses a linear interpolation operator as the network initialization module, mapping this linear interpolation operator to a graph adjacency matrix using the interpolation operator theorem. This combines the linear mapping capability of traditional interpolation methods with prior constraints on graph structure, achieving high interpolation accuracy and convergence stability in the early stages of network training. This structured initialization effectively avoids the convergence uncertainty problem caused by random parameter initialization, providing a stable performance lower bound and good interpretability for subsequent optimization stages, ensuring the model maintains robust basic reconstruction capabilities under different data scenarios. Furthermore, this invention introduces a two-stage perturbation mechanism based on the initial interpolation result (i.e., the initial interpolated image). The first stage uses a directed perturbation matrix to adaptively adjust the adjacency relationships between local nodes, enhancing the model's sensitivity to complex structural regions and detailed changes. The second stage uses an undirected perturbation matrix to smoothly optimize the global graph structure, thereby ensuring the overall consistency of the interpolation result and noise suppression effect. The two-level perturbation complements local fidelity and global coordination, significantly improving the accuracy and stability of interpolation results without significantly increasing computational cost. Furthermore, this invention expands the traditional DR split-iteration algorithm and the biconjugate gradient (BICG) solution process into an end-to-end trainable neural network structure. By setting the step size and regularization parameters of each layer as learnable variables, the model can adaptively optimize algorithm parameters during training, achieving a structured mapping from theoretical algorithms to learnable networks. This expanded design retains the interpretability and mathematical convergence of traditional optimization algorithms while possessing the strong representational power and adaptability of neural networks, enabling rapid convergence and high-quality interpolation results in complex scenarios. Simultaneously, the method of this invention has a small number of structural parameters and high computational efficiency, demonstrating good engineering deployment value and scalability potential. Attached Figure Description

[0024] Figure 1 This is a flowchart of the image interpolation method based on two-stage perturbation and DR splitting expansion in an embodiment of the present invention;

[0025] Figure 2 This is a detailed diagram illustrating the architecture of an image interpolation model based on DR splitting, which is expanded into a learnable network layer in a two-stage optimization process, as described in this embodiment of the invention. Figure 2 In the diagram, (a) shows the unfolded DR network structure, and (b) shows the details of the Kth layer of the DR network.

[0026] Figure 3 The results show the visual interpolation comparison between the method of this invention and the comparison method in two different sampling scenarios;

[0027] in Figure 3 (a) in the image is the input image. Figure 3In the diagram, (b), (c), (d), and (e) represent the interpolation results of traditional Bicubic interpolation, Restormer interpolation, uGTV interpolation, and the interpolation method of this invention, respectively. Figure 3 (f) in the image is the actual image. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment describes an image interpolation method based on graph algorithm unrolling, and in particular, an image interpolation method that combines graph signal processing theory, initializes a directed graph through a linear interpolator, and unfolds the optimization process based on Douglas–Rachford splitting into a learnable network layer in two stages.

[0031] This image interpolation method based on two-stage perturbation and DR splitting expansion includes the following steps:

[0032] Step 1. Obtain the original image data and divide it into multiple image blocks according to the preset sampling method. Each image block includes several known pixels and several pixels to be interpolated.

[0033] Before performing high-resolution reconstruction or interpolation on the original image, each image patch of the original image is first downsampled according to a sampling mask to simulate a pixel loss scenario. Sampling methods include uniform sampling and random sampling.

[0034] (1) Uniform sampling:

[0035] Remove 50% of the pixels from the original image according to the checkerboard pattern, and the remaining pixels form a regular grid.

[0036] (2) Random sampling:

[0037] 50% of the pixels are randomly removed from the original image grid, and the remaining pixels are distributed in an unstructured manner.

[0038] For each image patch, the known set of original pixels is denoted as . ; and the set of missing pixels to be interpolated, denoted as .in It contains several known pixels, and It contains several pixels to be interpolated.

[0039] Where M represents the number of known pixels and N represents the number of pixels to be interpolated.

[0040] In this embodiment, for the two sampling tasks mentioned above, the following settings can be configured: That is, the loss rate is 50%.

[0041] Step 2. Initialize the relationship between known pixels and pixels to be interpolated in each image block according to the preset linear interpolation operator, thereby establishing the initial adjacency matrix, i.e. the initial adjacency relationship, and obtaining the initial interpolated image.

[0042] This invention uses a linear interpolation operator as the initialization module of the network for initial interpolation. By using the interpolation operator theorem, the linear interpolation operator is mapped to a graph adjacency matrix. This combines the linear mapping capability of traditional interpolation methods with the prior constraints of graph structure, ensuring that the network has interpretable initialization baseline performance and ensuring the stability of the training process and the reliability of the interpolation results.

[0043] The initialization process is implemented based on the interpolation operator theorem. Traditional linear interpolation operators... Will Known pixels Mapped to One pixel to be interpolated ,Right now Therefore, the complete pixel signal It can be represented as:

[0044] .

[0045] in, express The identity matrix, which will signal Arrange and combine them according to the mask order to obtain the initial interpolated image.

[0046] According to the interpolation operator theorem, in and Under the premise of invertibility, this linear interpolation operator It is the solution to a maximum a posteriori probability (MAP) optimization problem with graph displacement variational GSV as the prior constraint.

[0047] The initialization of adjacency relations is the process of mapping the linear interpolation operator to the corresponding adjacency matrix using the interpolation operator theorem.

[0048] Initial adjacency matrix This corresponds to an initial directed graph structure used to describe the topological connections and adjacency relationships between known pixels and pixels to be interpolated in an image patch; this initial adjacency matrix It is defined as a sparse matrix, as shown in the following formula:

[0049] ;

[0050] in Represents the initial adjacency matrix; express × Adjacency matrix sub-blocks; , , Let M, N, and N represent the zero matrices, respectively.

[0051] Initial adjacency matrix sub-block Set as a linear interpolation operator inverse matrix ,Right now This ensures that the network has stable baseline performance. Linear interpolation operator. The preferred method is the improved bicubic interpolation operator Bicubic+.

[0052] Unlike traditional bicubic interpolation which uses a fixed analytical kernel, Bicubic+ utilizes a lightweight multilayer perceptron (MLP) network to adaptively predict w×w neighborhood interpolation weights based on the coordinate difference between the pixel to be interpolated and its neighboring pixels.

[0053] In this embodiment, w is preferably set to 4, that is, a 4×4 neighborhood.

[0054] Specifically, the MLP contains three hidden layers with dimensions of 32, 64, and 32 respectively; the input dimension is 2, representing the relative coordinate difference in the x and y directions; the output dimension is 1, representing the weight coefficient at the corresponding position. This weight coefficient is processed by the Sigmoid activation function to ensure its positive value, thereby generating accurate adaptive interpolation weights for the 16 pixels in the neighborhood.

[0055] This invention obtains the specific pixel value of the pixel to be interpolated by weighted summation of pixels within the neighborhood. By applying this linear interpolation operator to predict all pixels to be interpolated, an initial interpolated image can be obtained.

[0056] It should be noted that the parameters of the Bicubic+ operator are learned using a "pre-training-fixed" strategy.

[0057] Specifically, before building the MLP network, the MLP network is first trained independently on the training dataset to achieve optimal initial interpolation performance.

[0058] When training the graph algorithm unfolded network, the pre-trained Bicubic+ model was loaded as a fixed known operator, and its parameters remained unchanged. It did not participate in the backpropagation update during the subsequent two-stage optimization, that is, the parameters were fixed after pre-training.

[0059] In this embodiment, the "pre-training-fixed" strategy adopted for the parameters of the Bicubic+ model ensures that the optimization process begins with a deterministic, high-quality benchmark, avoiding the local minima problem caused by initialization randomness.

[0060] To ensure the rigor of subsequent mathematical mappings and the invertibility of operators, the method of this invention limits the number of known pixels to be equal to the number of pixels to be interpolated, i.e. This allows the initial adjacency graph to serve as a complete structured benchmark for the interpolation task.

[0061] However, the method of the present invention is also applicable. General interpolation scenarios.

[0062] According to a corollary of the interpolation operator theorem, for In such cases, the following adaptive measures can be taken:

[0063] when When the number of pixels to be interpolated is less than the number of known pixels, a new pixel is introduced into the set of pixels to be interpolated. The virtual pixels are used to complete the interpolation, so that the total number of pixels to be interpolated is... After completing the interpolation calculation, the data corresponding to these virtual pixels can be discarded.

[0064] when When the number of pixels to be interpolated is greater than the number of known pixels, construct... Each interpolator applies the method of the present invention, or the region to be interpolated is divided into multiple sub-regions, such that the total number of interpolations in each sub-region is [number missing]. Processed in batches.

[0065] Through the above adaptation, the method of the present invention can be flexibly applied to image reconstruction tasks of various resolution ratios.

[0066] Step 3. In the initial adjacency matrix Based on this, the first perturbation matrix is ​​introduced. First perturbation matrix This corresponds to a directed graph used to describe the connection relationships and similarity measures between the known pixels and the pixels to be interpolated.

[0067] To optimize the initial adjacency matrix using a data-driven approach Introducing the first perturbation matrix , Corresponding to a new directed graph It mainly connects the pixels to be interpolated with known pixels, and for the initial... Enhance the perturbation.

[0068] By enhancing A, a new adjacency matrix is ​​obtained. This allows the network to adjust the initial interpolation error bidirectionally based on local image features (i.e., it can both enhance and weaken the connection strength between pixels), thus correcting the initial interpolation error more flexibly.

[0069] Initial linear interpolation operator The baseline interpolation results are provided, while the first perturbation matrix... This is equivalent to applying a new interpolation operator on top of the baseline result. That is, the final equivalent interpolation operator .

[0070] This design allows the network to be optimized in two steps: first, a stable baseline is obtained using fixed operators, and then high-frequency detail errors are adaptively corrected using learnable perturbation terms.

[0071] First perturbation matrix Having an adjacency matrix with the initial adjacency matrix With a similar structure, the formula is expressed as follows:

[0072] ;

[0073] in For learnable perturbation submatrices, , , Let M, N, and N represent the zero matrices, respectively. First perturbation matrix. The corresponding directed graph Edge weights It contains symbols.

[0074] The purpose of designing signed weights is to allow the network to have a signed weight in the initial adjacency matrix. Based on this, the weights of the edges are adjusted bidirectionally (that is, the connection strength can be strengthened or weakened), thereby more flexibly correcting the error of the initial interpolation.

[0075] Edge weight in this embodiment It is based on pixel features extracted using a shallow convolutional neural network (CNN) from the initial interpolated image. and positive definite metric matrix Calculated feature distance Yes, that's confirmed. A shallow CNN consists of an input convolutional layer, a residual block containing two convolutional layers, and an output convolutional layer. Each layer uses a 3×3 convolutional kernel and outputs 48 feature channels. Shallow CNN networks achieve deep feature integration through the ReLU activation function and skip connections, mapping each pixel of the image to a high-dimensional feature vector. This lightweight design effectively controls the number of model parameters and computational complexity while ensuring feature extraction capabilities.

[0076] Feature distance The calculation formula is as follows:

[0077] .

[0078] Where index i represents the pixel to be interpolated, and index j represents a known pixel; It is a learnable positive definite metric matrix, M is initialized as an identity matrix, and its diagonal parameters are automatically updated by the network during training.

[0079] This represents the high-dimensional feature vector corresponding to the pixel to be interpolated. This represents the high-dimensional feature vector corresponding to a known pixel.

[0080] Signed edge weights The calculation formula is as follows:

[0081] ;in It is a preset offset used to define the boundary between similarity and dissimilarity, allowing the model to flexibly strengthen or weaken the connection strength between different pixels.

[0082] This edge weight The calculation formula is to use the feature distance Mapped to Within a given range, the greater the distance and the lower the similarity, the lower the edge weight. (Directed graph) Therefore, it is a similarity graph.

[0083] Step 4. Based on the new adjacency matrix after perturbation enhancement, establish a maximum a posteriori probability (MAP) optimization model with graph displacement variational GSV as the smoothing constraint prior, and solve the optimization model to obtain the interpolated image of the first stage.

[0084] Graph displacement variational GSV It is a measure of signal smoothness, defined as:

[0085] .

[0086] Where parameters This represents the enhanced new adjacency matrix. It represents the complete image pixel signal.

[0087] The first stage of the interpolation optimization process, which optimizes the maximum a posteriori probability (MAP objective function, a combination of data fidelity and graph smoothing priors), uses graph displacement variation (GSV) as the smoothing constraint prior. Its objective function is:

[0088] ;

[0089] in This represents the known pixel data of the original image. It is the enhanced adjacency matrix; It is a regularization parameter. It is a sampling matrix, from the complete signal Select before A known pixel signal.

[0090] in, express The identity matrix; express The zero matrix.

[0091] The first term represents the data fidelity between the interpolation result and the original known pixels. The second term represents the graph structure smoothing constraint term, which is based on the directed graph. Structural constraints are used to ensure signal smoothness.

[0092] The objective function of the first stage ensures that the interpolation results have global smoothness while maintaining observational consistency.

[0093] The model uses graphical displacement variation. As a measure of signal smoothness, minimizing the optimization objective function ensures that the interpolation results maintain observational consistency while conforming to the prior of the enhanced directed graph structure.

[0094] The process of solving the objective function involves transforming the convex optimization problem into a linear system and solving iteratively.

[0095] By analyzing the objective function with respect to Differentiate and set it to zero, then the pixel to be interpolated. The solution is simplified to the following linear system:

[0096] .

[0097] Furthermore, based on the aforementioned definition The above equation can be equivalently transformed into:

[0098] .

[0099] in express The identity matrix is ​​obtained, thus transforming the problem into a linear system problem.

[0100] Due to the coefficient matrix Since the coefficients are usually asymmetric, the biconjugate gradient (BICG) iterative algorithm is used to solve the linear system with the asymmetric coefficient matrix above in order to avoid matrix inversion operations.

[0101] To achieve adaptive optimization of algorithm parameters using a data-driven approach, the iterative process of the BICG algorithm is unfolded into a neural network layer, realizing the first stage of predicting the pixel value to be interpolated. Efficient computation and parameterized learning.

[0102] Then The image is merged with the known pixel data y to form a complete image, which is the first-stage interpolated image.

[0103] Step 5. Introduce the second perturbation matrix based on the interpolation results of the first stage. , Corresponding to an undirected graph, it is used to describe the connection relationship and smoothness constraints between the pixels to be interpolated, and to smooth and refine the interpolation results of the first stage.

[0104] Second perturbation matrix In mathematics, it is defined as a symmetric and positive semi-definite graph Laplace matrix, corresponding to an undirected positive graph, i.e., an undirected graph. Only connect One pixel to be interpolated.

[0105] Undirected graph Positive edge weights Features extracted based on shallow CNN and positive definite metric matrix Calculated feature distance Yes, that's correct. This shallow CNN maps each pixel of the image to a high-dimensional feature vector. .

[0106] A shallow CNN consists of an input convolutional layer, a residual block containing two convolutional layers, and an output convolutional layer. Each layer uses 3x3 convolutions. The network uses 3 convolutional kernels and outputs 48 feature channels. It achieves deep feature integration through ReLU activation and skip connections.

[0107] The formula for calculating feature distance is: Where indices i and j both represent pixels to be interpolated. It is a learnable PSD metric matrix, initialized as an identity matrix, and its parameters are automatically updated through data-driven processes.

[0108] Positive edge weights The calculation formula is: .

[0109] in It meets the requirements of Laplace regularization for undirected graphs, ensuring As a pure smoothing operator (i.e., guaranteeing the matrix) (Semi-positive definite), focusing on eliminating noise and artifacts after interpolation without destroying the geometry of the image.

[0110] Step 6. Based on the first perturbation matrix With the second perturbation matrix Establish a joint optimization model that includes a graph displacement variation term and a graph Laplace regularization term. Joint Optimization Model Defined as:

[0111] .

[0112] in It is a selection matrix used to extract the pixels to be interpolated; Represents an N×M zero matrix; This represents the identity matrix. The first two terms of this joint optimization model are the MAP optimization objective function in step 4, including a graph displacement variation term, used to address the issue caused by the first perturbation matrix. Smoothness of constraint signals on the enhanced directed graph structure.

[0113] The third term of this joint optimization model is the graphical Laplace term, corresponding to the term derived from the second perturbation matrix. The constructed undirected graph smoothing prior is used for denoising and detail refinement of the pixels to be interpolated.

[0114] The joint optimization model is solved by splitting the DR (Derivative Radical) algorithm. The DR iterative optimization process is then transformed into an end-to-end trainable network by unfolding, which contains K iterative layers, where K is the preset total number of iteration layers. Each iterative layer corresponds to one iteration of DR. The following process is executed in each iteration until the iteration ends.

[0115] The input image is constructed using graph learning. and In the first iteration layer, the interpolated image from the first stage is used as input, while in other iteration layers, the output image from the previous iteration layer is used as input.

[0116] Subproblems are constructed for the graph displacement variation term and the graph Laplacian regularization term, and the first and second intermediate variables are obtained by solving them using the expanded BICG iterative algorithm. The pixels to be interpolated are updated using the iterative update formula in combination with the first and second intermediate variables, and then stitched and fused with the original known pixels to obtain the output image of the current iterative layer.

[0117] The specific process of step 6 is explained in further detail below.

[0118] The input image is used to construct the first perturbation matrix through graph learning. Second perturbation matrix In the first iteration layer, the interpolated image from the first stage is used as input, while in other iteration layers, the output image from the previous iteration layer is used as input.

[0119] in This corresponds to a directed graph that describes the connection relationship between known pixels and pixels to be interpolated. , Corresponds to the undirected graph describing the smoothness constraints between the pixels to be interpolated. Graph learning is the process of calculating and constructing a first perturbation matrix and a second perturbation matrix from an input image, i.e., the first perturbation matrix... Second perturbation matrix The process of constructing the perturbation matrix.

[0120] Will Decomposed into terms including graphical displacement variation With the inclusion of graph Laplacian regularization terms Two parts:

[0121] ;

[0122] .

[0123] right and By taking the derivatives and setting them to zero, we can obtain two linear systems corresponding to the proximal mapping, namely the h-step linear system and the g-step linear system.

[0124] Taking the k-th iteration layer as an example, where Input image for layer k The prediction part of the pixels to be interpolated.

[0125] The h-step linear system is: .

[0126] in It is the identity matrix; This is the first perturbation submatrix of the k-th layer; As the first intermediate variable; Given pixel data; γ is the step size parameter; This is the predicted portion of the pixels to be interpolated corresponding to the (k-1)th layer.

[0127] Specifically, when k=1, Represents the predicted value of the pixel to be interpolated in the first stage of the interpolated image. .

[0128] This linear system effectively integrates observation consistency constraints with directed graph structure priors.

[0129] To avoid complex matrix inversion operations and to achieve adaptive optimization of algorithm parameters using a data-driven approach, this invention employs the BICG iterative algorithm to solve this linear system.

[0130] The g-step linear system is: .

[0131] in For regularization parameters; This is the second perturbation matrix of the k-th layer; This serves as the second intermediate variable. By introducing a smooth prior from an undirected graph, this linear system achieves denoising and detail refinement of the pixels to be interpolated.

[0132] Similarly, to avoid the high complexity of matrix inversion operations and to utilize data-driven adaptive parameter learning, this invention expands the solution process of this subproblem into a BICG iterative algorithm in the form of a neural network.

[0133] The DR split-iterative algorithm is used to alternately call the two linear systems (h-step linear system and g-step linear system) to solve for the intermediate variables z and v, and then iteratively solves the problem by combining the variable update formula.

[0134] After solving the h-step and g-step problems, the iterative solution is updated using the DR iterative update formula. Perform iterative updates:

[0135] .

[0136] in, The updated predicted pixel value to be interpolated will then be... By concatenating the image with the known pixel y using a sampling mask, the complete predicted image of the (k+1)th layer is obtained. , as input to the next iteration layer.

[0137] This invention expands each iteration of the DR splitting iterative algorithm into a neural network iterative layer, for a total of K layers. The perturbation matrix of each layer is... , The graph learning module dynamically generates the input image of the current layer and does not share it between layers; the step size parameter γ, the regularization parameter μ, and the internal parameters of BICG are all learned variables that are adaptively updated by the neural network.

[0138] Specifically, the input image X of the first iteration layer (k=1) 1 The first-stage interpolated image obtained in step 4 (the portion of pixels to be interpolated is...) = ); the input image X for subsequent layers (k=2,...,K) k The output image of the previous layer ,Right now .in for The part of the interpolation prediction of the pixel in the image, where y is... The known pixel portion.

[0139] By alternately executing the above K-layer h-step and g-step iterative solution steps, observational information and prior knowledge of the graph structure are continuously integrated, thereby achieving layer-by-layer refinement of the interpolation results. The final output of the K-th layer... This means converging the high-precision interpolation prediction value of the pixel to be interpolated, and then concatenating it with the known pixel y according to the original mask position to obtain the complete predicted image patch. .

[0140] Step 7. Take the multiple interpolated and optimized image patches obtained by step 6, expand and iteratively solve them, and stitch and fuse them according to their original spatial positions to construct a complete interpolated image and output it.

[0141] After interpolation optimization of all image blocks, these discrete, independently processed image blocks need to be recombined into a complete, high-resolution image. To eliminate potential discontinuities at block boundaries (i.e., block artifacts caused by block processing), an overlapping block strategy is adopted. Specifically, when dividing the image blocks in step 1, a 16-pixel overlap region is set between adjacent blocks. In step 7, during stitching, the pixel prediction values ​​of the overlap region are averaged, thereby ensuring that the final output complete interpolated image is visually smooth and natural with fluid detail transitions. Ultimately, the method of this invention outputs a complete, high-precision interpolated image.

[0142] In addition, the following experiments were conducted to verify the effectiveness of the method proposed in this invention.

[0143] In the experiments, the selected datasets included the Kodak dataset, the Urban100 dataset, and the McM dataset. Representative image interpolation methods selected included Bicubic, RFDN, RLFN, SAFMN, SwinIR-Light, Restormer, and uGTV.

[0144] Among them, Bicubic is a traditional bicubic interpolation method, a traditional method based on the weighted average of 4×4 neighborhood pixels; RFDN is a residual feature distillation network, a lightweight image super-resolution method from ECCV 2020; RLFN is a residual local feature network, an efficient super-resolution method from CVPR 2022; SAFMN is a spatial adaptive feature modulation network, a spatial adaptive image super-resolution method from ICCV 2023; SwinIR-Light is a lightweight Swin Transformer image restoration network, a Transformer-based image restoration method from ICCV 2021; Restormer is an efficient high-resolution image restoration Transformer, a high-performance image restoration method from CVPR 2022; uGTV is a lightweight network based on unfolded graph total variational prior, an interpretable graph algorithm unfolded interpolation method from NeurIPS 2024.

[0145] Table 1 shows the performance comparison of the method of this invention with other representative image interpolation methods on different datasets. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation metrics in the comparison. The upper and lower halves of Table 1 show the comparison results under uniform sampling and random sampling tasks, respectively.

[0146] Table 1. Performance comparison of the present invention with other representative image interpolation methods on different datasets.

[0147]

[0148] As can be seen from the comparison results in Table 1, the method of the present invention (Ours (Bicubic+)) achieves competitive performance in both uniform sampling and the more challenging random sampling tasks.

[0149] Of particular note is that this invention uses only 164,092 parameters, approximately 0.6% of the Restormer model (26M), yet achieves comparable or even better repair accuracy on the Kodak and McM datasets. This highlights the efficiency of initializing the unfolded network as a known interpolation operator (Bicubic+) and combining it with a dual perturbation matrix optimization strategy.

[0150] In addition, to verify the importance of each part of the method of the present invention, including the initialization operator, the weight type of the perturbation matrix P, and the two-stage cascade structure, the following ablation experiments were also conducted.

[0151] Table 2 shows the quantification results of the present invention under different configurations, using PSNR as the indicator.

[0152] In Table 2, the baseline is the traditional Bicubic. Variant 1 changes the interpolation operator in the baseline to Bicubic+. Variant 2 introduces a first perturbation matrix with positive edge weights based on Variant 1. Variant 3 is the first perturbation matrix of Variant 2. The positive edge weights are changed to signed edge weights.

[0153] Table 2. Quantification results of the present invention under different configurations

[0154]

[0155] Analysis of the ablation experiment results shows that:

[0156] This invention uses the Bicubic+ operator for initial interpolation (i.e., variant one), which brings a 3.46dB improvement in PSNR compared to the traditional Bicubic (baseline), thus verifying the benefits of the adaptive initializer of this invention.

[0157] Furthermore, this invention introduces a first perturbation matrix with positive edge weights. (Variant 2) delivers a 4.51 dB improvement in PSNR compared to the baseline, demonstrating that graph-based optimization refinement is effective.

[0158] Furthermore, this invention employs a perturbation matrix with signed edge weights. (i.e., variant 3), further improving performance to 5.74dB, validating the importance of signed edges for capturing more subtle relationships between pixels.

[0159] Finally, the present invention incorporates a second perturbation matrix for refinement processing. The introduction of a joint perturbation method (i.e. the method of this invention) brings additional gains, with a total improvement of 6.37 dB, thus highlighting the advantages of the two-stage cascaded design.

[0160] Figure 3 The results of visual interpolation of the present invention are shown in comparison under two different sampling scenarios: uniform sampling and random sampling.

[0161] in, Figure 3 The first row shows the interpolation results of the image "Urban96" in the Urban100 dataset under uniform sampling; Figure 3 The second row shows the interpolation results of the image "Urban62" in the Urban100 dataset under random sampling.

[0162] Figure 3 (a) in the image is the input image. Figure 3In the diagram, (b), (c), (d), and (e) represent the interpolation results of traditional Bicubic interpolation, Restormer interpolation, uGTV interpolation, and the interpolation method of this invention, respectively. Figure 3 (f) in the image is the actual image.

[0163] As observed, traditional Bicubic interpolation exhibits severe blurring and loss of detail in both scenarios. While large-scale deep learning models such as Restormer restore better clarity, their computational cost is prohibitive.

[0164] While the uGTV image signal processing method preserves the image structure to some extent, it is still insufficient when processing high-frequency details such as building lines (Urban96) or dense grid textures (Urban62).

[0165] In contrast, the method of our invention (Ours Bicubic+) can not only accurately reconstruct the texture structure of buildings under uniform sampling, but also effectively recover the regular window frame texture of buildings and maintain sharp structural edges under random sampling. Its visual quality is superior to similar lightweight methods (such as uGTV) and comparable to the current state-of-the-art large models (such as Restormer).

[0166] The above further verifies the robustness and effectiveness of the dual-perturbation graph algorithm expansion framework of this invention under different tasks.

[0167] Example 2

[0168] This embodiment 2 describes an image interpolation system based on two-stage perturbation and DR split expansion. This image interpolation system based on two-stage perturbation and DR split expansion is based on the same inventive concept as the image interpolation method based on two-stage perturbation and DR split expansion in the above embodiment 1.

[0169] The image interpolation system based on two-stage perturbation and DR splitting expansion in this embodiment includes the following modules:

[0170] The image block segmentation module is used to acquire the raw image data and divide it into multiple image blocks;

[0171] The initial adjacency matrix and initial interpolation image acquisition module is used to initialize the relationship between known pixels and pixels to be interpolated in each image block, and to establish the initial adjacency matrix. And obtain the initial interpolated image;

[0172] The first perturbation enhancement module is used to construct a first perturbation matrix using the initial interpolated image. ,right Enhance the perturbation;

[0173] The first-stage interpolation image acquisition module is used to establish a GSV-based MAP optimization model based on the new adjacency matrix after perturbation enhancement, and solve the optimization model through the expanded BICG iterative algorithm to obtain the first-stage interpolation image.

[0174] The second perturbation enhancement module is used to introduce the second perturbation matrix. Smooth and refine the interpolated image from the first stage;

[0175] And a network unfolding and interpolation processing module, used for processing based on the first perturbation matrix. With the second perturbation matrix Establish a joint optimization model that includes a graph displacement variation term and a graph Laplace regularization term;

[0176] The joint optimization model is solved by splitting the DR and the DR iterative optimization process is transformed into an end-to-end trainable network by unfolding, which contains multiple iterative layers, each of which corresponds to one iteration of DR.

[0177] The following procedure is performed in each iteration until the iteration ends;

[0178] The input image is constructed using graph learning. and In the first iteration layer, the interpolated image from the first stage is used as input, while in other iteration layers, the output image from the previous iteration layer is used as input.

[0179] Subproblems are constructed for the graph displacement variation term and the graph Laplace regularization term, and the first and second intermediate variables are obtained by solving them using the expanded BICG iterative algorithm, respectively.

[0180] The pixel to be interpolated is updated using the iterative update formula combined with the first and second intermediate variables, and then stitched and fused with the original known pixel to obtain the output image of the current iterative layer;

[0181] The stitching and fusion module is used to stitch and fuse all optimized image patches to construct a complete interpolated image.

[0182] It should be noted that any content not mentioned in the above-described functional modules of the system described in this embodiment can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0183] Example 3

[0184] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the image interpolation method based on two-stage perturbation and DR splitting expansion described in embodiment 1 above.

[0185] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0186] Example 4

[0187] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, is used to implement the steps of the image interpolation method based on two-stage perturbation and DR splitting expansion in embodiment 1 above.

[0188] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0189] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. An image interpolation method based on two-stage perturbation and DR splitting expansion, characterized in that, Includes the following steps: Step 1. Obtain the raw image data and divide it into multiple image blocks; Step 2. Initialize the relationship between known pixels and pixels to be interpolated in each image patch, and establish an initial adjacency matrix. And obtain the initial interpolated image; Step 3. Construct the first perturbation matrix using the initial interpolated image. ,right Enhance the perturbation; Step 4. Based on the new adjacency matrix after perturbation enhancement, establish a MAP optimization model based on GSV, and solve the optimization model through the expanded BICG iterative algorithm to obtain the interpolated image of the first stage; Step 5. Introduce the second perturbation matrix. The interpolated image from the first stage is smoothed and refined. Step 6. Based on and Establish a joint optimization model that includes a graph displacement variation term and a graph Laplace regularization term; The joint optimization model is solved by splitting the DR and the DR iterative optimization process is transformed into an end-to-end trainable network by unfolding, which contains multiple iterative layers, each of which corresponds to one iteration of DR. The following procedure is performed in each iteration until the iteration ends; The input image is constructed using graph learning. and In the first iteration layer, the interpolated image from the first stage is used as input, while in other iteration layers, the output image from the previous iteration layer is used as input. Subproblems are constructed for the graph displacement variation term and the graph Laplace regularization term, and the first and second intermediate variables are obtained by solving them using the expanded BICG iterative algorithm, respectively. The pixel to be interpolated is updated using the iterative update formula combined with the first and second intermediate variables, and then stitched and fused with the original known pixel to obtain the output image of the current iterative layer; Step 7. Stitch and merge all the optimized image patches to construct a complete interpolated image.

2. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 1, characterized in that, The first perturbation matrix This corresponds to a directed graph used to describe the connectivity and similarity measure between known pixels and pixels to be interpolated. This corresponds to an undirected graph, used to describe the connectivity and smoothness constraints between pixels to be interpolated.

3. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 1, characterized in that, In step 2, the relationship between known pixels and pixels to be interpolated in each image block is initialized according to a preset linear interpolation operator to establish an initial adjacency matrix. And obtain the initial interpolated image, the process is as follows: The initial adjacency matrix A is defined as a sparse block matrix, and its mathematical form is as follows: ； in It is × The initial adjacency matrix sub-blocks are used to describe the adjacency matrix sub-blocks from the initial adjac The pixel to be interpolated is connected back The directed edge relationship of a known pixel; , , They are respectively × , × , × The zero matrix; and These represent the number of known pixels and the number of pixels to be interpolated in each image patch, respectively. ; Given a linear interpolation operator Establish a mapping relationship between the known pixel set and the pixels to be interpolated; while ensuring the linear interpolation operator... Under the premise of reversibility, Set as The inverse matrix, i.e. .

4. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 3, characterized in that, In step 2, the linear interpolation operator For the improved bicubic interpolation operator Bicubic+; Bicubic+ utilizes a lightweight multilayer perceptron (MLP) network, which adaptively predicts 4×4 neighborhood interpolation weights based on the coordinate differences between the pixel to be interpolated and its neighboring pixels. The MLP contains three hidden layers with dimensions of 32, 64, and 32 respectively; the input dimension is 2, representing the relative coordinate difference in the x and y directions; the output dimension is 1, representing the weight coefficient at the corresponding position. The weight coefficients are processed by the Sigmoid activation function to ensure their positive values, thereby generating accurate adaptive interpolation weights for the 16 pixels in the neighborhood, which improves the accuracy of image edge and texture detail restoration during the initialization phase.

5. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 2, characterized in that, The first perturbation matrix The corresponding directed graph Edge weights It is signed; the first perturbation matrix. Mathematically, it is defined as a sparse block matrix, and its structure is represented as: ； It is a learnable perturbation submatrix used to describe the perturbation from The pixel to be interpolated is connected back Directed connectivity relationships and similarity metrics for known pixels. , , The zero matrices are M×M, N×M, and N×N respectively; in and These represent the number of known pixels and the number of pixels to be interpolated in each image block, respectively. By introducing this block matrix, the initial adjacency matrix Enhanced into a new adjacency matrix ; Learnable perturbation submatrix The specific value is calculated based on the pixel features extracted from the initial interpolated image using a shallow CNN, which maps each pixel of the image to a high-dimensional feature vector. ; For a pair of pixels, their feature distance Through formula Calculated; Where index i represents the pixel to be interpolated, and index j represents a known pixel. It is a learnable positive definite metric matrix; This represents the high-dimensional feature vector corresponding to the pixel to be interpolated. This represents the high-dimensional feature vector corresponding to a known pixel; Finally, the first perturbation matrix Signed edge weights Through formula Sure; in It is a preset offset that serves as the boundary point for defining similarity and dissimilarity.

6. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 1, characterized in that, In step 4, the objective function of the MAP optimization model using GSV as the smoothness constraint prior is defined as: ； in The known pixel data of the original image; It is the enhanced new adjacency matrix; It is a regularization parameter; It is the identity matrix; Indicates a complete signal; It is a sampling matrix used to extract data from the complete signal. Select before One known pixel signal; express The identity matrix; express The zero matrix; By analyzing the above objective function with respect to Differentiate and set it to zero, then the pixel to be interpolated. The solution is simplified to the following linear system: ； in It is × The initial adjacency matrix sub-blocks, It is a learnable perturbation submatrix; utilizing With linear interpolation operators Inverse matrix mapping relationship The equivalent transformation of this linear system is: ;in express The identity matrix; For the above linear system, the biconjugate gradient (BICG) iterative algorithm is used for solution. The iterative process of BICG is expanded into a neural network layer to achieve the first-stage prediction of the pixel to be interpolated. Computation and parameterization learning; Then The image is merged with the known pixel data y to form a complete image, i.e., the first-stage interpolated image.

7. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 2, characterized in that, The second perturbation matrix The corresponding undirected graph Used to describe the connection relationship and smoothness constraints between pixels to be interpolated, aiming to further denoise and refine the interpolation results of the first stage; Defined as a symmetric and positive semi-definite graph Laplace matrix, whose structure consists of only connecting... One pixel to be interpolated; The perturbation matrix The specific values ​​are adaptively generated based on the pixel features extracted by a shallow CNN from the first-stage interpolation results. This shallow CNN maps each pixel of the image to a high-dimensional feature vector. ; For a given set of pixels to be interpolated, their feature distances Through formula calculate; Where indices i and j both represent pixels to be interpolated. It is a learnable positive definite metric matrix; This represents the high-dimensional feature vector corresponding to the pixel i to be interpolated. This represents the high-dimensional feature vector corresponding to the pixel j to be interpolated; Ultimately, undirected graph Positive edge weights Through formula Sure.

8. The image interpolation method based on two-stage perturbation and DR splitting expansion according to claim 6, characterized in that, The joint optimization model, including the graph displacement variation term and the graph Laplace regularization term, is defined as follows: ;in It is a selection matrix used to extract the pixels to be interpolated; Represents an N×M zero matrix; express The identity matrix; The joint optimization model The first two terms are the MAP optimization model in step 4, which includes the graph displacement variation term; The third term is the Graph Laplace term, corresponding to the term derived from the second perturbation matrix. The resulting undirected graph has a smooth prior. Solve using the expanded DR splitting iterative algorithm. The process is as follows: First of all Decomposed into the graph displacement variation term With Thulaplac regularization term : ； ； right and By taking the derivatives and setting them to zero, we obtain two linear systems corresponding to the proximal mapping, with each iteration layer consisting of an h-step linear system and a g-step linear system; The h-step linear system is expressed as follows: ； The g-step linear system is expressed as follows: ； Where the superscript k is the number of the kth iteration layer; As the first intermediate variable; This is the step size parameter; and These are the interpolation results for the current iteration layer and the previous iteration layer, respectively. For regularization parameters; This is the second perturbation matrix; It is the second intermediate variable; The DR split-iteration algorithm solves for the first and second intermediate variables by alternately calling two linear systems, namely the h-step linear system and the g-step linear system, and then iteratively solving them by combining the variable update formula. After solving the h-step and g-step problems, the iterative solution of the k-th layer is updated using the DR iterative update formula. Perform iterative updates: ； in, This is the updated solution, i.e., the predicted pixel to be interpolated.

9. A computer device, comprising a memory and one or more processors; characterized in that, The memory stores executable code, which, when executed by the processor, is used to implement the image interpolation method based on two-stage perturbation and DR splitting as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the image interpolation method based on two-stage perturbation and DR splitting expansion as described in any one of claims 1 to 8.

Images