An image reconstruction method and device based on accelerated optimization and attention mechanism

Abstract

The specification discloses an image reconstruction method and device based on accelerated optimization and attention mechanism. The iterative process in a low-rank matrix completion target function constructed according to a traditional image reconstruction algorithm is optimized by applying a deep unfolding network, and an auxiliary acceleration variable is introduced in the optimization process. In each round of iterative process, the output image of the round of iterative process is determined according to the output image of the last round of iterative process and the auxiliary acceleration variable, the auxiliary variable of the round of iterative process is determined according to the output image of the round of iterative process and the auxiliary acceleration variable of the last round of iterative process, and the auxiliary acceleration variable of the round of iterative process is determined according to the auxiliary variable of the round of iterative process and the auxiliary variable of the last round of iterative process. In this way, there is no direct correlation between the auxiliary acceleration variables of adjacent two rounds, which speeds up the iteration, so that the method further improves the efficiency of image reconstruction while ensuring the accuracy of image reconstruction.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an image reconstruction method and apparatus based on acceleration optimization and attention mechanisms. Background Technology

[0002] With societal progress and the development of science and technology, human society generates a vast number of images every day. However, during the generation and transmission of images, degradation phenomena such as decreased image sharpness, low contrast, and noise are inevitable. These degradation phenomena significantly impact subsequent image analysis, necessitating image reconstruction to optimize image quality.

[0003] In existing technologies, there are image reconstruction methods based on prior constraints when dealing with the problem of degraded image reconstruction. These methods first use prior knowledge to construct prior constraints and then use these constraints to reconstruct the image to be reconstructed. However, this method requires a large amount of prior information as training data to achieve good image reconstruction results. The discovery of a high-speed, high-quality image processing algorithm is an urgent problem to be solved.

[0004] This specification provides an optimized algorithm based on deep unfolded networks to solve the problem of image restoration and reconstruction, which can achieve high performance with relatively low training cost. Summary of the Invention

[0005] This specification provides an image reconstruction method and apparatus based on acceleration optimization and attention mechanisms to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides an image reconstruction method based on acceleration optimization and attention mechanisms, including:

[0008] Obtain the image to be reconstructed, and construct a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed.

[0009] The objective function for low-rank matrix completion is iteratively optimized. For each iteration of the objective function for low-rank matrix completion, the auxiliary variables required to solve the objective function for low-rank matrix completion are determined based on the alternating direction multiplier method, and the auxiliary acceleration variables corresponding to the auxiliary variables are determined.

[0010] Based on the deep unfolded network, the objective function is completed by the low-rank matrix to construct an image reconstruction model;

[0011] The input image of this iteration process and the auxiliary acceleration variables of the previous iteration process are input into the image reconstruction model. Feature extraction is performed on the input image to obtain the first feature, and feature extraction is performed on the auxiliary acceleration variables to obtain the second feature.

[0012] The first feature and the second feature are superimposed to obtain a fused feature, and the fused feature is then attention-weighted to obtain the output image of this iteration process. Based on the output image of this iteration process and the auxiliary acceleration variable of the previous iteration process, the auxiliary variable of this iteration process is determined. Based on the auxiliary variable determined in this iteration process and the previous iteration process, the auxiliary acceleration variable of this iteration process is determined for use in the next iteration process.

[0013] The output image obtained from the first iteration and the auxiliary acceleration variables determined for the first iteration are used as the input for the next iteration of the image reconstruction model. The iteration continues until the specified number of iterations is reached, and the output image obtained from the last iteration is used as the reconstructed image.

[0014] Optionally, based on the image to be reconstructed, a low-rank matrix completion objective function to be optimized is constructed, specifically including:

[0015] The image to be reconstructed is reconstructed to determine the mask corresponding to the image to be reconstructed. The mask identifies the part of the image to be reconstructed that needs to be reconstructed.

[0016] Based on the preset constraints of the reconstructed image, determine the prior regularization term;

[0017] Based on the prior regularization term, a low-rank matrix completion objective function is constructed with the goal of minimizing the adjustment of the parts of the image to be reconstructed that require image reconstruction.

[0018] Optionally, based on the alternating direction multiplier method, auxiliary variables required to solve the low-rank matrix completion objective function are determined, and auxiliary acceleration variables corresponding to the auxiliary variables are determined, specifically including:

[0019] Based on the alternating direction multiplier method, the augmented Lagrange function corresponding to the low-rank matrix completion objective function is constructed according to the low-rank matrix completion objective function;

[0020] Based on the augmented Lagrangian function, determine the auxiliary variables for solving the low-rank matrix completion model;

[0021] Based on the auxiliary variable, the corresponding auxiliary acceleration variable is initialized.

[0022] Based on the update rates of the auxiliary variables and the Lagrange multipliers, the corresponding auxiliary acceleration variables are determined.

[0023] Optionally, the auxiliary variables include a first auxiliary variable and a second auxiliary variable, and the functional expressions of the first auxiliary variable and the second auxiliary variable are derived from the augmented Lagrange function.

[0024] Determine the auxiliary variables required to solve the objective function of the low-rank matrix completion, and determine the corresponding auxiliary acceleration variables, specifically including:

[0025] Based on the output image of this iteration process and the second auxiliary variable of the previous iteration process, determine the first auxiliary variable of this iteration process;

[0026] Based on the output image of this iteration process, the second auxiliary variable of the previous iteration process, and the first auxiliary variable of this iteration process, determine the second auxiliary variable of this iteration process;

[0027] Based on the first auxiliary variables of the current iteration process and the previous iteration process, determine the first auxiliary acceleration variable of the current iteration process;

[0028] Based on the second auxiliary variable of the current iteration process and the previous iteration process, determine the second auxiliary acceleration variable of the current iteration process.

[0029] Optionally, the image reconstruction model includes at least: a feature extraction subnetwork, an attention weighting subnetwork, and a denoising subnetwork;

[0030] The input image of this iteration and the auxiliary acceleration variables of the previous iteration are input into the image reconstruction model. Feature extraction is performed on the input image to obtain the first feature, and feature extraction is performed on the auxiliary acceleration variables to obtain the second feature, specifically including:

[0031] The input image of this iteration process is input into the image reconstruction model, and the first feature is obtained by extracting features from the input image through the feature extraction sub-network.

[0032] The first and second auxiliary acceleration variables of the iteration process are summed to obtain a composite variable. The feature extraction subnet is then used to extract features from the composite variable to obtain a second feature.

[0033] Optionally, attention weighting is applied to the fused features to obtain the output image of this iteration process, specifically including:

[0034] The fused features are then weighted by the attention-weighted subnet to obtain the output image for this iteration process.

[0035] Optionally, based on the output image of this iteration and the auxiliary acceleration variables of the previous iteration, auxiliary variables for this iteration are determined, specifically including:

[0036] The output image obtained in this iteration process is summed with the second auxiliary acceleration variable of the previous iteration process to obtain the first auxiliary summation variable of this iteration process. The first auxiliary summation variable is input into the denoising subnet to determine the first auxiliary variable of this iteration process.

[0037] The second auxiliary acceleration variable from the previous iteration, the output image from the current iteration, and the first auxiliary variable from the current iteration are superimposed to obtain the second auxiliary superimposed variable for the current iteration. The second auxiliary superimposed variable is then passed through an attention-weighted subnet to determine the second auxiliary variable for the current iteration.

[0038] Optionally, based on the auxiliary variables determined in the current iteration and the previous iteration, auxiliary acceleration variables for the current iteration are determined, specifically including:

[0039] The first auxiliary variable of the current iteration process and the previous iteration process are superimposed to obtain the first accelerated superposition variable of the current iteration process. The first accelerated superposition variable is then passed through the attention weighted subnet to determine the first auxiliary accelerated variable of the current iteration process.

[0040] The second auxiliary variable of the current iteration process and the previous iteration process are superimposed to obtain the second accelerated superposition variable of the current iteration process. The second accelerated superposition variable is then passed through an attention-weighted subnet to determine the second auxiliary accelerated variable of the current iteration process.

[0041] Optionally, the feature extraction subnetwork includes at least a global feature extraction layer and a local feature extraction layer, and the input data of the feature extraction subnetwork includes: the input image or the synthetic variable;

[0042] The input image of this iteration process is input into the image reconstruction model. Features are extracted from the input image through the feature extraction subnetwork to obtain the first feature. The first and second auxiliary acceleration variables of this iteration process are summed to obtain a composite variable. Features are then extracted from the composite variable through the feature extraction subnetwork to obtain the second feature, specifically including:

[0043] The input data of this iteration process is input into the global feature extraction layer to determine the channel features;

[0044] The input data of this iteration process is input into the local feature extraction layer to determine spatial features;

[0045] The enhanced features of this iteration process are obtained by multiplying the channel features and the spatial features.

[0046] The enhanced features are multiplied with the input data of the iteration process to determine the output result of the feature extraction subnet of the iteration process. The output result includes the first feature and the second feature.

[0047] Optionally, the denoising subnet includes at least an encoding layer and a decoding layer;

[0048] The first auxiliary summation variable is input into the denoising subnet to determine the first auxiliary variable for this iteration process, specifically including:

[0049] The first auxiliary summation variable is input into the coding layer to obtain the intermediate coding vector of the coding layer and the coding vector of the first auxiliary summation variable;

[0050] The encoding vector of the first auxiliary summation variable is input into the decoding layer to obtain the intermediate decoding vector of the decoding layer. The intermediate decoding vector corresponding to the intermediate encoding vector is determined, and the residual between the intermediate encoding vector and the intermediate decoding vector corresponding to the intermediate encoding vector is calculated.

[0051] Based on the residual and the intermediate decoding vector, the first auxiliary variable for this round of iteration is determined.

[0052] This specification provides an image reconstruction apparatus based on acceleration optimization and attention mechanisms, including:

[0053] The function construction module obtains the image to be reconstructed, and constructs a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed.

[0054] The auxiliary acceleration variable generation module iteratively optimizes the low-rank matrix completion objective function. For each iteration of the low-rank matrix completion objective function, based on the alternating direction multiplier method, it determines the auxiliary variables required to solve the low-rank matrix completion objective function and determines the auxiliary acceleration variables corresponding to the auxiliary variables.

[0055] The model building module, based on a deep unfolded network, constructs an image reconstruction model by completing the objective function using the low-rank matrix.

[0056] The feature extraction module inputs the input image of the current iteration process and the auxiliary acceleration variables of the previous iteration process into the image reconstruction model, performs feature extraction on the input image to obtain the first feature, and performs feature extraction on the auxiliary acceleration variables to obtain the second feature;

[0057] The variable iteration module superimposes the first feature and the second feature to obtain a fused feature, and applies attention weighting to the fused feature to obtain the output image of the current iteration process; based on the output image of the current iteration process and the auxiliary acceleration variables of the previous iteration process, it determines the auxiliary variables of the current iteration process; based on the current iteration process and the auxiliary variables determined in the previous iteration process, it determines the auxiliary acceleration variables of the current iteration process for use in the next iteration process.

[0058] The image reconstruction module takes the output image obtained from the current iteration process and the determined auxiliary acceleration variables of the current iteration process as the input of the image reconstruction model for the next round, and continues to iterate and optimize until the specified number of iterations is reached. The output image obtained from the last iteration is then used as the reconstructed image.

[0059] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image reconstruction method based on acceleration optimization and attention mechanisms.

[0060] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described image reconstruction method based on acceleration optimization and attention mechanisms.

[0061] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0062] As can be seen from the above method, this application optimizes the iterative process of the low-rank matrix completion objective function constructed based on traditional image reconstruction algorithms using a deep unfolding network, and introduces auxiliary acceleration variables during the optimization process. In each iteration, the output image of the current iteration is determined based on the output image of the previous iteration and the auxiliary acceleration variables; the auxiliary variables of the current iteration are determined based on the output image of the current iteration and the auxiliary acceleration variables of the previous iteration; and the auxiliary acceleration variables of the current iteration are determined based on the auxiliary variables of the current iteration and the previous iteration. In this way, there is no direct correlation between the auxiliary acceleration variables of adjacent iterations, which speeds up the iteration and allows this method to further improve the efficiency of image reconstruction while ensuring the accuracy of image reconstruction. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0064] Figure 1This is a flowchart illustrating an image reconstruction method based on acceleration optimization and attention mechanisms provided in this specification.

[0065] Figure 2 This document provides a schematic diagram illustrating the workflow of an image reconstruction method based on acceleration optimization and attention mechanisms.

[0066] Figure 3 This is a schematic diagram of the structure of a feature extraction subnetwork provided in this specification;

[0067] Figure 4 This is a schematic diagram of the structure of an attention-weighted subnet provided in this specification;

[0068] Figure 5 This is a schematic diagram of the structure of a noise reduction subnet provided in this specification;

[0069] Figure 6 This is a schematic diagram of an image reconstruction device based on acceleration optimization and attention mechanism provided in this specification.

[0070] Figure 7 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0072] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0073] Figure 1 This document provides a flowchart illustrating an image reconstruction method based on acceleration optimization and attention mechanisms, comprising the following steps:

[0074] S100: Obtain the image to be reconstructed, and construct a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed.

[0075] During the generation and transmission of medical images, data loss can lead to degraded images, resulting in reduced image clarity, low contrast, and noise, which affects subsequent analysis of the images. Therefore, this application is used to reconstruct degraded images to recover the lost data.

[0076] The execution subject of this manual can be either the terminal device used by the user for image reconstruction or a server. The terminal device can include mobile devices such as mobile phones and tablets. For ease of explanation, the following description uses a server as the execution subject to illustrate the image reconstruction method based on acceleration optimization and attention mechanisms provided in this manual.

[0077] Specifically, the server can respond to user input and acquire the image to be reconstructed. It can also acquire the image using any existing means; this specification does not impose any specific limitations. The image to be reconstructed here can be a medical image or other types of images; this specification does not impose any specific limitations.

[0078] The server performs reconstruction analysis on the image to be reconstructed to determine the corresponding mask. This mask identifies the parts of the image that need to be reconstructed. After all, not all the data in the entire image needs to be reconstructed; the data loss in the image is only a minority, so it is necessary to first identify these parts that need to be reconstructed.

[0079] Then, the server determines a priori regularization terms based on preset constraints for the reconstructed image. These constraints can be based on local smoothing or image patch self-similarity, which are not specifically limited in this specification. Since the reconstructed image is to "repair" defects existing in the original image, the overall difference between the reconstructed image and the original image should not be too large. Therefore, the constraint is to minimize the adjustment of the parts of the image to be reconstructed, and a low-rank matrix completion objective function to be optimized is constructed.

[0080] The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed. The low-rank matrix objective function can be constructed using the following formula:

[0081]

[0082] Where x represents the output reconstructed image, y represents the image to be reconstructed, and F represents the mask in the Fourier domain. The objective is to minimize the adjustment of the part of the image to be reconstructed that needs to be reconstructed. R(x) represents the prior regularization term, and λ represents the balance parameter.

[0083] Based on this low-rank matrix completion objective function, the image to be reconstructed can be further optimized.

[0084] In step S100 above, the server can also simply take minimizing the adjustment of the part of the image to be reconstructed that needs to be reconstructed as the objective and construct the low-rank matrix completion objective function to be optimized, without considering the prior regularization term.

[0085] S102: Iteratively optimize the low-rank matrix completion objective function. For each iteration of the low-rank matrix completion objective function, determine the auxiliary variables required to solve the low-rank matrix completion objective function based on the alternating direction multiplier method, and determine the auxiliary acceleration variables corresponding to the auxiliary variables.

[0086] The server can use the alternating direction multiplier method to solve the above low-rank matrix completion objective function. Specifically, the augmented Lagrangian function corresponding to the low-rank matrix completion objective function can be constructed using the following formula:

[0087]

[0088] Where z is the first auxiliary variable, b is the second auxiliary variable, and ρ is the equilibrium parameter.

[0089] Then, based on the augmented Lagrange function, the function with respect to all variables is separated as follows:

[0090]

[0091] z t+1 =proxNet E,τ (x t+1 +b t )

[0092] b t+1 =b t +β t+1 (x t+1 -z t+1 )

[0093] in, Let τ represent the intermediate image of the iteration process determined with the goal of minimizing the difference between the reconstructed image and the second auxiliary variable of the previous round. τ is the penalty coefficient, E(x) is the residual, and β is the update rate of the Lagrange multiplier. Initially, let x = z and t = 0.

[0094] To optimize the iterative process, auxiliary acceleration variables z′ and b′ are introduced. The optimization result for the above function can be expressed as:

[0095]

[0096] z t+1 =proxNet E,τ(x t+1 +b′ t )

[0097] b t+1 =b′ t +β t+1 (x t+1 -z t+1 )

[0098] b′ t+1 =b t+1 +β t+1 (b t+1 -b t )

[0099] z′ t+1 =z t+1 +β t+1 (z t+1 -z t )

[0100] Where z′ is the first auxiliary acceleration variable and b′ is the second auxiliary acceleration variable. Initially, z′ = z and b′ = b.

[0101] In the initial iteration, at t=0, the initial output image is set to x=z, the initial first auxiliary acceleration variable is z′=z, and the initial second auxiliary acceleration variable is b′=b. Based on the initial output image and the initial auxiliary acceleration variable, the output image of the first iteration is determined. An intermediate image of the first iteration is determined with the goal of minimizing the difference between the reconstructed image and the initial second auxiliary acceleration variable. The residuals between this intermediate image, the output image of the first iteration, and the initial second auxiliary acceleration variable are summed to obtain the first auxiliary variable for the first iteration.

[0102] Then, the difference between the output image of the first iteration process and the first auxiliary variable of the first iteration process is determined as the first auxiliary difference. The first auxiliary difference is weighted by the update rate of the Lagrange multiplier in the first iteration process to obtain the first auxiliary weighted variable. The first auxiliary weighted variable is then summed with the initial second auxiliary acceleration variable to obtain the second auxiliary variable of the first iteration process.

[0103] Then, the difference between the first auxiliary variable and the initial first auxiliary variable in the first iteration process is determined as the first acceleration difference. The first acceleration difference is weighted by the update rate of the Lagrange multiplier in the first iteration process to obtain the first acceleration weighted variable. The first acceleration weighted variable is then summed with the first auxiliary variable in the first iteration process to obtain the first auxiliary acceleration variable in the first iteration process.

[0104] Similarly, the difference between the second auxiliary variable and the initial second auxiliary variable in the first iteration process is determined as the second acceleration difference. This second acceleration difference is weighted by the update rate of the Lagrange multiplier in the first iteration process to obtain the second acceleration weighted variable. This second acceleration weighted variable is then summed with the second auxiliary variable in the first iteration process to obtain the second auxiliary acceleration variable in the first iteration process.

[0105] Finally, the output image of the first iteration, the auxiliary variables of the first iteration, and the auxiliary acceleration variables of the first iteration are used as the input of the second iteration. Furthermore, in subsequent iterations, the output of the previous round is used as the input of the next round until the specified number of iterations is reached.

[0106] In this way, after introducing auxiliary acceleration variables, there is no direct correlation between two adjacent iterations of x. The output image of each iteration is determined by the output image of the previous iteration and the two auxiliary acceleration variables, which speeds up the iteration process.

[0107] For x t+1 The least squares method can be used to solve it, and the results are as follows:

[0108] x t+1 =(F T F+ρI) -1 (F T y+ρz′ t -b′ t )

[0109] Where I is the identity matrix.

[0110] In step S102 above, the server applies a weighting operation using the update rate of the Lagrange multiplier when determining the second auxiliary variable, the first auxiliary acceleration variable, and the second auxiliary acceleration variable. In this embodiment, the weighting operation can also be omitted to obtain the second auxiliary variable, the first auxiliary acceleration variable, and the second auxiliary acceleration variable for each iteration.

[0111] S104: Based on the deep unfolded network, the objective function is completed by the low-rank matrix to construct an image reconstruction model.

[0112] Deep unfolded networks are deep neural networks built on the basis of traditional iterative frameworks, combining the interpretability of traditional iterative frameworks with the high computational performance of deep neural networks.

[0113] The server, based on a deep unfolded network, can construct an image reconstruction model. This model includes at least three subnetworks: a feature extraction subnetwork, an attention-weighted subnetwork, and a denoising subnetwork. The iterative process is further optimized using the following method:

[0114]

[0115] z t+1 =D t (x t+1 +b′ t )

[0116]

[0117]

[0118]

[0119] in, This indicates superposition along the channel direction, + indicates point-by-point summation, C att (g) represents the attention-weighted subnet, D t (g) represents the denoising subnet. This represents the feature extraction subnet.

[0120] When the server performs an iterative process using an image reconstruction model, such as Figure 2 As shown, Figure 2 This document provides a schematic diagram illustrating the workflow of an image reconstruction method based on acceleration optimization and attention mechanisms.

[0121] first step,

[0122] For each iteration, the server takes the input image of the current iteration, the first auxiliary acceleration variable from the previous iteration, and the second auxiliary acceleration variable from the previous iteration, and inputs them into the image reconstruction model. The model then uses a feature extraction subnetwork to extract features from the input image, obtaining the first feature. Simultaneously, the server sums the first and second auxiliary acceleration variables of the current iteration to obtain a composite variable. This composite variable is then used by the feature extraction subnetwork to extract features, yielding the second feature. Finally, the first and second features are fused to obtain the fused feature. This fused feature is then attention-weighted using an attention-weighting subnetwork to obtain the output image for the current iteration.

[0123] In order to obtain an output image of the same size as the input image, zero padding can be performed around the input image when reconstructing the model with the input image in each iteration process.

[0124] Step two, z t+1 =D t (x t+1 +b′ t ):

[0125] The output image obtained in this iteration is summed with the second auxiliary acceleration variable from the previous iteration to obtain the first auxiliary sum variable for this iteration. This first auxiliary sum variable is then passed through a denoising subnet to determine the first auxiliary variable for this iteration.

[0126] The third step,

[0127] The second auxiliary acceleration variable from the previous iteration, the output image from the current iteration, and the first auxiliary variable from the current iteration are superimposed to obtain the second auxiliary superimposed variable for the current iteration. This second auxiliary superimposed variable is then passed through an attention-weighted subnet to determine the second auxiliary variable for the current iteration.

[0128] Step 4

[0129] Based on the update rate of the Lagrange multiplier, the first auxiliary variable of the current iteration process and the first auxiliary variable of the previous iteration process are weighted respectively to obtain the first weighted variable of the current iteration process and the first weighted variable of the previous iteration process. The first weighted variable of the current iteration process and the first weighted variable of the previous iteration process are superimposed to obtain the first accelerated superposition variable of the current iteration process. The first accelerated superposition variable is passed through the attention weighting subnet to determine the first auxiliary accelerated variable of the current iteration process.

[0130] Step 5

[0131] Based on the update rate of the Lagrange multiplier, the second auxiliary variable of the current iteration process and the second auxiliary variable of the previous iteration process are weighted respectively to obtain the second weighted variable of the current iteration process and the second weighted variable of the previous iteration process. The second weighted variable of the current iteration process and the second weighted variable of the previous iteration process are superimposed to obtain the second acceleration superposition variable of the current iteration process. The second acceleration superposition variable is then passed through the attention weighting subnet to determine the second auxiliary acceleration variable of the current iteration process.

[0132] In this way, with the output image, auxiliary variables, and auxiliary acceleration variables of this round, these parameters can be used as inputs for the next round of the image reconstruction model to continue the iterative process.

[0133] In step S102 above, when the server uses the image reconstruction model to determine the first auxiliary acceleration variable and the second auxiliary acceleration variable, a weighting operation is performed using the update rate of the Lagrange multiplier. In this embodiment, the first auxiliary acceleration variable and the second auxiliary acceleration variable for each iteration process can also be obtained without performing a weighting operation.

[0134] S106: Input the input image of this iteration process and the auxiliary acceleration variables of the previous iteration process into the image reconstruction model, perform feature extraction on the input image to obtain the first feature, and perform feature extraction on the auxiliary acceleration variables to obtain the second feature.

[0135] The server extracts features from the input image and auxiliary acceleration variables in each iteration through a feature extraction subnet, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a feature extraction subnetwork provided in this specification. The feature extraction subnetwork includes at least a global feature extraction layer and a local feature extraction layer, wherein the global feature extraction layer includes an average pooling sublayer, a global convolutional sublayer, and a ReLU activation function sublayer, and the local feature extraction layer includes a local convolutional sublayer and a ReLU activation function sublayer.

[0136] Both the ReLU and Sigmoid activation functions are non-linear activation functions, and their calculation formulas are as follows:

[0137] f Relu (x) = max(0,x)

[0138]

[0139] The input data for the feature extraction subnetwork includes the input image or the synthesized variables. In each iteration, the input image of the feature extraction subnetwork first passes through a global feature extraction layer to obtain global image features. These global features are then passed sequentially through a first convolutional layer and a first sigmoid activation function layer to obtain image channel features. Simultaneously, local image features are obtained through a local feature extraction layer. The local and global image features are superimposed to obtain the basic image features. These basic features are then passed sequentially through a second convolutional layer and a second sigmoid activation function layer to obtain image spatial features. The image channel features and image spatial features are multiplied to obtain image enhancement features. These enhanced features are then multiplied with the input image to obtain the first feature for that iteration.

[0140] In each iteration, the first and second auxiliary acceleration variables are summed to obtain the composite variable. Similarly, when the server extracts features from this composite variable through the feature extraction subnet, it first passes the global feature extraction layer to obtain the global features of the variable. These global features are then passed sequentially through the first convolutional layer and the first sigmoid activation function layer to obtain the channel features of the variable. Simultaneously, it passes the local feature extraction layer to obtain the local features of the variable. The local and global features are then superimposed to obtain the basic features of the variable. These basic features are then passed sequentially through the second convolutional layer and the second sigmoid activation function layer to obtain the spatial features of the variable. The channel features and spatial features are then multiplied to obtain the enhanced features of the variable. Finally, the enhanced features are multiplied by the input composite variable to obtain the second feature of that iteration.

[0141] To achieve the above process, the global convolutional sublayer can be set to have 32 input channels, 8 output channels, a 3×3 kernel size, and a stride of 1; the first convolutional layer can be set to have 8 input channels, 32 output channels, a 3×3 kernel size, and a stride of 1. Simultaneously, the local convolutional sublayer can be set to have 32 input channels, 8 output channels, a 3×3 kernel size, and a stride of 1; the second convolutional layer can be set to have 8 input channels, 32 output channels, a 3×3 kernel size, and a stride of 1. Of course, other convolutional layer settings can also be used depending on the specific situation, and this manual does not specifically limit this.

[0142] The feature extraction subnetwork employs a method that fuses global and local features, as well as channel attention and spatial attention, resulting in more accurate feature extraction and more precise image reconstruction.

[0143] S108: Superimpose the first feature and the second feature to obtain a fused feature, and apply attention weighting to the fused feature to obtain the output image of this iteration process; determine the auxiliary variables of this iteration process based on the output image of this iteration process and the auxiliary acceleration variables of the previous iteration process; determine the auxiliary acceleration variables of this iteration process based on the auxiliary variables determined in this iteration process and the previous iteration process, for use in the next iteration process.

[0144] The server performs a superposition operation on the first and second features obtained from the feature extraction subnet to obtain a fused feature, and then applies attention weighting to the fused feature through an attention weighting subnet, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an attention-weighted subnetwork provided in this specification. The attention-weighted subnetwork includes at least an average pooling layer, a convolutional layer, a ReLU activation function layer, and a Sigmoid activation function layer.

[0145] Specifically, in each iteration, the fused features are sequentially passed through the average pooling layer, the first convolutional layer, the ReLU activation function layer, the second convolutional layer, and the Sigmoid activation function layer of the attention-weighted subnet to obtain the first intermediate feature; the first intermediate feature is multiplied by the input fused features to obtain the second intermediate feature; the second intermediate feature is passed through the third convolutional layer to obtain the output image of that iteration.

[0146] Figure 5 This is a schematic diagram of the structure of a denoising subnet provided in this specification. The denoising subnet includes at least an encoding layer and a decoding layer. The encoding layer includes a first convolutional sublayer, a second convolutional sublayer, an encoding ViT sublayer, a third convolutional sublayer, and a fourth convolutional sublayer. The decoding layer includes a first deconvolutional sublayer, a second deconvolutional sublayer, a decoding ViT sublayer, a third deconvolutional sublayer, a fourth deconvolutional sublayer, and a ReLU activation function sublayer.

[0147] In addition, the encoding ViT sublayer and the decoding ViT sublayer contain the VisionTransform module, whose calculation formula is as follows:

[0148]

[0149] Where (Q,K,V) represents the vector group after eigenvalue decomposition, d represents the vector length, softmax is a non-linear mapping function, and T represents the transpose operation.

[0150] The VisionTransform module assigns different weights to each data point based on its impact on the reconstructed image in each round of input. The attention weight probability for each data point is then calculated using softmax. Data points with a large impact on the reconstructed image undergo feature enhancement, while data points with a small impact are weakened, resulting in a more accurate reconstructed image.

[0151] For each iteration, the server generates the first auxiliary variable for that iteration using the denoising subnet. Specifically, the server sums the output image obtained in the current iteration with the second auxiliary acceleration variable from the previous iteration to obtain the first auxiliary sum variable. This first auxiliary sum variable is then passed sequentially through the first and second convolutional sublayers of the denoising subnet's coding layer to obtain the first intermediate coding vector. This first intermediate coding vector is then passed sequentially through the coding ViT sublayer and the third convolutional sublayer to obtain the second intermediate coding vector. Finally, this second intermediate coding vector is passed through the fourth convolutional sublayer to obtain the coding vector of the first auxiliary variable for that iteration.

[0152] Then, the encoded vector is passed through the first deconvolutional sublayer of the decoding layer to obtain the first decoded vector; the residual between the first decoded vector and the second encoded vector is calculated as the first residual; based on the first residual, the first decoded vector is passed through the second deconvolutional sublayer and the decoding ViT sublayer in sequence to obtain the second decoded vector; the residual between the second decoded vector and the first encoded vector is calculated as the second residual; based on the second residual, the second decoded vector is passed through the third deconvolutional sublayer and the fourth deconvolutional sublayer in sequence to obtain the third decoded vector; the residual between the third decoded vector and the input of the denoising subnet coding layer is calculated as the third residual; based on the third residual, the third decoded vector is passed through the ReLU activation function sublayer to obtain the first auxiliary variable of this iteration process.

[0153] It should be noted that in the process of determining the first auxiliary variable based on the residual in the denoising subnet, the first residual, the second residual, and the third residual used above are only one way of selecting residuals. Other residuals between the coding and decoding layers can also be used in the iterative process.

[0154] Furthermore, the server uses attention-weighted subnets to determine the second auxiliary variable, the first auxiliary acceleration variable, and the second auxiliary acceleration variable for each iteration.

[0155] First, the server superimposes the output image of the current iteration, the first auxiliary variable of the current iteration, and the second auxiliary acceleration variable of the previous iteration to obtain a second auxiliary superimposed variable. This second auxiliary superimposed variable is then passed through an attention-weighted subnet to determine the second auxiliary variable for the current iteration.

[0156] Specifically, the server sequentially passes the aforementioned second auxiliary superimposed variable through the average pooling layer, the first convolutional layer, the ReLU activation function layer, the second convolutional layer, and the Sigmoid activation function layer of the attention-weighted subnet to obtain the second auxiliary intermediate variable; it then multiplies the second auxiliary intermediate variable with the aforementioned second auxiliary superimposed variable to obtain the second auxiliary weighted variable; finally, it passes the second auxiliary weighted variable through the third convolutional layer of the attention-weighted subnet to obtain the second auxiliary variable for this iteration process.

[0157] Secondly, the server superimposes the first auxiliary variable of the current iteration with the first auxiliary variable of the previous iteration to obtain the first accelerated superposition variable. This first accelerated superposition variable is then passed through an attention-weighted subnet to determine the first auxiliary accelerated variable of the current iteration.

[0158] Specifically, the server sequentially passes the aforementioned first acceleration superposition variable through the average pooling layer, the first convolutional layer, the ReLU activation function layer, the second convolutional layer, and the Sigmoid activation function layer of the attention-weighted subnet to obtain the first acceleration intermediate variable; the server then multiplies the first acceleration intermediate variable with the aforementioned first acceleration superposition variable to obtain the first acceleration weighted variable; and finally passes the first acceleration weighted variable through the third convolutional layer of the attention-weighted subnet to obtain the first auxiliary acceleration variable for this iteration process.

[0159] Then, the server performs a superposition operation on the second auxiliary variable of the current iteration and the previous iteration to obtain the second accelerated superposition variable. This second accelerated superposition variable is then passed through an attention-weighted subnet to determine the second auxiliary accelerated variable for the current iteration.

[0160] Specifically, the server sequentially passes the aforementioned second acceleration superposition variable through the average pooling layer, the first convolutional layer, the ReLU activation function layer, the second convolutional layer, and the Sigmoid activation function layer of the attention-weighted subnet to obtain the second acceleration intermediate variable. This second acceleration intermediate variable is then multiplied by the aforementioned second acceleration superposition variable to obtain the second acceleration weighted variable. Finally, this second acceleration weighted variable is passed through the third convolutional layer of the attention-weighted subnet to obtain the second auxiliary acceleration variable for this iteration process.

[0161] It should be noted that the execution process of determining the first and second auxiliary acceleration variables in each iteration can be done in any order, or the first and second auxiliary acceleration variables can be determined simultaneously.

[0162] To achieve the above process, the first convolutional layer in the attention-weighted subnet can be set to have 64 input channels, 8 output channels, a kernel size of 1×1, and a stride of 1; the second convolutional layer can be set to have 8 input channels, 64 output channels, a kernel size of 1×1, and a stride of 1; and the third convolutional layer can be set to have 64 input channels, 32 output channels, a kernel size of 1×1, and a stride of 1.

[0163] Meanwhile, the convolutional sublayers in the coding layer of the denoising subnet are all set with 32 input channels, 32 output channels, a kernel size of 5×5, and a stride of 1; the deconvolutional sublayers in the decoding layer are all set with 32 input channels, 32 output channels, a kernel size of 5×5, and a stride of 1. Of course, depending on the specific situation, other convolutional layer settings can also be used, and this specification does not specify any limitations on this.

[0164] In step S110 above, the feature extraction subnetwork uses a method that fuses global and local features, and channel attention and spatial attention to enhance features. In the denoising subnetwork, the VisionTransform module further enhances features, making the extracted features more accurate and improving the accuracy of the repair.

[0165] S110: The output image obtained in the current iteration process and the auxiliary acceleration variable determined in the current iteration process are used as the input of the image reconstruction model in the next round. The iteration optimization continues until the specified number of iterations is reached, and the output image obtained in the last iteration is used as the reconstructed image.

[0166] In each iteration, the server uses the output image obtained from the iteration and the auxiliary acceleration variables determined in that iteration as the input for the next iteration of the image reconstruction model, and continues the iteration process until the preset number of iterations is reached. The output image obtained in the last iteration is then used as the reconstructed image.

[0167] Figure 6 This is a schematic diagram of the structure of an image reconstruction device based on acceleration optimization and attention mechanism provided in this specification, specifically including:

[0168] The function construction module 200 obtains the image to be reconstructed and constructs a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed.

[0169] The auxiliary acceleration variable generation module 202 iteratively optimizes the low-rank matrix completion objective function. For each iteration of the low-rank matrix completion objective function, based on the alternating direction multiplier method, it determines the auxiliary variables required to solve the low-rank matrix completion objective function and determines the auxiliary acceleration variables corresponding to the auxiliary variables.

[0170] The model building module 204, based on a deep unfolded network, constructs an image reconstruction model by completing the objective function with the low-rank matrix.

[0171] The feature extraction module 206 inputs the input image of the current iteration process and the auxiliary acceleration variables of the previous iteration process into the image reconstruction model, performs feature extraction on the input image to obtain the first feature, and performs feature extraction on the auxiliary acceleration variables to obtain the second feature.

[0172] The variable iteration module 208 superimposes the first feature and the second feature to obtain a fused feature, and applies attention weighting to the fused feature to obtain the output image of the current iteration process; based on the output image of the current iteration process and the auxiliary acceleration variables of the previous iteration process, it determines the auxiliary variables of the current iteration process; based on the current iteration process and the auxiliary variables determined in the previous iteration process, it determines the auxiliary acceleration variables of the current iteration process for use in the next iteration process.

[0173] The image reconstruction module 210 takes the output image obtained in the current iteration process and the determined auxiliary acceleration variables of the current iteration process as the input of the next iteration of the image reconstruction model, and continues to iterate and optimize until the specified number of iterations is reached. The output image obtained in the last iteration is used as the reconstructed image.

[0174] Optionally, the function construction module 200 is specifically used to: perform reconstruction analysis on the image to be reconstructed; determine the mask corresponding to the image to be reconstructed, wherein the mask identifies the part of the image to be reconstructed that needs to be reconstructed; determine a prior regularization term according to the preset constraints of the reconstructed image; and construct a low-rank matrix completion objective function to be optimized based on the prior regularization term, with the goal of minimizing the adjustment of the part of the image to be reconstructed that needs to be reconstructed.

[0175] Optionally, the auxiliary acceleration variable generation module 202 is specifically used to: construct an augmented Lagrangian function corresponding to the low-rank matrix completion objective function based on the alternating direction multiplier method and the low-rank matrix completion objective function; determine auxiliary variables for solving the low-rank matrix completion model based on the augmented Lagrangian function; initialize auxiliary acceleration variables corresponding to the auxiliary variables based on the auxiliary variables; and determine the auxiliary acceleration variables corresponding to the auxiliary variables based on the update rate of the auxiliary variables and the Lagrangian multipliers.

[0176] Optionally, the auxiliary variables include a first auxiliary variable and a second auxiliary variable, the functional expressions of which are derived from the augmented Lagrange function. Specifically, the auxiliary acceleration variable generation module 202 is used to: determine the first auxiliary variable for the current iteration based on the output image of the current iteration and the second auxiliary variable of the previous iteration; determine the second auxiliary variable for the current iteration based on the output image of the current iteration, the second auxiliary variable of the previous iteration, and the first auxiliary variable of the current iteration; determine the first auxiliary acceleration variable for the current iteration based on the first auxiliary variable of the current iteration and the previous iteration; and determine the second auxiliary acceleration variable for the current iteration based on the second auxiliary variable of the current iteration and the previous iteration.

[0177] Optionally, the image reconstruction model includes at least a feature extraction subnetwork, an attention weighting subnetwork, and a denoising subnetwork. The feature extraction module 206 is specifically used to input the input image of the iteration process into the image reconstruction model, extract features from the input image through the feature extraction subnetwork to obtain a first feature; sum the first and second auxiliary acceleration variables of the iteration process to obtain a synthetic variable, and extract features from the synthetic variable through the feature extraction subnetwork to obtain a second feature.

[0178] Optionally, the feature extraction subnetwork includes at least a global feature extraction layer and a local feature extraction layer. The input data of the feature extraction subnetwork includes the input image or the synthetic variable. The feature extraction module 206 is specifically used to: input the input data of the iteration process into the global feature extraction layer to determine channel features; input the input data of the iteration process into the local feature extraction layer to determine spatial features; perform a product operation on the channel features and the spatial features to obtain the enhanced features of the iteration process; and perform a product operation on the enhanced features and the input data of the iteration process to determine the output result of the feature extraction subnetwork of the iteration process. The output result includes a first feature and a second feature.

[0179] Optionally, the variable iteration module 208 is specifically used to perform attention weighting on the fused features through the attention weighting subnet to obtain the output image of this iteration process.

[0180] Optionally, the variable iteration module 208 is specifically used to: sum the output image obtained in the current iteration process and the second auxiliary acceleration variable of the previous iteration process to obtain the first auxiliary sum variable of the current iteration process; input the first auxiliary sum variable into the denoising subnetwork to determine the first auxiliary variable of the current iteration process; superimpose the second auxiliary acceleration variable of the previous iteration process, the output image obtained in the current iteration process, and the determined first auxiliary variable of the current iteration process to obtain the second auxiliary superimposed variable of the current iteration process; and pass the second auxiliary superimposed variable through the attention weighting subnetwork to determine the second auxiliary variable of the current iteration process.

[0181] Optionally, the variable iteration module 208 is specifically used to: superimpose the first auxiliary variable of the current iteration process and the previous iteration process to obtain the first accelerated superimposed variable of the current iteration process; pass the first accelerated superimposed variable through an attention-weighted subnet to determine the first auxiliary accelerated variable of the current iteration process; superimpose the second auxiliary variable of the current iteration process and the previous iteration process to obtain the second accelerated superimposed variable of the current iteration process; pass the second accelerated superimposed variable through an attention-weighted subnet to determine the second auxiliary accelerated variable of the current iteration process.

[0182] Optionally, the denoising subnet includes at least an encoding layer and a decoding layer. The variable iteration module 208 is specifically used to: input the first auxiliary summation variable into the encoding layer to obtain the intermediate encoding vector of the encoding layer and the encoding vector of the first auxiliary summation variable; input the encoding vector of the first auxiliary summation variable into the decoding layer to obtain the intermediate decoding vector of the decoding layer; determine the intermediate decoding vector corresponding to the intermediate encoding vector; calculate the residual between the intermediate encoding vector and the intermediate decoding vector corresponding to the intermediate encoding vector; and determine the first auxiliary variable for this round of iteration based on the residual and the intermediate decoding vector.

[0183] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described image reconstruction method based on acceleration optimization and attention mechanisms.

[0184] This instruction manual also provides Figure 7 The diagram shows a schematic structural representation of the electronic device. Figure 7 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to achieve the image reconstruction based on acceleration optimization and attention mechanisms.

[0185] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0186] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0187] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0188] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0189] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0190] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0194] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0195] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0196] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0197] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0198] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0200] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0201] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. An image reconstruction method based on acceleration optimization and attention mechanism, characterized in that, include: Obtain the image to be reconstructed, and construct a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed. The objective function for low-rank matrix completion is iteratively optimized. For each iteration of the objective function for low-rank matrix completion, the auxiliary variables required to solve the objective function for low-rank matrix completion are determined based on the alternating direction multiplier method, and the auxiliary acceleration variables corresponding to the auxiliary variables are determined. Based on a deep unfolded network, an image reconstruction model is constructed by completing the objective function with the low-rank matrix, wherein the image reconstruction model includes at least a feature extraction subnetwork; The input image of the current iteration process and the auxiliary acceleration variables of the previous iteration process are input into the image reconstruction model. The feature extraction subnetwork is used to extract features from the input image to obtain the first feature. The feature extraction subnetwork is used to extract features from the synthetic variable obtained by summing the first and second auxiliary acceleration variables of the current iteration process to obtain the second feature. The first feature and the second feature are superimposed to obtain a fused feature, and the fused feature is then weighted by attention to obtain the output image of this iteration process; Based on the output image of this iteration process and the auxiliary acceleration variables of the previous iteration process, determine the auxiliary variables of this iteration process; Based on the auxiliary variables determined in this iteration process and the previous iteration process, determine the auxiliary acceleration variables for this iteration process, which will be used in the next iteration process. The output image obtained from the first iteration and the auxiliary acceleration variables determined for the first iteration are used as the input for the next iteration of the image reconstruction model. The iteration continues until the specified number of iterations is reached, and the output image obtained from the last iteration is used as the reconstructed image.

2. The method as described in claim 1, characterized in that, Based on the image to be reconstructed, a low-rank matrix completion objective function to be optimized is constructed, specifically including: The image to be reconstructed is reconstructed to determine the mask corresponding to the image to be reconstructed. The mask identifies the part of the image to be reconstructed that needs to be reconstructed. Based on the preset constraints of the reconstructed image, determine the prior regularization term; Based on the prior regularization term, a low-rank matrix completion objective function is constructed with the goal of minimizing the adjustment of the parts of the image to be reconstructed that require image reconstruction.

3. The method as described in claim 1, characterized in that, Based on the alternating direction multiplier method, auxiliary variables required to solve the objective function of the low-rank matrix completion are determined, and auxiliary acceleration variables corresponding to these auxiliary variables are also determined, specifically including: Based on the alternating direction multiplier method, the augmented Lagrange function corresponding to the low-rank matrix completion objective function is constructed according to the low-rank matrix completion objective function; Based on the augmented Lagrangian function, determine the auxiliary variables for solving the low-rank matrix completion model; Based on the auxiliary variable, the corresponding auxiliary acceleration variable is initialized. Based on the update rates of the auxiliary variables and the Lagrange multipliers, the corresponding auxiliary acceleration variables are determined.

4. The method as described in claim 1, characterized in that, The auxiliary variables include a first auxiliary variable and a second auxiliary variable, and the functional expressions of the first auxiliary variable and the second auxiliary variable are derived by separating the augmented Lagrange function; Determine the auxiliary variables required to solve the objective function of the low-rank matrix completion, and determine the corresponding auxiliary acceleration variables, specifically including: Based on the output image of this iteration process and the second auxiliary variable of the previous iteration process, determine the first auxiliary variable of this iteration process; Based on the output image of this iteration process, the second auxiliary variable of the previous iteration process, and the first auxiliary variable of this iteration process, determine the second auxiliary variable of this iteration process; Based on the first auxiliary variables of the current iteration process and the previous iteration process, determine the first auxiliary acceleration variable of the current iteration process; Based on the second auxiliary variable of the current iteration process and the previous iteration process, determine the second auxiliary acceleration variable of the current iteration process.

5. The method as described in claim 1, characterized in that, The image reconstruction model further includes an attention-weighted subnetwork, wherein the fused features are attention-weighted to obtain the output image of this iteration process, specifically including: The fused features are then weighted by the attention-weighted subnet to obtain the output image for this iteration process.

6. The method as described in claim 1, characterized in that, The image reconstruction model further includes a denoising subnetwork. The determination of auxiliary variables for the current iteration process based on the output image of the current iteration and the auxiliary acceleration variables of the previous iteration specifically includes: The output image obtained in this iteration process is summed with the second auxiliary acceleration variable of the previous iteration process to obtain the first auxiliary summation variable of this iteration process. The first auxiliary summation variable is input into the denoising subnet to determine the first auxiliary variable of this iteration process. The second auxiliary acceleration variable from the previous iteration, the output image from the current iteration, and the first auxiliary variable from the current iteration are superimposed to obtain the second auxiliary superimposed variable for the current iteration. The second auxiliary superimposed variable is then passed through an attention-weighted subnet to determine the second auxiliary variable for the current iteration.

7. The method as described in claim 1, characterized in that, Based on the auxiliary variables determined in this iteration and the previous iteration, determine the auxiliary acceleration variables for this iteration, specifically including: The first auxiliary variable of the current iteration process and the previous iteration process are superimposed to obtain the first accelerated superposition variable of the current iteration process. The first accelerated superposition variable is then passed through the attention weighted subnet to determine the first auxiliary accelerated variable of the current iteration process. The second auxiliary variable of the current iteration process and the previous iteration process are superimposed to obtain the second accelerated superposition variable of the current iteration process. The second accelerated superposition variable is then passed through an attention-weighted subnet to determine the second auxiliary accelerated variable of the current iteration process.

8. The method as described in claim 1, characterized in that, The feature extraction subnetwork includes at least a global feature extraction layer and a local feature extraction layer, and the input data of the feature extraction subnetwork includes: the input image or the synthetic variable; The input image of this iteration process is input into the image reconstruction model. Features are extracted from the input image through the feature extraction subnetwork to obtain the first feature. The first and second auxiliary acceleration variables of this iteration process are summed to obtain a composite variable. Features are then extracted from the composite variable through the feature extraction subnetwork to obtain the second feature, specifically including: The input data of this iteration process is input into the global feature extraction layer to determine the channel features; The input data of this iteration process is input into the local feature extraction layer to determine spatial features; The enhanced features of this iteration process are obtained by multiplying the channel features and the spatial features. The enhanced features are multiplied with the input data of the iteration process to determine the output result of the feature extraction subnet of the iteration process. The output result includes the first feature and the second feature.

9. The method as described in claim 6, characterized in that, The denoising subnet includes at least an encoding layer and a decoding layer; The first auxiliary summation variable is input into the denoising subnet to determine the first auxiliary variable for this iteration process, specifically including: The first auxiliary summation variable is input into the coding layer to obtain the intermediate coding vector of the coding layer and the coding vector of the first auxiliary summation variable; The encoding vector of the first auxiliary summation variable is input into the decoding layer to obtain the intermediate decoding vector of the decoding layer. The intermediate decoding vector corresponding to the intermediate encoding vector is determined, and the residual between the intermediate encoding vector and the intermediate decoding vector corresponding to the intermediate encoding vector is calculated. Based on the residual and the intermediate decoding vector, the first auxiliary variable for this round of iteration is determined.

10. An image reconstruction apparatus based on acceleration optimization and attention mechanisms, characterized in that, include: The function construction module obtains the image to be reconstructed, and constructs a low-rank matrix completion objective function to be optimized based on the image to be reconstructed. The low-rank matrix completion objective function is used to characterize the relationship between the reconstructed image and the image to be reconstructed. The auxiliary acceleration variable generation module iteratively optimizes the low-rank matrix completion objective function. For each iteration of the low-rank matrix completion objective function, based on the alternating direction multiplier method, it determines the auxiliary variables required to solve the low-rank matrix completion objective function and determines the auxiliary acceleration variables corresponding to the auxiliary variables. The model building module, based on a deep unfolded network, constructs an image reconstruction model by completing the objective function with the low-rank matrix, wherein the image reconstruction model includes at least a feature extraction subnetwork; The feature extraction module inputs the input image of the current iteration process and the auxiliary acceleration variables of the previous iteration process into the image reconstruction model. It extracts features from the input image through the feature extraction subnetwork to obtain the first feature. It also extracts features from the synthetic variable obtained by summing the first and second auxiliary acceleration variables of the current iteration process through the feature extraction subnetwork to obtain the second feature. The variable iteration module superimposes the first feature and the second feature to obtain a fused feature, and applies attention weighting to the fused feature to obtain the output image of the current iteration process; based on the output image of the current iteration process and the auxiliary acceleration variables of the previous iteration process, it determines the auxiliary variables of the current iteration process; based on the current iteration process and the auxiliary variables determined in the previous iteration process, it determines the auxiliary acceleration variables of the current iteration process for use in the next iteration process. The image reconstruction module takes the output image obtained from the current iteration process and the determined auxiliary acceleration variables of the current iteration process as the input of the image reconstruction model for the next round, and continues to iterate and optimize until the specified number of iterations is reached. The output image obtained from the last iteration is used as the reconstructed image.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 9.

Images