An image restoration method, device, equipment and readable storage medium
By introducing the Anderson acceleration module into the depth fixed-point method and using intermediate states for iterative estimation, the problem of slow inference speed caused by a large number of iterations is solved, and the efficiency of image restoration is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2022-12-16
- Publication Date
- 2026-04-24
AI Technical Summary
Existing deep fixed-point methods require a large number of iterations of the compression mapping function based on deep neural network parameterization during the iterative approximation of the fixed point, resulting in slow inference speed.
The Anderson acceleration module is introduced into the hidden layer representation neural network. By using the intermediate states before the current state in the fixed-point iteration process for estimation and correction, the number of iterations is reduced.
It improves the convergence speed of depth fixed-point mapping, reduces the number of iterations, and enhances the efficiency of image restoration.
Smart Images
Figure CN115797219B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image restoration, and in particular to an image restoration method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] Image restoration refers to the reconstruction of a high-quality output image from a low-quality input image using restoration methods, and it is one of the fundamental problems in computer vision. With the rapid development of deep neural networks and their breakthroughs in many areas of artificial intelligence, hidden layer neural network methods based on deep fixed-point methods have received widespread attention and research. In deep fixed-point methods, the fixed-point mapping function is parameterized by the deep neural network. By iterating this mapping function, that is, by using the feature representation of the deep fixed-point output as input features again, the equivalent depth of the deep neural network can be effectively increased without increasing the number of network layers or parameters, thereby achieving a good feature representation value and improving the quality of the restored image.
[0003] Currently, although the deep fixed-point method can realize implicitly infinitely deep neural networks with limited memory consumption, it requires a large number of iterations of the compression mapping function parameterized by the deep neural network during the iterative approximation of the fixed point, usually on the order of hundreds or thousands, which leads to slow inference speed. Summary of the Invention
[0004] The purpose of this invention is to provide an image restoration method, apparatus, device, and computer-readable storage medium for application in the field of image restoration. The method of this invention reduces the number of iterations for depth fixed-point mapping and improves the convergence speed of depth fixed-point mapping by introducing Anderson acceleration into the hidden layer representation neural network.
[0005] To achieve the above objectives, the present invention proposes an image restoration method, comprising:
[0006] Receive the image to be recovered, and input the image into a head neural network to obtain feature values;
[0007] The feature values and initialized latent variables are input into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module;
[0008] Determine whether the feature neural network satisfies the compression mapping condition;
[0009] If so, stop the iteration and input the corresponding precise feature values into the tail neural network to obtain the restored image;
[0010] If not, continue iterating until it is determined that the feature neural network satisfies the compression mapping condition.
[0011] Optionally, the step of inputting the feature values and initialized latent variables into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables includes:
[0012] The feature values and the initial hidden variables are input into the hidden layer representation neural network for iteration to obtain the precise feature values and the iterative hidden variables; wherein the feature neural network is a residual structure convolutional neural network.
[0013] Optionally, the step of inputting the feature values and initialized latent variables into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables includes:
[0014] In the hidden layer representation neural network, the first weighting coefficient is determined, the second weighting coefficient is determined, and the residual is determined;
[0015] The feature values, the initialized latent variables, the first weighting coefficient, the second weighting coefficient, and the residual are input into the first model in the hidden layer representation neural network and iterated to obtain the precise feature values and the iterative latent variables; wherein, the mathematical expression of the first model is:
[0016]
[0017] In the formula, norm is the normalization operation of the deep neural network layer, conv is the convolution operator of the deep neural network, and z x Let z be the first weighting coefficient. h The second weighting coefficient is denoted as , resi is the residual, and f is the weighting coefficient. θ For the feature neural network, x t x is the precise feature value of the current step. t+1 h represents the precise feature value after one iteration in the current step. t h is the latent variable for the current iteration step. t+1 This is the iterative hidden variable after the current step has been iterated once.
[0018] Optionally, determining the residual includes:
[0019] The residual is determined based on the precise feature values using a second model in the hidden layer representation neural network; wherein the mathematical expression of the second model is:
[0020] resi = conv(f θ (x t )-x t )
[0021] In the formula, conv is the convolution operator of the deep neural network, and f θ For the feature neural network, x t This is the precise feature value for the current step.
[0022] Optionally, determining the second weighting coefficient includes:
[0023] The second weighting coefficients are determined using a third model in the hidden layer representation neural network based on the residuals and the iterative hidden variables; wherein the mathematical expression of the third model is:
[0024] z h =sigmoid(conv(resi)+conv(h) t ))
[0025] In the formula, sigmoid is the nonlinear activation function of the deep neural network, conv is the convolution operator of the deep neural network, resi is the residual, and h t Let be the hidden variable for the current iteration step.
[0026] Optionally, determining the first weighting coefficient includes:
[0027] The first weighting coefficient is determined based on the second weighting coefficient using the fourth model in the hidden layer representation neural network; wherein, the mathematical expression of the fourth model is:
[0028] z x =sigmoid(conv(z h ))
[0029] In the formula, z h is the second weighting coefficient, sigmoid is the nonlinear activation function of the deep neural network, and conv is the convolution operator of the deep neural network.
[0030] To achieve the above objectives, the present invention also provides an image restoration apparatus, comprising:
[0031] The receiving module is used to receive the image to be recovered and input the image into the head neural network to obtain feature values;
[0032] An iterative module is used to input the feature values and initialized latent variables into a hidden layer representation neural network for iterative processing to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module;
[0033] The judgment module is used to determine whether the feature neural network satisfies the compression mapping condition;
[0034] The execution module is used to stop the iteration if the condition is met, and input the corresponding precise feature values into the tail neural network to obtain the restored image.
[0035] The loop module is used to continue iterating if not, until it is determined that the feature neural network satisfies the compression mapping condition.
[0036] Optionally, the iterative module includes:
[0037] The residual structure submodule is used to input the feature values and the initial hidden variables into the hidden layer representation neural network for iteration to obtain the precise feature values and the iterative hidden variables; wherein the feature neural network is a residual structure convolutional neural network.
[0038] To achieve the above objectives, the present invention also provides an image restoration device, comprising:
[0039] Memory, used to store computer programs;
[0040] A processor for executing the computer program and implementing any of the steps of the image restoration method described above.
[0041] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the steps of the image restoration method described above.
[0042] The present invention provides an image restoration method, comprising: receiving an image to be restored; inputting the image into a head neural network to map and obtain feature values; inputting the feature values and initialized latent variables into a hidden layer representation neural network for iteration to obtain precise feature values and iterative latent variables; wherein the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module; determining whether the feature neural network satisfies the compression mapping condition; if yes, stopping the iteration and inputting the corresponding precise feature values into a tail neural network to map and obtain the restored image; if no, continuing the iteration until it is determined that the feature neural network satisfies the compression mapping condition.
[0043] As can be seen, the method of the present invention reduces the number of iterations of fixed-point mapping by introducing the Anderson acceleration module, thereby improving the convergence speed of deep fixed-point mapping. This avoids the problem in existing related technologies where a large number of iterations are required on the fixed-point mapping function parameterized by deep neural networks during the iterative approximation of fixed points, which is generally on the order of hundreds or thousands, resulting in slow inference speed. The method reduces the number of iterations and improves the efficiency of image restoration. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 A flowchart of an image restoration method provided in an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram of a depth fixed-point network structure provided in an embodiment of the present invention;
[0047] Figure 3 A pseudocode diagram of depth fixed-point iteration provided in an embodiment of the present invention;
[0048] Figure 4 A pseudocode diagram of depth fixed-point iteration based on the Anderson acceleration module is provided for an embodiment of the present invention;
[0049] Figure 5 A schematic diagram of an Anderson acceleration module provided in an embodiment of the present invention;
[0050] Figure 6 A specific embodiment of an image restoration method provided by the present invention is illustrated in the figure.
[0051] Figure 7 This is a structural block diagram of an image restoration device provided in an embodiment of the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] Image restoration refers to the process of reconstructing a high-quality output image from a low-quality input image using restoration methods, and it is one of the fundamental problems in computer vision. The image restoration problem is ill-posed because many feasible high-quality restoration results exist. To effectively address the ill-posed nature of image restoration, traditional restoration methods employ various image prior models designed by experts, such as sparse representation, non-local self-similarity, and kernel norm. With the rapid development of deep neural networks and their breakthroughs in multiple fields of artificial intelligence, image restoration methods based on deep neural networks have also been extensively studied.
[0054] Image restoration methods based on deep neural networks directly parameterize the mapping between low-quality input images and high-quality original images. While this improves the quality of restored images compared to traditional methods, it encounters bottlenecks in effectively utilizing deeper networks. To increase the depth of neural networks and thus improve the quality of restored images, implicit deep neural network methods based on deep fixed-point methods have received widespread attention and research. In deep fixed-point methods, the fixed-point mapping function is parameterized by the deep neural network. By iterating this mapping function repeatedly—that is, using the output of the deep fixed point as input again—the equivalent depth of the deep neural network can be effectively increased without increasing the number of network layers or parameters, thereby reaching a feasible fixed point that is beneficial for improving the quality of restored images. Although deep fixed-point methods achieve implicitly infinitely deep neural networks with limited memory consumption, the iterative approximation of the fixed point requires a large number of iterations of the compression mapping function parameterized by the deep neural network, resulting in slow convergence speed.
[0055] This invention utilizes the Anderson acceleration method, which estimates and corrects the next state using m intermediate states preceding the current state in the fixed-point iteration process, thereby effectively reducing the number of fixed-point iterations. Inspired by this acceleration method, we designed a specialized neural network structure to parameterize the Anderson acceleration process, combining it with the deep fixed-point method to further reduce the number of iterations and improve the convergence speed of the deep fixed-point method.
[0056] The following combination Figure 1 , Figure 1 A flowchart of an image restoration method provided in an embodiment of the present invention, the method may include:
[0057] S101: Receive the image to be recovered, input the image into the head neural network to obtain feature values.
[0058] It is understood that the received image to be recovered is input into the head neural network of a deep fixed-point network. The feature values of the image are obtained through mapping by the head neural network. This embodiment does not limit the specific type of the head neural network; generally, a convolutional neural network can be used as the head neural network to extract the feature values of the image to be recovered. The structure of the deep fixed-point network can include a head neural network, a hidden layer representation neural network, and a tail neural network, specifically as follows: Figure 2 As shown.
[0059] S102: Input the feature values and initialized latent variables into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module.
[0060] It is understood that in this embodiment, the extracted feature values and initialized latent variables are input into the hidden layer neural network for iteration. Each iteration yields the precise feature value and the iterative latent variable for the current step. These two values are then used as input for the next iteration to obtain the precise feature value and latent variable after one iteration. The hidden layer represents the neural network, which includes a feature neural network and an Anderson acceleration module. Furthermore, this embodiment does not limit the specific type of the feature neural network; it can generally be a residual structure convolutional neural network. The method proposed in this embodiment can be applied to situations where the feature neural network is a residual structure convolutional neural network, as well as situations where the feature neural network is another type of neural network.
[0061] Fixed-point iteration is an important method for finding approximate roots of single-variable linear equations. It first determines an approximate initial value near the root of the equation, and then uses a successive approximation method, continuously updating this initial value using an iterative formula to make it constantly approach the accurate value. This embodiment can be based on a deep fixed-point method for image restoration. A deep neural network parameterizes the fixed-point mapping function and iterates over this function to obtain accurate feature values that satisfy the compression mapping conditions. The pseudocode for this process can be as follows: Figure 3 As shown, x0 represents the initial feature value obtained by mapping the image to be restored through the head neural network, where x t and x t+1 Let f represent the exact eigenvalues at the current step and the exact eigenvalues after the first iteration at the current step, respectively. θ This represents the fixed-point mapping function f(·) as a feature neural network with θ as a parameter, ε being a number greater than zero and infinitely close to zero, and T being the value of f. θ Given a limited number of iterations, continuously iterate f θ until the compression mapping condition, i.e., x, is satisfied. t+1 =f θ (x t When the fixed-point iteration process converges, the iteration stops, and x is output. * The precise feature values that finally converge during the image restoration process are obtained, i.e., the corresponding precise feature values.
[0062] The deep fixed-point method requires numerous iterations of the compression mapping function parameterized by the deep neural network during the iterative approximation of the fixed point, resulting in slow inference speed. This embodiment can introduce the Anderson acceleration method, which utilizes a preset number m intermediate states before the current step t in the fixed-point iteration process. For the next state x t+1 Estimation and correction are performed to effectively reduce the number of fixed-point iterations. This embodiment does not limit the specific size of the preset number m, which can be set according to the actual application scenario.
[0063] Anderson's acceleration method uses the iteration point x at step t+1. t+1 Defined as a linear combination of the function values of the intermediate state points of a pre-set quantity m, where Let g(x) be the combination coefficient, such that g(x) t )=x t -f θ (x t The L2 norm of ) is the smallest, that is and pass The constraints make the solution obtained by the least squares method... Then the new iteration point x at step t+1 t+1 for Although Anderson's acceleration method increases the computational cost for calculating combination coefficients and the storage cost for storing the function values of the previous m steps, it can reduce the number of iterations from hundreds or thousands to tens or hundreds.
[0064] Furthermore, feature neural networks with a large number of parameters lead to increased computational load and memory consumption. To simplify and speed up the computation, this embodiment can use the residual from the first preset number of m iterations. Using the latent variable h t In this case, Anderson acceleration can be formally represented as x t+1 ,h t+1 =G(x) t ,h t ), where G(·,·) represents the Anderson acceleration mapping function. Through the latent variable h t The original calculation, which relied on a preset number of m intermediate states, is now changed to only involve the previous h. t-1 Relevant. Latent variable h t In other words, by integrating the previous intermediate states, this embodiment solves the problems of coupling, complex dependencies and computation during gradient backpropagation by introducing latent variables.
[0065] The pseudocode for the depth fixed-point iteration process based on the Anderson acceleration module can be seen as follows: Figure 4 As shown, x0 represents the initial feature value obtained by mapping the image to be restored through the head neural network, where x t and x t+1 Let x' represent the exact eigenvalue of the current step and the exact eigenvalue after the first iteration of the current step, respectively. t+1 f represents the estimated feature value after one iteration in the current step. θ It represents the fixed-point mapping function f(·) as a feature neural network with θ as a parameter, and G(·,·) as a feature neural network with μ as a parameter.μ ε is a number greater than zero and infinitely close to zero, and T is f. θ The limited number of iterations, h t h0 represents the residual after the first preset number of m iterations, and h0 is the initialization hidden variable.
[0066] This embodiment does not limit the implementation of the Anderson acceleration module. A typical schematic diagram of the Anderson acceleration module can be seen as follows: Figure 5 As shown, G μ For the parameterized representation of G(·,·), the initial value of h0 can be set to 0, and the current state x t , x' t+1 =f θ (x t ) and latent variable h t As input to the Anderson acceleration module.
[0067] First, in the hidden layer representation neural network, the first weighting coefficient, the second weighting coefficient, and the residual are determined. During the first iteration, the mapped image feature value x0, the initialized hidden variable h0, the first weighting coefficient, the second weighting coefficient, and the residual are input into the first model in the hidden layer representation neural network to iterate and obtain the next precise feature value and iterative hidden variable. Then, the precise feature value and iterative hidden variable output from each iteration are used as the input for the next iteration. The mathematical expression of the first model is:
[0068]
[0069] In the computational flow of the Anderson acceleration module, the residuals are calculated using the input feature values through the second model. Then, the residuals and latent variables are used to calculate the second weighting coefficients through the third model. Finally, the second weighting coefficients are used to calculate the first weighting coefficients through the fourth model. In this embodiment, the residuals can be determined based on the precise feature values using the second model in the hidden layer representation neural network; the mathematical expression of the second model is:
[0070] resi = conv(f θ (x t )-x t )
[0071] Furthermore, the exact feature value input when calculating the residual for the first time can be x0, that is, x0 is equivalent to the exact feature value of the zeroth step.
[0072] In this embodiment, the second weighting coefficient is calculated from the residuals and iterative latent variables. The second weighting coefficient can be determined using the third model in the hidden layer representation neural network based on the residuals and iterative latent variables; the mathematical expression of the third model is:
[0073] z h =sigmoid(conv(resi)+conv(h) t ))
[0074] Furthermore, the iterative latent variable input when the second weighting coefficient is first calculated can be h0, that is, h0 is equivalent to the iterative latent variable in the zeroth step.
[0075] In this embodiment, the first weighting coefficient is calculated from the second weighting coefficient, and the first weighting coefficient can be determined based on the second weighting coefficient using the fourth model in the hidden layer representation neural network; wherein, the mathematical expression of the fourth model is:
[0076] z x =sigmoid(conv(z h ))
[0077] Figure 5 In this context, conv, sgm, and norm correspond to conv, sigmoid, and norm in the mathematical expressions of the aforementioned models, and respectively represent the convolution operator, sigmoid nonlinear activation function, and layer normalization operation in deep neural networks. Where z h and z x The role of weighting coefficients is to adjust the current state x. t 、x' t+1 =f θ (x t ) and latent variable h t The weighted summation yields new, precise eigenvalues x. t+1 and the new hidden variable h t+1 This embodiment does not limit the method for obtaining the weighting coefficients. They can be obtained through neural network calculation or through the least squares method. In this embodiment, deep neural network calculation is generally chosen to improve computational efficiency.
[0078] S103: Determine whether the feature neural network satisfies the compression mapping condition.
[0079] S104: If yes, stop the iteration and input the corresponding precise feature values into the tail neural network to obtain the restored image.
[0080] S105: If not, continue iterating until the feature neural network satisfies the compression mapping condition.
[0081] Understandably, during the iterative process, it is determined whether the feature neural network satisfies the compression mapping condition. If it does, the iteration stops, and the corresponding precise feature value output in the final iteration is the final converged precise feature value. Inputting this precise feature value into the tail neural network for mapping yields the restored image. If the condition is not met, iteration continues until the feature neural network satisfies the compression mapping condition. This embodiment does not limit the specific type of the tail neural network; it can generally be a convolutional neural network.
[0082] To achieve higher image restoration performance, this embodiment trains the deep fixed-point network. This embodiment does not limit the specific training method; generally, multiple low-quality input images are fed into the network to obtain multiple output images. The loss functions of the output images and their corresponding real images are calculated, and the average of these loss functions is used to construct a batch of loss function data to obtain the gradient. Then, the parameters of the head neural network, hidden layer representation neural network, and tail neural network in the deep fixed-point network are updated using backpropagation to achieve the training objective. Furthermore, since the feature neural network does not always satisfy the compression mapping condition, a special training method is required during the training of the deep fixed-point iterative process. Gradient clipping can generally be used to ensure the Lipshitz property. Furthermore, this embodiment does not limit the size of T. When using the deep fixed-point iterative method for image restoration problems, T is generally relatively large to meet the requirement of a small ε, such as tens or even hundreds of iterations.
[0083] The method of this invention reduces the number of iterations for fixed-point mapping by introducing the Anderson acceleration module, thereby improving the convergence speed of deep fixed-point mapping. This avoids the problem in existing related technologies where a large number of iterations are required for the fixed-point mapping function parameterized by deep neural networks during the iterative approximation of fixed points, typically on the order of hundreds or thousands, which leads to slow inference speed. This method reduces the number of iterations and improves the efficiency of image restoration.
[0084] The following combination Figure 6 , Figure 6 This is a specific embodiment of an image restoration method provided by the present invention. This specific embodiment may include:
[0085] 1. Input the low-quality image to be restored into the head neural network, and obtain the feature value of the image through mapping.
[0086] 2. Input the feature value and the initialized hidden variable into the hidden layer of the neural network for iteration to obtain the accurate feature value and iterated hidden variable after iteration.
[0087] 3. After each iteration, determine whether the feature neural network satisfies the compression mapping condition.
[0088] 4. If the condition is met, stop the iteration, input the corresponding precise feature value output at this time into the tail neural network, and obtain the restored image after mapping.
[0089] 5. If not satisfied, continue iterating until the feature neural network satisfies the compression mapping condition.
[0090] The following describes the image restoration apparatus, device, and storage medium provided in the embodiments of the present invention. The image restoration apparatus, device, and storage medium described below can be referred to in correspondence with the image restoration method described above.
[0091] The following combination Figure 7 , Figure 7 This is a structural block diagram of an image restoration apparatus provided in an embodiment of the present invention. The apparatus may include:
[0092] The receiving module 100 is used to receive the image to be recovered and input the image into the head neural network to obtain feature values;
[0093] The iteration module 200 is used to input the feature values and initialized latent variables into the hidden layer representation neural network for iteration to obtain the accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module;
[0094] The judgment module 300 is used to determine whether the feature neural network satisfies the compression mapping condition;
[0095] Execution module 400 is used to stop the iteration if the condition is met, and input the corresponding precise feature values into the tail neural network to obtain the restored image;
[0096] Loop module 500 is used to continue iterating if not, until it is determined that the deep neural network satisfies the compression mapping condition.
[0097] Based on the above embodiments, the method of the present invention introduces the Anderson acceleration module to reduce the number of iterations of fixed-point mapping, improve the convergence speed of deep fixed-point mapping, and avoid the problem in existing related technologies that require a large number of iterations of the fixed-point mapping function parameterized by deep neural networks during the iterative approximation of fixed points, which is generally on the order of hundreds or thousands, resulting in slow inference speed. This reduces the number of iterations and improves the efficiency of image restoration.
[0098] Based on the above embodiments, the iteration module 200 may include:
[0099] The residual structure submodule is used to input the feature values and initialized latent variables into the hidden layer representation neural network for iterative processing to obtain the accurate feature values and iterative latent variables; the hidden layer neural network is a residual structure convolutional neural network.
[0100] Based on the above embodiments, the iteration module 200 may include:
[0101] The first model submodule is used to determine the first weighting coefficient, the second weighting coefficient, and the residual in the hidden layer representation neural network.
[0102] The precise feature values and iterative latent variables are obtained by iteratively inputting the feature values, initial latent variables, first weighting coefficients, second weighting coefficients, and residuals into the first model of the neural network representing the hidden layer; wherein, the mathematical expression of the first model is:
[0103]
[0104] In the formula, norm is the normalization operation of the deep neural network layer, conv is the convolution operator of the deep neural network, and z x z is the first weighting coefficient. h The second weighting coefficient is denoted by , resi is the residual, and f is the weighting factor. θ For a feature neural network, x t x is the precise feature value of the current step. t+1 h represents the precise feature value after one iteration in the current step. t h is the latent variable for the current iteration step. t+1 This is the iterative hidden variable after the current step has been iterated once.
[0105] Based on the above embodiments, the first model submodule may include:
[0106] The second model unit is used to determine the residuals based on precise feature values using the second model in the hidden layer representation neural network; the mathematical expression of the second model is:
[0107] resi = conv(f θ (x t )-x t )
[0108] In the formula, conv is the convolution operator of a deep neural network, and f θ For a feature neural network, x t This is the precise feature value for the current step.
[0109] Based on the above embodiments, the second model unit may include:
[0110] The third model subunit is used to determine the second weighting coefficients based on the residuals and iterative latent variables using the third model in the hidden layer representation neural network; the mathematical expression for the fourth model is:
[0111] z h =sigmoid(conv(resi)+conv(h) t))
[0112] In the formula, sigmoid is the nonlinear activation function of the deep neural network, conv is the convolution operator of the deep neural network, resi is the residual, and h t This is the hidden variable for the current iteration step.
[0113] Based on the above embodiments, the third model subunit may include:
[0114] The fourth model is derived from a unit used to determine the first weighting coefficients based on the second weighting coefficients using the third model in the hidden layer representation neural network; wherein, the mathematical expression of the third model is:
[0115] z x =sigmoid(conv(z h ))
[0116] In the formula, z h is the second weighting coefficient, sigmoid is the non-linear activation function of the deep neural network, and conv is the convolution operator of the deep neural network.
[0117] Based on the above embodiments, the present invention also provides an image restoration device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the device may also include various necessary network interfaces, a power supply, and other components.
[0118] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an execution terminal or processor, can implement the image restoration method provided in the embodiments of the present invention; the storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0121] The above provides a detailed description of the image restoration method, apparatus, device, and readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. An image restoration method, characterized in that, include: The image to be recovered is received, and the image is input into the head neural network of a deep fixed-point network to obtain feature values; wherein, the deep fixed-point network includes: a head neural network, a hidden layer representation neural network, and a tail neural network; The feature values and initialized latent variables are input into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module; Determine whether the feature neural network satisfies the compression mapping condition; If so, stop the iteration and input the corresponding precise feature values into the tail neural network to obtain the restored image; If not, continue iterating until it is determined that the feature neural network satisfies the compression mapping condition; Specifically, the process of inputting the feature values and initialized latent variables into the hidden layer representation neural network for iteration to obtain accurate feature values and iterative latent variables includes: In the hidden layer representation neural network, the first weighting coefficient is determined, the second weighting coefficient is determined, and the residual is determined; The feature values, the initialized latent variables, the first weighting coefficient, the second weighting coefficient, and the residual are input into the first model in the hidden layer representation neural network and iterated to obtain the precise feature values and the iterative latent variables; wherein, the mathematical expression of the first model is: ； In the formula, This is a normalization operation for deep neural network layers. For deep neural network convolution operators, This is the first weighting coefficient. This is the second weighting coefficient. For the residual, For the feature neural network, For the precise feature value of the current step, These are the precise feature values after one iteration in the current step. For the hidden variables of the current iteration step, This is the implicit variable for the iteration after the current step has been iterated once; The determination of residuals includes: The residual is determined based on the precise feature values using a second model in the hidden layer representation neural network; wherein the mathematical expression of the second model is: ； In the formula, For the deep neural network convolution operator, For the feature neural network, The precise feature value of the current step; Determining the second weighting coefficient includes: The second weighting coefficients are determined using a third model in the hidden layer representation neural network based on the residuals and the iterative hidden variables; wherein the mathematical expression of the third model is: ； In the formula, For nonlinear activation functions in deep neural networks, For the deep neural network convolution operator, For the residual, The current step iteration hidden variable; Determining the first weighting coefficient includes: The first weighting coefficient is determined based on the second weighting coefficient using the fourth model in the hidden layer representation neural network; wherein, the mathematical expression of the fourth model is: ； In the formula, This is the second weighting coefficient. The nonlinear activation function of the deep neural network is... This refers to the convolution operator of the deep neural network.
2. The image restoration method according to claim 1, characterized in that, The process involves inputting the feature values and initialized latent variables into a hidden layer representation neural network for iterative processing to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module, comprising: The feature values and the initial hidden variables are input into the hidden layer representation neural network for iteration to obtain the precise feature values and the iterative hidden variables; wherein the feature neural network is a residual structure convolutional neural network.
3. An image restoration device, characterized in that, include: A receiving module is used to receive the image to be recovered and input the image into the head neural network of a deep fixed-point network to obtain feature values; wherein, the deep fixed-point network includes: a head neural network, a hidden layer representation neural network, and a tail neural network; An iterative module is used to input the feature values and initialized latent variables into a hidden layer representation neural network for iterative processing to obtain accurate feature values and iterative latent variables; wherein, the hidden layer representation neural network includes a feature neural network and an Anderson acceleration module; The judgment module is used to determine whether the feature neural network satisfies the compression mapping condition; The execution module is used to stop the iteration if the condition is met, and input the corresponding precise feature values into the tail neural network to obtain the restored image. The loop module is used to continue iterating if not, until it is determined that the feature neural network satisfies the compression mapping condition; The iterative module includes: The first model submodule is used to determine the first weighting coefficient, the second weighting coefficient, and the residual in the hidden layer representation neural network. The feature values, the initialized latent variables, the first weighting coefficient, the second weighting coefficient, and the residual are input into the first model in the hidden layer representation neural network and iterated to obtain the precise feature values and the iterative latent variables; wherein, the mathematical expression of the first model is: ； In the formula, This is a normalization operation for deep neural network layers. For deep neural network convolution operators, This is the first weighting coefficient. This is the second weighting coefficient. For the residual, For the feature neural network, For the precise feature value of the current step, These are the precise feature values after one iteration in the current step. For the hidden variables of the current iteration step, This is the implicit variable for the iteration after the current step has been iterated once; The first model submodule includes: The second model unit is used to determine the residual based on the precise feature values using the second model in the hidden layer representation neural network; wherein the mathematical expression of the second model is: ； In the formula, For the deep neural network convolution operator, For the feature neural network, The precise feature value of the current step; The second model unit includes: The third model subunit is used to determine the second weighting coefficients based on the residuals and iterative latent variables using the third model in the hidden layer representation neural network; the mathematical expression of the third model is: ； In the formula, For nonlinear activation functions in deep neural networks, For the deep neural network convolution operator, For the residual, The current step iteration hidden variable; The third model subunit includes: The fourth model is derived from a unit used to determine the first weighting coefficients based on the second weighting coefficients within the hidden layer representation neural network; the mathematical expression for the fourth model is: ； In the formula, This is the second weighting coefficient. The nonlinear activation function of the deep neural network is... This refers to the convolution operator of the deep neural network.
4. The apparatus according to claim 3, characterized in that, The iterative module includes: The residual structure submodule is used to input the feature values and the initial hidden variables into the hidden layer representation neural network for iteration to obtain the precise feature values and the iterative hidden variables; wherein the feature neural network is a residual structure convolutional neural network.
5. An image restoration device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program and implementing the steps of the image restoration method as described in any one of claims 1 to 2.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image restoration method as described in any one of claims 1 to 2.