A distortion image reconstruction method based on deep learning and related device

Abstract

The application discloses a distortion image reconstruction method based on deep learning and a related device, and the method comprises the following steps: constructing a distortion image reconstruction training set according to a high-definition image set, wherein the distortion image reconstruction training set comprises preset distortion images and corresponding preset image reconstruction matrices; performing network layer gradient iterative training on an initial DFISTA network model by using the distortion image reconstruction training set to obtain a preset DFISTA network model, wherein the initial DFISTA network model is constructed according to a FISTA algorithm and a deep neural network; and performing image reconstruction on a current distortion image by using the preset DFISTA network model to obtain a distortion image reconstruction matrix. The model is trained in a gradient iterative mode, which can not only ensure the accuracy and reliability of the model, but also accelerate the convergence speed of the model. Therefore, the application can solve the technical problems that the prior art is either too complex or lacks accuracy, resulting in poor actual distortion image processing effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a distortion image reconstruction method based on deep learning and related device. BACKGROUND

[0002] The development trend of the network is from virtual to reality, so video call is becoming one of the most commonly used communication means in people's daily work and life. Through video call, communication and expression can be more real, and the intimacy of communication is also enhanced. However, due to network reasons or call device lag, video distortion problem may occur, affecting the experience of video call. Video is composed of multiple images with gradual change rules. When the number of images changing per second exceeds 25, the human eye cannot distinguish each image, and what is seen is a video. Therefore, the essence of video distortion is image distortion. The existing distortion image processing technology is either too complex and causes large calculation amount, or cannot ensure the accuracy of reconstruction, resulting in poor actual processing effect. SUMMARY

[0003] The present application provides a distortion image reconstruction method based on deep learning and related device, which is used to solve the technical problem that the existing technology is either too complex or lacks accuracy, resulting in poor actual distortion image processing effect.

[0004] Therefore, the first aspect of the present application provides a distortion image reconstruction method based on deep learning, comprising:

[0005] constructing a distortion image reconstruction training set from a high-definition image set, the distortion image reconstruction training set comprising preset distortion images and corresponding preset image reconstruction matrices;

[0006] performing network layer gradient iterative training on an initial DFISTA network model using the distortion image reconstruction training set to obtain a preset DFISTA network model, the initial DFISTA network model being constructed according to a FISTA algorithm and a deep neural network;

[0007] performing image reconstruction on a current distortion image through the preset DFISTA network model to obtain a distortion image reconstruction matrix.

[0008] Preferably, the distortion image reconstruction training set is constructed from a high-definition image set, and the distortion image reconstruction training set comprises preset distortion images and corresponding preset image reconstruction matrices, comprising:

[0009] performing distortion processing and matrix expression operations on the high-definition image set respectively to obtain the preset distortion images and the corresponding preset image reconstruction matrices;

[0010] The preset distorted image is taken as training input data, and the preset image reconstruction matrix is taken as training output actual label to obtain a distorted image reconstruction training set.

[0011] Preferably, the distorted image reconstruction training set is constructed according to the high-definition image set, and the distorted image reconstruction training set includes the preset distorted image and the corresponding preset image reconstruction matrix, and further includes:

[0012] The clear images in multiple scenes are collected by using a preset high-definition device to obtain an initial clear image set.

[0013] The images in the initial clear image set are subjected to optimized preprocessing to obtain a high-definition image set.

[0014] Preferably, the distorted image reconstruction training set is used to perform network layer gradient iterative training on an initial DFISTA network model to obtain a preset DFISTA network model, and the initial DFISTA network model is constructed according to a FISTA algorithm and a deep neural network.

[0015] The initial DFISTA network model is constructed according to the FISTA algorithm and the deep neural network, and the initial DFISTA network model includes an initial gradient descent step.

[0016] The distorted image reconstruction training set is used to perform network layer gradient iterative training on the initial DFISTA network model, and an intermediate reconstruction result vector is calculated.

[0017] If the intermediate reconstruction result vector meets a preset reconstruction requirement, the training is stopped, and the preset DFISTA network model is obtained.

[0018] The second aspect of the application provides a distorted image reconstruction device based on deep learning, which includes:

[0019] A data construction unit is configured to construct a distorted image reconstruction training set according to a high-definition image set, and the distorted image reconstruction training set includes a preset distorted image and a corresponding preset image reconstruction matrix.

[0020] A model training unit is configured to use the distorted image reconstruction training set to perform network layer gradient iterative training on an initial DFISTA network model to obtain a preset DFISTA network model, and the initial DFISTA network model is constructed according to a FISTA algorithm and a deep neural network.

[0021] An image reconstruction unit is configured to perform image reconstruction on a current distorted image by using the preset DFISTA network model to obtain a distorted image reconstruction matrix.

[0022] Preferably, the data construction unit is specifically configured to:

[0023] The high-definition image set is respectively subjected to distortion processing and matrix expression operation to obtain a preset distorted image and a corresponding preset image reconstruction matrix;

[0024] The preset distorted image is taken as training input data, and the preset image reconstruction matrix is taken as training output actual label to obtain a distorted image reconstruction training set.

[0025] Preferably, the method further comprises:

[0026] An image acquisition unit is configured to acquire clear images in multiple scenes by using a preset high-definition device to obtain an initial clear image set;

[0027] An image processing unit is configured to perform optimized preprocessing on images in the initial clear image set to obtain a high-definition image set.

[0028] Preferably, the model training unit is specifically configured to:

[0029] An initial DFISTA network model is constructed according to the FISTA algorithm and a deep neural network, and the initial DFISTA network model includes an initial gradient descent step;

[0030] The initial DFISTA network model is subjected to network layer gradient iterative training by using the distorted image reconstruction training set, and an intermediate reconstruction result vector is calculated;

[0031] If the intermediate reconstruction result vector meets a preset reconstruction requirement, the training is stopped to obtain a preset DFISTA network model.

[0032] The third aspect of the application provides a distorted image reconstruction device based on deep learning, and the device includes a processor and a memory;

[0033] The memory is configured to store program code and transmit the program code to the processor;

[0034] The processor is configured to execute the method for reconstructing a distorted image based on deep learning according to instructions in the program code.

[0035] The fourth aspect of the application provides a computer readable storage medium configured to store program code, and the program code is configured to execute the method for reconstructing a distorted image based on deep learning.

[0036] As can be seen from the above technical solutions, the embodiments of the application have the following advantages:

[0037] In the present application, a distortion image reconstruction method based on deep learning is provided, which comprises: constructing a distortion image reconstruction training set according to a high-definition image set, the distortion image reconstruction training set comprising preset distortion images and corresponding preset image reconstruction matrices; performing network layer gradient iterative training on an initial DFISTA network model using the distortion image reconstruction training set to obtain a preset DFISTA network model, the initial DFISTA network model being constructed according to a FISTA algorithm and a deep neural network; and performing image reconstruction on a current distortion image through the preset DFISTA network model to obtain a distortion image reconstruction matrix.

[0038] The distortion image reconstruction method based on deep learning provided by the present application can simplify the algorithm scheme and improve the reconstruction efficiency by constructing a preset DFISTA network model based on a FISTA algorithm and a deep neural network for end-to-end distortion image reconstruction. Moreover, the FISTA algorithm can improve the calculation speed by means of the nonlinear calculation mode of the neural network, and the FISTA algorithm can be expanded in the form of a neural network layer to realize gradient iterative operation, which not only ensures the accuracy and reliability of the model, but also speeds up the convergence speed of the model. Therefore, the present application can solve the technical problems that the prior art is either too complex or lacks accuracy, resulting in poor actual distortion image processing effect. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of a distortion image reconstruction method based on deep learning provided by an embodiment of the present application is shown in the figure.

[0040] Figure 2 A structural diagram of a distortion image reconstruction device based on deep learning provided by an embodiment of the present application is shown in the figure.

[0041] Figure 3 A structural diagram of a preset DFISTA network model provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0042] In order to enable personnel in the technical field to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0043] For the sake of understanding, please refer to Figure 1 An embodiment of a distortion image reconstruction method based on deep learning provided by the present application comprises:

[0044] Step 101, constructing a distorted image reconstruction training set according to the high-definition image set, the distorted image reconstruction training set including preset distorted images and corresponding preset image reconstruction matrices.

[0045] Further, step 101 includes:

[0046] The high-definition image set is respectively subjected to distortion processing and matrix expression operation to obtain the preset distorted images and the corresponding preset image reconstruction matrices.

[0047] The preset distorted images are taken as training input data, and the preset image reconstruction matrices are taken as training output actual labels to obtain the distorted image reconstruction training set.

[0048] The high-definition image set is collected by a specific device or method, and a large number of images are included in the high-definition image set in the embodiment, which facilitates model training. The images in the high-definition image set are directly subjected to distortion processing to obtain the preset distorted images. The distortion processing can be set according to actual conditions, for example, blurring processing.

[0049] Each high-definition image can be regarded as being composed of a plurality of pixels, that is, an image can be expressed as a matrix, and each pixel in the image corresponds to an element in the matrix. Therefore, the matrix expression is performed on each image in the high-definition image set to obtain the corresponding preset image reconstruction matrix H = [h1, h2,... h k ], where h k is a reconstruction vector, and the form is a sequence.

[0050] Since the preset distorted images and the preset image reconstruction matrices are both obtained based on the high-definition images, they correspond to each other, and the distorted image reconstruction training set composed of the preset distorted images and the preset image reconstruction matrices can be obtained. The training set includes input data X for inputting the model for training, that is, the preset distorted images, and also includes output actual labels Y for verification, that is, the preset image reconstruction matrices.

[0051] Further, step 101 further includes:

[0052] Collecting clear images in multiple scenes by using a preset high-definition device to obtain an initial clear image set;

[0053] Optimizing and preprocessing the images in the initial clear image set to obtain the high-definition image set.

[0054] The embodiment provides a method for obtaining a high-definition image set. The method includes collecting initial clear images by using a preset high-definition device to form an initial clear image set, and performing optimization preprocessing operation on some abnormal or not high-definition enough images in the image set to obtain the high-definition image set. The specific means of the preset high-definition device and the optimization preprocessing can be selected according to actual conditions, and are not limited herein.

[0055] Step 102, the initial DFISTA network model is trained by gradient iteration of network layer using the distorted image reconstruction training set, and a preset DFISTA network model is obtained.

[0056] Further, step 102 comprises:

[0057] The initial DFISTA network model is constructed according to the FISTA algorithm and the deep neural network, and the initial DFISTA network model comprises an initial gradient descent step size.

[0058] The initial DFISTA network model is trained by gradient iteration of network layer using the distorted image reconstruction training set, and an intermediate reconstruction result vector is calculated.

[0059] If the intermediate reconstruction result vector meets a preset reconstruction requirement, the training is stopped, and the preset DFISTA network model is obtained.

[0060] It should be noted that the FISTA algorithm is a Fast Iterative Shrinkage-Thresholding Algorithm, and the preset DFISTA (Deep Fast Iterative Shrinkage Thresholding) network model is an optimization algorithm based on the FISTA algorithm and the deep neural network.

[0061] Referring to Figure 3 , the function expression of the initial DFISTA network model for image reconstruction is:

[0062]

[0063] Wherein, A is an image reconstruction matrix of model reconstruction, which is a matrix composed of the intermediate reconstruction result vector in the training stage, is an estimated intermediate reconstruction vector, n is an additive white Gaussian noise, and due to the sparsity of the model, the estimation of can be converted into the recovery of a sparse signal:

[0064]

[0065] The above formula can also be equivalent to:

[0066]

[0067] Wherein, ε is an error, and h(X k-1 ) are a vector and a linear regression value, respectively, and h(X respectively represent the value at k-1 and gradient descent value, X k-1 represents the input matrix at k-1. t k is the gradient descent step, the initial gradient descent step is set to 1.

[0068] The above optimization problem is solved using a more general quadratic approximation model, at point y: H(x) = h(x) + g(x) is expressed as a quadratic approximation function:

[0069]

[0070]

[0071] where Q L (x,y), P L (y) respectively represent the second order approximation function and the minimum point, h(y) represents the derivative of h with respect to y, represents the gradient operator, L is the step size, g(x) is the hypothesis function, x, y are two-dimensional coordinate values in the function concept.

[0072] The FISTA algorithm completes iteration by calculating the position of y, and the specific solution equation is expressed as:

[0073] y1 = x0, t1 = 1

[0074] v t = y - Ax t

[0075]

[0076]

[0077]

[0078] where x0, y1, t1 are the initial model input data, the initial training intermediate reconstruction sequence and the initial gradient descent step, respectively, v t is the noise quantity at iteration step t, x k+1 , y k+1 respectively represent the input and output at k+1, x t is the input at t, η, β, λ are all related coefficients.

[0079] From the operation process, after obtaining the distorted image reconstruction training set, first set the initial gradient descent step, that is, 1; then calculate the approximate gradient descent vector according to the iterative operation; then calculate the reconstruction threshold vector in the image reconstruction process according to the approximate gradient descent vector; then the intermediate reconstruction result vector can be obtained according to the reconstruction threshold vector; if the intermediate reconstruction result vector meets the preset reconstruction requirement, it means that the model parameter at this time meets the training requirement, and the training is stopped to obtain the preset DFISTA network model; if the intermediate reconstruction result vector does not meet the preset reconstruction requirement, the iteration number is increased by 1, and the gradient descent iterative training is continued. It can be understood that the preset reconstruction requirement can be set according to the actual situation, which is not limited here.

[0080] Based on the deep learning idea, the FISTA algorithm is expanded into a deep neural network structure DFISTA, each iteration of the FISTA algorithm corresponds to each layer of the DFISTA network model, and the performance of the algorithm is improved by learning and optimizing related parameters in each layer of the network. Or, the traditional gradient descent algorithm can infinitely approach the final target result through continuous gradient descent, and the preset DFISTA network model in the embodiment obtains the final optimal solution by continuously increasing the number of network layers.

[0081] Step 103, reconstructing the current distorted image through the preset DFISTA network model to obtain a distorted image reconstruction matrix.

[0082] It can be understood that the current distorted image is the image that needs to be reconstructed, which can be a real-time acquired image or an image without a corresponding reconstruction matrix label. The image is directly input into the preset DFISTA network model to complete image reconstruction, and the corresponding distorted image reconstruction matrix is obtained to form a reconstructed image.

[0083] The distorted image reconstruction method based on deep learning provided by the embodiment of the application reconstructs the distorted image end to end based on the FISTA algorithm and the deep neural network to construct the preset DFISTA network model, which can simplify the algorithm scheme and improve the reconstruction efficiency. Moreover, the FISTA algorithm can improve the calculation speed by means of the nonlinear calculation method of the neural network. Expanding the FISTA algorithm in the form of a neural network layer and realizing gradient iteration operation can not only ensure the accuracy and reliability of the model, but also speed up the convergence speed of the model. Therefore, the embodiment of the application can solve the technical problems that the prior art is too complex or lacks accuracy, resulting in poor actual distorted image processing effect.

[0084] For ease of understanding, please refer to Figure 2 The application provides an embodiment of a distorted image reconstruction device based on deep learning, which comprises:

[0085] The data construction unit 201 is configured to construct a distorted image reconstruction training set according to the high-definition image set, the distorted image reconstruction training set including preset distorted images and corresponding preset image reconstruction matrices.

[0086] The model training unit 202 is configured to perform network layer gradient iterative training on an initial DFISTA network model by using the distorted image reconstruction training set to obtain a preset DFISTA network model, the initial DFISTA network model being constructed according to a FISTA algorithm and a deep neural network.

[0087] The image reconstruction unit 203 is configured to perform image reconstruction on a current distorted image by using the preset DFISTA network model to obtain a distorted image reconstruction matrix.

[0088] Further, the data construction unit 201 is specifically configured to:

[0089] perform distortion processing and matrix expression operations on the high-definition image set respectively to obtain the preset distorted images and the corresponding preset image reconstruction matrices;

[0090] use the preset distorted images as training input data and use the preset image reconstruction matrices as training output actual labels to obtain the distorted image reconstruction training set.

[0091] Further, the device further comprises:

[0092] The image acquisition unit 204 is configured to acquire clear images in multiple scenes by using a preset high-definition device to obtain an initial clear image set.

[0093] The image processing unit 205 is configured to perform optimized preprocessing on the images in the initial clear image set to obtain the high-definition image set.

[0094] Further, the model training unit 202 is specifically configured to:

[0095] construct the initial DFISTA network model according to the FISTA algorithm and the deep neural network, the initial DFISTA network model including an initial gradient descent step size;

[0096] perform network layer gradient iterative training on the initial DFISTA network model by using the distorted image reconstruction training set and calculate an intermediate reconstruction result vector;

[0097] if the intermediate reconstruction result vector meets a preset reconstruction requirement, stop the training to obtain the preset DFISTA network model.

[0098] The application also provides a distorted image reconstruction device based on deep learning, the device comprising a processor and a memory.

[0099] The memory is configured to store the program code and transmit the program code to the processor.

[0100] The processor is configured to execute the distortion image reconstruction method based on deep learning in the method embodiments according to the instructions in the program code.

[0101] The application further provides a computer readable storage medium configured to store program code, and the program code is configured to execute the distortion image reconstruction method based on deep learning in the method embodiments.

[0102] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic, and the division of the units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0103] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0105] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions for executing all or part of the steps of the method described in various embodiments of the present application by a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0106] The above-described and the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reconstructing distorted images based on deep learning, characterized in that, include: A training set for distorted image reconstruction is constructed based on a high-resolution image set. The training set includes preset distorted images and corresponding preset image reconstruction matrices. The specific process is as follows: The high-resolution image set is subjected to distortion processing and matrix representation operations to obtain a preset distorted image and the corresponding preset image reconstruction matrix; The preset distorted image is used as the training input data, and the preset image reconstruction matrix is ​​used as the training output actual label to obtain the distorted image reconstruction training set. The initial DFISTA network model is trained iteratively using a gradient-based method on the network layers using the distorted image reconstruction training set to obtain a preset DFISTA network model. The initial DFISTA network model is constructed based on the FISTA algorithm and a deep neural network, and the specific process is as follows: An initial DFISTA network model is constructed based on the FISTA algorithm and a deep neural network, wherein the initial DFISTA network model includes an initial gradient descent step size; The initial DFISTA network model is trained using the distorted image reconstruction training set, and intermediate reconstruction result vectors are calculated. If the intermediate reconstruction result vector meets the preset reconstruction requirements, then training is stopped and the preset DFISTA network model is obtained. The distorted image is reconstructed using the preset DFISTA network model to obtain the distorted image reconstruction matrix.

2. The deep learning-based distortion image reconstruction method according to claim 1, characterized in that, The step of constructing a distorted image reconstruction training set based on a high-resolution image set, wherein the distorted image reconstruction training set includes preset distorted images and corresponding preset image reconstruction matrices, and further includes: An initial set of clear images is obtained by acquiring clear images in various scenarios using pre-installed high-definition equipment; The images in the initial clear image set are optimized and preprocessed to obtain a high-definition image set.

3. A distortion image reconstruction device based on deep learning, characterized in that, include: The data construction unit is used to construct a distorted image reconstruction training set based on a high-resolution image set. The distorted image reconstruction training set includes preset distorted images and corresponding preset image reconstruction matrices. Specifically, the data construction unit is used for: The high-resolution image set is subjected to distortion processing and matrix representation operations to obtain a preset distorted image and the corresponding preset image reconstruction matrix; The preset distorted image is used as the training input data, and the preset image reconstruction matrix is ​​used as the training output actual label to obtain the distorted image reconstruction training set. The model training unit is used to perform gradient-based iterative training of the initial DFISTA network model using the distorted image reconstruction training set to obtain a preset DFISTA network model. The initial DFISTA network model is constructed based on the FISTA algorithm and a deep neural network. Specifically, the model training unit is used for: An initial DFISTA network model is constructed based on the FISTA algorithm and a deep neural network, wherein the initial DFISTA network model includes an initial gradient descent step size; The initial DFISTA network model is trained using the distorted image reconstruction training set, and intermediate reconstruction result vectors are calculated. If the intermediate reconstruction result vector meets the preset reconstruction requirements, then training is stopped and the preset DFISTA network model is obtained. The image reconstruction unit is used to reconstruct the current distorted image using the preset DFISTA network model to obtain the distorted image reconstruction matrix.

4. The deep learning-based distortion image reconstruction apparatus according to claim 3, characterized in that, Also includes: The image acquisition unit is used to acquire clear images of various scenes through a pre-set high-definition device to obtain an initial clear image set; An image processing unit is used to perform optimized preprocessing on the images in the initial clear image set to obtain a high-definition image set.

5. A distortion image reconstruction device based on deep learning, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the deep learning-based distortion image reconstruction method according to any one of claims 1-2 according to the instructions in the program code.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for performing the deep learning-based distortion image reconstruction method according to any one of claims 1-2.

Images