High-resolution image reconstruction method and apparatus, computer device and storage medium

Abstract

Embodiments of the present application relate to the technical field of image processing, and provide a high-resolution image reconstruction method and apparatus, a computer device and a storage medium. The method comprises: acquiring a low-resolution image, wherein the low-resolution image is obtained by reducing the resolution of a high-resolution target image; inputting the low-resolution image into an image reconstruction network of at least two stages; by means of the image reconstruction network, carrying out high-resolution reconstruction on the low-resolution image to obtain a preliminarily reconstructed image; on the basis of the preliminarily reconstructed image and the target image, carrying out parameter adjustment on the image reconstruction network of the next stage; by means of the adjusted image reconstruction network, carrying out high-resolution reconstruction on the low-resolution image to obtain an updated reconstructed image; and on the basis of the updated reconstructed image and the target image, carrying out parameter adjustment on the image reconstruction network of the further next stage until the image reconstruction network of the final stage outputs a high-resolution image. The embodiments of the present application can effectively improve the reconstruction effect of high-resolution images by means of prior information.

Description

High-resolution image reconstruction method and device, computer equipment and storage medium Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a high-resolution image reconstruction method and apparatus, computer equipment, and storage medium. Background Art

[0002] High-resolution image reconstruction is the process of restoring a low-resolution image to a high-resolution image, allowing for subsequent image processing. For example, in the field of magnetic resonance imaging of blood vessel walls, constructing high-resolution images facilitates vessel wall and lumen segmentation during image processing.

[0003] Related technologies for high-resolution image reconstruction primarily utilize deep learning methods to learn the mapping relationship between low-resolution magnetic resonance images and high-resolution images from large image datasets, thereby generating high-resolution images. However, deep learning methods fail to effectively utilize the underlying prior information of the image, making it difficult to accurately output high-resolution images. Therefore, how to effectively utilize prior information and accurately output high-resolution images has become a pressing technical challenge.

[0004] Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a high-resolution image reconstruction method and apparatus, computer equipment and storage medium, aiming to effectively utilize prior information to improve the reconstruction effect of high-resolution images.

[0006] To achieve the above objectives, a first aspect of an embodiment of the present application provides a high-resolution image reconstruction method, the method comprising:

[0007] Acquire a low-resolution image; wherein the low-resolution image is obtained by reducing the resolution of a preset target image, and the target image is a high-resolution image;

[0008] Inputting the low-resolution image into a preset image reconstruction model; wherein the image reconstruction model includes at least two stages of image reconstruction networks;

[0009] Reconstructing the low-resolution image with high resolution through the image reconstruction network to obtain a preliminary reconstructed image;

[0010] Adjusting parameters of the image reconstruction network in the next stage according to the preliminary reconstructed image and the target image;

[0011] Reconstructing the low-resolution image with high resolution using the adjusted image reconstruction network to obtain an updated reconstructed image;

[0012] The parameters of the image reconstruction network in the next stage are adjusted according to the updated reconstructed image and the target image, until the image reconstruction network in the final stage outputs a high-resolution image.

[0013] In some embodiments, before inputting the low-resolution image into a preset image reconstruction model, the method further includes:

[0014] Constructing the image reconstruction model specifically includes:

[0015] Get the original reconstructed model;

[0016] The original reconstruction model is transformed according to a preset half-splitting operator and preset auxiliary variables to obtain an equivalent reconstruction model;

[0017] Decoupling the equivalent reconstruction model to obtain a data module and a priori module;

[0018] The data module and the priori module are combined to obtain the image reconstruction model.

[0019] In some embodiments, performing high-resolution reconstruction on the low-resolution image by the image reconstruction network to obtain a preliminary reconstructed image includes:

[0020] Performing convolution processing on the low-resolution image through the data module to obtain a preliminary processed image;

[0021] The preliminary processed image is subjected to denoising processing by the priori module to obtain the preliminary reconstructed image.

[0022] In some embodiments, adjusting parameters of the image reconstruction network in the next stage according to the preliminary reconstructed image and the target image includes:

[0023] performing resolution loss calculation on the preliminary reconstructed image and the target image to obtain resolution loss data;

[0024] Parameters of the image reconstruction network in the next stage are adjusted according to the resolution loss data.

[0025] In some embodiments, adjusting parameters of the image reconstruction network in the next stage according to the resolution loss data includes:

[0026] Adjusting the first trade-off parameter of the data module in the next stage by using a preset hyperparameter module and the resolution loss data;

[0027] The second trade-off parameter of the priori module in the next stage is adjusted through the preset hyperparameter module and the resolution loss data.

[0028] In some embodiments, acquiring a low-resolution image includes:

[0029] Acquire the target image;

[0030] The target image is interpolated according to a preset number of interpolation times to obtain the low-resolution image.

[0031] In some embodiments, acquiring the target image includes:

[0032] Acquire an original image set; wherein the original image set includes original images of different sizes;

[0033] The original image is cropped according to a preset image size to obtain the target image.

[0034] To achieve the above-mentioned objectives, a second aspect of an embodiment of the present application provides a high-resolution image reconstruction device, comprising:

[0035] An image acquisition module, configured to acquire a low-resolution image; wherein the low-resolution image is obtained by reducing the resolution of a preset target image, and the target image is a high-resolution image;

[0036] An input module, configured to input the low-resolution image into a preset image reconstruction model; wherein the image reconstruction model comprises an image reconstruction network of at least two stages;

[0037] A first reconstruction module is configured to perform high-resolution reconstruction on the low-resolution image through the image reconstruction network to obtain a preliminary reconstructed image;

[0038] A first parameter adjustment module, configured to adjust parameters of the image reconstruction network in the next stage according to the preliminary reconstructed image and the target image;

[0039] a second reconstruction module, configured to perform high-resolution reconstruction on the preliminary reconstructed image using the adjusted image reconstruction network to obtain an updated reconstructed image;

[0040] The second parameter adjustment module is used to adjust the parameters of the image reconstruction network in the next stage according to the updated reconstructed image and the target image until the image reconstruction network in the final stage outputs a high-resolution image.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0042] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0043] The high-resolution image reconstruction method and apparatus, computer device, and storage medium proposed in this application construct an image reconstruction model comprising at least two stages of image reconstruction networks. After each image reconstruction network completes high-resolution image reconstruction, it adjusts the image reconstruction network of the next stage based on the reconstructed image and the original high-resolution image. This process repeatedly performs image reconstruction and network parameter adjustments, applying prior information to image reconstruction. Furthermore, the image reconstruction results of each stage influence the image reconstruction of the next stage, ensuring that the final stage outputs the optimal reconstructed image. Therefore, prior information is effectively used to guide image reconstruction, improving the reconstruction effect of high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a flow chart of a high-resolution image reconstruction method provided by an embodiment of the present application;

[0045] FIG2 is a flow chart of step S101 in FIG1 ;

[0046] FIG3 is a flow chart of step S201 in FIG2 ;

[0047] FIG4 is a flow chart of a high-resolution image reconstruction method provided by another embodiment of the present application;

[0048] FIG5 is a flow chart of a high-resolution image reconstruction method provided in an embodiment of the present application.

[0049] FIG6 is a flow chart of step S103 in FIG1 ;

[0050] FIG7 is a flow chart of step S104 in FIG1 ;

[0051] FIG8 is a flow chart of step S702 in FIG7 ;

[0052] FIG9 is a schematic structural diagram of a high-resolution image reconstruction device provided in an embodiment of the present application;

[0053] FIG10 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0057] First, let’s analyze some of the terms used in this application:

[0058] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0059] Deep learning algorithms are machine learning algorithms that mimic the structure and function of neural networks in the human brain. Based on a multi-layered neural network architecture, they learn and extract features by training on large amounts of data. Deep learning algorithms learn complex patterns and relationships through nonlinear transformations across multiple neural network layers, achieving remarkable results in many fields, such as image recognition, speech recognition, and natural language processing. Common deep learning algorithms include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs).

[0060] Super-resolution reconstruction refers to the process of converting a low-resolution image into a high-resolution image using an algorithm or model. This process can improve image clarity and detail. One of the most common super-resolution reconstruction methods is based on deep learning, which uses neural network models to learn the mapping from low-resolution images to high-resolution images. These models typically learn the complex relationship between low-resolution and high-resolution images through extensive training data, enabling them to predict the details and structures in the high-resolution image.

[0061] Magnetic Resonance Imaging (MRI): A medical imaging technique that uses strong magnetic fields and harmless radio waves to produce detailed images of the inside of the human body. It can provide information about body structures and tissues, helping doctors diagnose diseases and plan treatments.

[0062] Convolutional Neural Networks: A deep learning model commonly used for image and video processing tasks. They extract features through multiple convolutional and pooling layers, and perform classification or regression tasks through fully connected layers and output layers. CNNs have a wide range of applications in computer vision, including image recognition, object detection, and image segmentation.

[0063] Super-resolution reconstruction is the process of reconstructing low-resolution images into high-resolution images for subsequent image processing in areas requiring high image resolution. For example, in the field of magnetic resonance imaging, high-resolution images are difficult to obtain due to limitations in hardware conditions, signal-to-noise ratio, scanning time, and patient comfort. High-resolution magnetic resonance vessel wall images facilitate lumen and wall segmentation in image post-processing. By reconstructing three-dimensional magnetic resonance vessel wall images, the vascular bed where the wall lesions are located is located, and then the local area where the lesions are located is magnified to observe the morphological characteristics of the lesions from different angles. In addition, the image thickness of the vessel wall is at the sub-millimeter level. If high-resolution highlights of the vessel wall can be obtained, this will also facilitate lumen and wall segmentation in image post-processing, thereby facilitating subsequent quantitative analysis of morphological parameters.

[0064] Among related technologies, super-resolution reconstruction techniques based on deep learning, while showing potential in terms of image resolution, have some limitations, particularly a lack of physical interpretability and a neglect of prior image information. In addition, super-resolution reconstruction using deep learning models exists, but these models are often viewed as "black boxes," lacking intuitive physical explanations for their internal decision-making processes. This is particularly important in the field of medical imaging, where model decisions must be explained and validated.

[0065] Based on this, the embodiments of the present application provide a high-resolution image reconstruction method and apparatus, a computer device, and a storage medium. The method aims to construct an image reconstruction model comprising at least two stages of image reconstruction networks. After each image reconstruction network completes high-resolution image reconstruction, the image reconstruction network of the next stage is adjusted based on the reconstructed image and the original high-resolution image. This method repeatedly reconstructs and adjusts network parameters, applies prior information to image reconstruction, and the image reconstruction effect of each stage affects the image reconstruction of the next stage, so that the final stage outputs the best reconstructed image. Therefore, prior information is effectively used to guide image reconstruction, improving the reconstruction effect of high-resolution images.

[0066] The high-resolution image reconstruction method and apparatus, computer equipment, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the high-resolution image reconstruction method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0068] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] The high-resolution image reconstruction method provided in the embodiment of the present application relates to the field of image processing technology. The high-resolution image reconstruction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the high-resolution image reconstruction method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0071] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] FIG1 is an optional flowchart of a high-resolution image reconstruction method provided in an embodiment of the present application. The method in FIG1 may include but is not limited to steps S101 to S106 .

[0073] Step S101, obtaining a low-resolution image; wherein the low-resolution image is obtained by reducing the resolution of a preset target image, and the target image is a high-resolution image;

[0074] Step S102: inputting the low-resolution image into a preset image reconstruction model; wherein the image reconstruction model includes at least two stages of image reconstruction networks;

[0075] Step S103, reconstructing the low-resolution image into a high-resolution image through an image reconstruction network to obtain a preliminary reconstructed image;

[0076] Step S104, adjusting parameters of the image reconstruction network of the next stage according to the preliminary reconstructed image and the target image;

[0077] Step S105, reconstructing the low-resolution image into a high-resolution image using the adjusted image reconstruction network to obtain an updated reconstructed image;

[0078] Step S106: Adjust parameters of the image reconstruction network in the next stage according to the updated reconstructed image and the target image, until the image reconstruction network in the final stage outputs a high-resolution image.

[0079] In the embodiment of the present application, steps S101 to S106 are illustrated by setting up a multi-stage image reconstruction network, and after each image reconstruction network reconstructs a low-resolution image into a high-resolution preliminary reconstructed image, the image reconstruction network of the next stage is adjusted based on the reconstructed preliminary reconstructed image and the high-resolution target image. The updated reconstructed image and target image output by the image reconstruction network of the next stage continue to be used for the image reconstruction network of the next stage, until the image reconstruction network of the last stage outputs a high-resolution image. Therefore, when performing high-resolution reconstruction, the image reconstruction network of the next stage is adjusted based on the image output by the reconstruction network of the previous stage and the target image, effectively utilizing prior information to optimize the image reconstruction network, and intuitively feeling that the image reconstruction effect is better the later it is, so the high-resolution image with the best reconstruction effect is finally output, thereby improving the high-resolution reconstruction effect.

[0080] Please refer to FIG. 2 . In some embodiments, step S101 may include but is not limited to steps S201 to S202 :

[0081] Step S201, acquiring a target image;

[0082] Step S202 : performing interpolation processing on the target image according to a preset number of interpolation times to obtain a low-resolution image.

[0083] In step S201 of some embodiments, the target image is a high-resolution image. For example, in the field of magnetic resonance imaging, the target image may be a high-resolution magnetic resonance vessel wall image, which clearly shows the vessel wall and lumen. It should be noted that during the image reconstruction model training process, the target image is a high-resolution magnetic resonance vessel wall image; during the image reconstruction model usage process, the target image is a directly acquired magnetic resonance vessel wall image, which is a low-resolution magnetic resonance vessel wall image.

[0084] In step S202 of some embodiments, during the model training process, in order to train the image reconstruction model, the image reconstruction model inputs a low-resolution image. In order to save the number of image acquisitions, the high-resolution target image can be directly reduced in resolution. Therefore, a low-resolution image is obtained by performing difference processing on the target image according to a preset number of interpolation operations. It should be noted that image interpolation is to create new pixel values ​​between pixels, so this embodiment obtains a low-resolution image by reducing the pixel values ​​of the target image. For example, if the preset number of interpolation operations is 3 and the target image has a resolution of 256*256, a low-resolution image is obtained by performing three interpolation operations on the target image.

[0085] In steps S201 to S202 shown in this embodiment, by acquiring a high-resolution target image and then performing interpolation processing on the target image to obtain a low-resolution image, there is no need to acquire a low-resolution image to train the image reconstruction model, which can reduce the amount of data collected.

[0086] Please refer to FIG3 . In some embodiments, step S201 may include but is not limited to steps S301 to S302 :

[0087] Step S301, obtaining an original image set; wherein the original image set includes original images of different sizes;

[0088] Step S302 : cropping the original image according to a preset image size to obtain a target image.

[0089] In step S301 of some embodiments, multiple original images are combined into an original image set, and at least two original images of different sizes are present. It should be noted that if the original images are magnetic resonance vessel wall images, obtained from 234 patients with atherosclerosis using a T1-weighted 3DMATRIX sequence on a 3T whole-body magnetic resonance system, the 3D magnetic resonance vessel wall images need to be sliced ​​into 2D magnetic resonance vessel wall images because they are initially acquired. In this embodiment, 3,600 2D magnetic resonance vessel wall images are used.

[0090] In some embodiments, in step S302, because the MRI vessel wall images vary in size, i.e., the matrices corresponding to the MRI vessel wall images vary in size, the MRI vessel wall images are cropped into target images of the same size based on a preset image size. It should be noted that the resolution of the cropped target images is 256*256.

[0091] In steps S301 to S302 shown in this embodiment, original images of different sizes are cropped into target images of uniform size, so that subsequent use of the target images to train the image reconstruction model is simplified, thereby achieving better training effects of the image reconstruction model.

[0092] In some embodiments, before inputting the low-resolution image into the image reconstruction model, it is necessary to first construct the image reconstruction model, that is, to construct a reconstruction model capable of performing image super-resolution reconstruction.

[0093] Referring to FIG. 4 , in some embodiments, before step S102 , the high-resolution image reconstruction method may include but is not limited to steps S401 to S404 :

[0094] Step S401, obtaining the original reconstructed model;

[0095] Step S402, transforming the original reconstruction model according to a preset half-splitting operator and preset auxiliary variables to obtain an equivalent reconstruction model;

[0096] Step S403: Decoupling the equivalent reconstruction model to obtain a data module and a priori module;

[0097] Step S404: combining the data module and the priori module to obtain an image reconstruction model.

[0098] In some embodiments, in step S401, the original reconstruction model is a traditional image reconstruction model, and the original reconstruction model is a deep learning model used for super-resolution reconstruction. Specifically, the deep learning model can be a convolutional neural network or a generative adversarial network, and the deep learning model learns the mapping relationship from the low-resolution image to the high-resolution image, thereby generating the high-resolution image.

[0099] Specifically, in this embodiment, the original reconstruction model is shown in formula (1):

[0100] Where y is the low-resolution image, represents the two-dimensional convolution with the blur kernel, ↓s represents the standard s-fold downsampling operator, f(·) represents the image denoiser, σ is the first trade-off parameter, and λ is the second trade-off parameter.

[0101] In step S402 of some embodiments, the original reconstruction model is converted into using a half-splitting operator and auxiliary variables, that is, an equivalent reconstruction model that is similar to the original reconstruction model is constructed. Specifically, the constructed equivalent reconstruction model is represented as shown in formula (2):

[0102] Where μ is the penalty parameter.

[0103] In step S403 of some embodiments, in order to further solve the original reconstruction model, the equivalent reconstruction model is decoupled, that is, x and z are iteratively solved to obtain the data module and the priori module.

[0104] It should be noted that the data module is obtained by solving Z for the equivalent reconstruction model, that is, the closed form solution is obtained using the fast Fourier transform, so the data module is shown in formula (3):

[0105] It should be noted that the priori module is obtained by solving x for the equivalent reconstruction model, and the priori module is shown in formula (4):

[0106] In step S404 of some embodiments, the original reconstruction model is decomposed into a data module and a priori module, and the data module is equivalent to convolution processing, while the priori module is equivalent to a denoiser, so the data module and the priori module are combined into an image reconstruction model, so that the structure setting within the image reconstruction model is simple, and the image reconstruction model can be constructed by only using a convolutional neural network and a denoiser to match.

[0107] In steps S401 to S404 shown in this embodiment, the traditional original reconstruction model is converted into an approximate equivalent reconstruction model, and the equivalent reconstruction model is then decoupled into a data module and a priori module, and then the data module and the priori module are combined into an image reconstruction model, so that the image reconstruction model is easy to construct and convenient for reconstructing low-resolution images into high-resolution images.

[0108] In step S102 of some embodiments, the image reconstruction model includes at least two stages of image reconstruction networks, each of which includes a data module and a priori module. As shown in FIG5 , in this embodiment, multiple image reconstruction networks are provided, and a low-resolution image is input to an image reconstruction network. The image output by each image reconstruction network is passed to the image reconstruction network of the next stage, until the last image reconstruction network outputs a high-resolution image.

[0109] Please refer to FIG. 6 . In some embodiments, step S103 may include but is not limited to steps S601 to S602 .

[0110] Step S601, performing convolution processing on the low-resolution image through the data module to obtain a preliminary processed image;

[0111] Step S602 , performing denoising processing on the preliminary processed image through a priori module to obtain a preliminary reconstructed image.

[0112] In step S601 of some embodiments, a low-resolution image is input to each image reconstruction network, that is, to the data module of each image reconstruction network. It should be noted that the data module performs convolution processing on the low-resolution image to obtain a preliminary processed image, and the preliminary processed image is already an image with increased resolution. For example, as shown in FIG5 , a low-resolution magnetic resonance vascular wall image y is input to each data module for convolution processing and output. Specifically, the data module performs Fourier transform and inverse Fourier transform on the low-resolution magnetic resonance vascular wall image, and performs conjugate transposition processing on the low-resolution magnetic resonance vascular wall image using the conjugate transpose corresponding to the Fourier transform equation. The processed magnetic resonance vascular wall image is then processed according to a preset sampling operator to obtain a preliminary processed image.

[0113] In step S602 of some embodiments, the a priori module performs denoising on the preliminary processed image to obtain a preliminary reconstructed image. It should be noted that the a priori module includes a denoiser, which denoises the preliminary processed image to obtain the preliminary reconstructed image. Furthermore, the a priori module is used to compare the reconstructed image with the high-resolution image to verify the image reconstruction effect, thereby calculating the resolution loss between the reconstructed image and the high-resolution image, and adjusting the image reconstruction network in the next stage based on the resolution loss.

[0114] In steps S601 to S602 shown in this embodiment, the low-resolution image is convolved into a preliminary processed image by setting a data module, and then the preliminary processed image is denoised by the prior module to obtain a preliminary reconstructed image, so that the image reconstruction operation is simple.

[0115] Please refer to FIG. 7 . In some embodiments, step S104 includes but is not limited to steps S701 to S702 :

[0116] Step S701, performing resolution loss calculation on the preliminary reconstructed image and the target image to obtain resolution loss data;

[0117] Step S702: Adjust parameters of the image reconstruction network in the next stage according to the resolution loss data.

[0118] In step S701 of some embodiments, the resolution loss between the preliminary reconstructed image and the target image is calculated by the prior module to obtain resolution loss data. It should be noted that the resolution loss data is used to determine whether the image reconstruction network can produce a reconstructed image close to the target image, that is, to analyze the effect of the image reconstruction network in performing high-resolution reconstruction. Specifically, if the resolution loss data is larger, it indicates that the high-resolution reconstruction effect of the current image reconstruction network is poor. Conversely, if the resolution loss is smaller, it indicates that the high-resolution reconstruction effect of the current image reconstruction network is better, and it can generate a reconstructed image close to the real target image.

[0119] In step S702 of some embodiments, the network parameters of the image reconstruction network in the next stage are adjusted according to the resolution loss data, that is, σ and λ in equations (3) and (4) are adjusted, that is, the first trade-off parameter and the second trade-off parameter are adjusted.

[0120] Specifically, formula (3) can be equivalent to formula (5):

[0121] Where F() is the Fourier transform, F -1 () is the inverse Fourier transform, is the conjugate transpose of F(), θ s denotes different block processing operators with element-wise multiplication, and ↓s denotes different block downsampling operators.

[0122] Convert formula (3) into formula (6):

[0123] x j =P(z j ,β j )=β j *CNN denoiser (x j-1 )+z j (6)

[0124] It should be noted that updating the reconstructed image and the target image to adjust the parameters of the image reconstruction network in the next stage is similar to step S701 to step S702, and will not be repeated here.

[0125] In steps S701 to S702 shown in this embodiment, the resolution loss data between the target image and the preliminary constructed image is calculated, and the image reconstruction network of the next stage is adjusted according to the resolution loss data to construct an image reconstruction network with better reconstruction effect. This allows the image reconstruction process to simultaneously set up multiple stages of network training and output a high-resolution image with the best reconstruction effect.

[0126] In some embodiments, during the use of the image reconstruction model, a target image does not exist. Therefore, the resolution of the preliminary reconstructed image output by the image reconstruction network is calculated to obtain the reconstruction resolution. The resolution loss between the reconstruction resolution and the preset target resolution is then used to adjust the image reconstruction network of the next stage, so that the image reconstruction network of the final stage can output a high-resolution image with a resolution close to the target resolution. For example, if the resolution of the input low-resolution image is 64*64 and the target resolution is 256*256, and the resolution of the preliminary reconstructed image is 128*128, the parameters of the image reconstruction network of the next stage are adjusted until the image reconstruction network of the final stage outputs a high-resolution image with a resolution close to 256*256.

[0127] Please refer to FIG8 . In some embodiments, step S702 may include but is not limited to steps S801 to S802 :

[0128] Step S801, adjusting the first trade-off parameter of the data module of the next stage by using the preset hyperparameter module and resolution loss data;

[0129] Step S802 : adjusting the second trade-off parameter of the priori module of the next stage through the preset hyperparameter module and resolution loss data.

[0130] In step S801 of some embodiments, it can be seen from formula (1) that the first trade-off parameter to be adjusted is σ, but it can be seen from formula (5) that the first trade-off parameter after conversion to the data module is So adjust the first trade-off parameter in the data module through the hyperparameter module Let the data module in the next stage construct the low-resolution image into a high-resolution image that is closer to the target image, or a high-resolution image with a resolution close to the target resolution.

[0131] In step S802 of some embodiments, the second trade-off parameter to be adjusted is λ according to formula (1), but the second trade-off parameter after conversion to the priori module is [β1, ..., β j Therefore, the second trade-off parameter [β1,……,β j It should be noted that the training of the data module and the prior module is joint training to construct an image reconstruction network with better reconstruction effect.

[0132] Specifically, the hyperparameter module adjusts the parameters of the data module and the prior module according to formula (7):

[0133] In steps S801 to S802 shown in this embodiment, the first trade-off parameter of the data module is adjusted through the hyperparameter module and the resolution loss data, and the second trade-off parameter of the prior module is adjusted to achieve joint parameter adjustment, thereby constructing an image reconstruction network with better reconstruction effect, so as to output a reconstructed image close to the target image or with a resolution close to the target resolution.

[0134] In step S105 of some embodiments, the specific operation of the adjusted image reconstruction network to perform high-resolution reconstruction on the low-resolution image is the same as steps S601 to S602. It can also be considered that there is no adjusted image reconstruction network or the adjusted image reconstruction network performs high-resolution reconstruction on the low-resolution image similar to steps S601 to S602, which will not be repeated here.

[0135] In step S106 of some embodiments, the specific steps of updating the reconstructed image and adjusting the target image in the next stage of the image reconstruction network are the same as the adjustment steps of the image reconstruction network in the next stage. Please refer to steps S701 to S702 for details, which will not be repeated here.

[0136] As shown in FIG6 , when training the image reconstruction model, the embodiment of the present application first obtains original images of different sizes, and then crops the original images of different sizes into target images of the same size, and the target images are high-resolution images. Low-resolution images are obtained by performing multiple interpolation processes on the high-resolution images, and the low-resolution images are respectively input into the data modules of each image reconstruction network. When the data module of the first stage performs convolution processing on the low-resolution images, a reconstructed image is obtained. The priori module performs denoising on the reconstructed image, and at the same time, the priori module calculates the resolution loss data between the reconstructed image after denoising and the target image, so as to adjust the first trade-off parameter of the data module of the next stage through the super-resolution module and the resolution loss data, and at the same time adjust the second trade-off parameter of the priori module of the next stage. Similarly, after each image reconstruction network completes the image reconstruction, it will calculate the resolution loss data to adjust the image reconstruction network of the next stage, so the later image reconstruction networks learn from previous experience and can produce high-resolution images with better reconstruction effects. Therefore, the reconstructed image output by the last image reconstruction network is used as the high-resolution image. If, during the use of the image reconstruction model, low-resolution images are also input into the data module of each stage until the parameters of the data module and the prior module are adjusted, the image reconstruction network of the next stage is adjusted based on the resolution of the reconstructed image output by the previous stage and the preset target resolution, and training is iterated until the last image reconstruction network outputs a high-resolution image. Therefore, whether in training or actual use, the image reconstruction network optimizes the image reconstruction network of the next stage with each reconstruction, making full use of the prior information and combining it with the flexibility of the deep learning model, allowing a single image reconstruction model to perform super-resolution processing on low-resolution images of different scale factors, and combining it with the advantages of the deep learning algorithm to generate a high-resolution image that is closer to the target image, or closer to the resolution requirement.

[0137] Referring to FIG. 9 , an embodiment of the present application further provides a high-resolution image reconstruction device that can implement the above-mentioned high-resolution image reconstruction method. The device includes:

[0138] An image acquisition module 901 is configured to acquire a low-resolution image; wherein the low-resolution image is obtained by reducing the resolution of a preset target image, where the target image is a high-resolution image;

[0139] An input module 902 is configured to input a low-resolution image into a preset image reconstruction model; wherein the image reconstruction model includes an image reconstruction network of at least two stages;

[0140] A first reconstruction module 903 is configured to perform high-resolution reconstruction on the low-resolution image through an image reconstruction network to obtain a preliminary reconstructed image;

[0141] A first parameter adjustment module 904 is used to adjust parameters of the image reconstruction network in the next stage according to the preliminary reconstructed image and the target image;

[0142] A second reconstruction module 905 is configured to perform high-resolution reconstruction on the preliminary reconstructed image using the adjusted image reconstruction network to obtain an updated reconstructed image;

[0143] The second parameter adjustment module 906 is used to adjust the parameters of the image reconstruction network in the next stage according to the updated reconstructed image and the target image until the image reconstruction network in the final stage outputs a high-resolution image.

[0144] The specific implementation of the high-resolution image reconstruction device is substantially the same as the specific embodiment of the high-resolution image reconstruction method described above, and will not be described in detail here.

[0145] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the high-resolution image reconstruction method described above when executing the computer program. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0146] Please refer to FIG10 , which illustrates a hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0147] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0148] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the high-resolution image reconstruction method of the embodiments of this application.

[0149] Input / output interface 1003, used to implement information input and output;

[0150] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0151] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0152] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0153] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned high-resolution image reconstruction method is implemented.

[0154] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The high-resolution image reconstruction method and apparatus, computer equipment, and storage medium provided in the embodiments of the present application are constructed by constructing an image reconstruction network with at least two stages, and after each image reconstruction network completes high-resolution image reconstruction, the image reconstruction network of the next stage is adjusted according to the reconstructed image and the original high-resolution image, thereby repeatedly reconstructing and adjusting network parameters. Therefore, the embodiments of the present application apply prior information to image reconstruction and combine the flexibility of deep learning models to allow a single image reconstruction model to perform super-resolution processing on low-resolution images of different scale factors, and combine the advantages of deep learning algorithms to generate high-resolution images that are closer to the target image or closer to the resolution requirement.

[0156] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0157] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0159] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0160] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that in this application, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0166] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A high-resolution image reconstruction method, characterized in that, the method includes: Obtain a low-resolution image; wherein, the low-resolution image is obtained by reducing the resolution of a preset target image, and the target image is a high-resolution image; Input the low-resolution image into a preset image reconstruction model; wherein, the image reconstruction model includes image reconstruction networks of at least two stages; Perform high-resolution reconstruction on the low-resolution image through the image reconstruction network to obtain a preliminary reconstructed image; Adjust the parameters of the image reconstruction network of the next stage according to the preliminary reconstructed image and the target image; Perform high-resolution reconstruction on the low-resolution image through the adjusted image reconstruction network to obtain an updated reconstructed image; Adjust the parameters of the image reconstruction network of the next-next stage according to the updated reconstructed image and the target image until the image reconstruction network of the last stage outputs a high-resolution image.

2. The method according to claim 1, characterized in that, before inputting the low-resolution image into the preset image reconstruction model, the method further includes: Construct the image reconstruction model, specifically including: Obtain an original reconstruction model; Convert the original reconstruction model according to a preset semi-splitting operator and a preset auxiliary variable to obtain an equivalent reconstruction model; Perform decoupling processing on the equivalent reconstruction model to obtain a data module and a prior module; Combine the data module and the prior module to obtain the image reconstruction model.

3. The method according to claim 2, characterized in that, performing high-resolution reconstruction on the low-resolution image through the image reconstruction network to obtain a preliminary reconstructed image includes: Perform convolution processing on the low-resolution image through the data module to obtain a preliminary processed image; Perform denoising processing on the preliminary processed image through the prior module to obtain the preliminary reconstructed image.

4. The method according to claim 2, characterized in that, adjusting the parameters of the image reconstruction network of the next stage according to the preliminary reconstructed image and the target image includes: Calculate the resolution loss between the preliminary reconstructed image and the target image to obtain resolution loss data; Adjust the parameters of the image reconstruction network of the next stage according to the resolution loss data.

5. The method according to claim 4, characterized in that, adjusting the parameters of the image reconstruction network of the next stage according to the resolution loss data includes: Adjust the first trade-off parameter of the data module of the next stage through a preset hyperparameter module and the resolution loss data; Adjust the second trade-off parameter of the prior module of the next stage through a preset hyperparameter module and the resolution loss data.

6. The method according to any one of claims 1 to 5, characterized in that, obtaining the low-resolution image includes: Obtain a target image; Perform interpolation processing on the target image according to a preset number of interpolation times to obtain the low-resolution image.

7. The method according to claim 6, characterized in that, The obtaining of the target image includes: Obtaining an original image set; wherein, the original image set includes original images with different sizes; Performing a cropping process on the original image according to a preset image size to obtain the target image.

8. A high-resolution image reconstruction device Characterized in that The device includes: An image acquisition module, configured to acquire a low-resolution image; wherein, the low-resolution image is obtained by reducing the resolution of a preset target image, and the target image is a high-resolution image; An input module, configured to input the low-resolution image into a preset image reconstruction model; wherein, the image reconstruction model includes an image reconstruction network with at least two stages; A first reconstruction module, configured to perform high-resolution reconstruction on the low-resolution image through the image reconstruction network to obtain a preliminary reconstruction image; A first parameter adjustment module, configured to adjust the parameters of the image reconstruction network in the next stage according to the preliminary reconstruction image and the target image; A second reconstruction module, configured to perform high-resolution reconstruction on the preliminary reconstruction image through the adjusted image reconstruction network to obtain an updated reconstruction image; A second parameter adjustment module, configured to adjust the parameters of the image reconstruction network in the next next stage according to the updated reconstruction image and the target image until the image reconstruction network in the last stage outputs a high-resolution image.

9. A computer device Characterized in that The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the high-resolution image reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, the computer-readable storage medium stores a computer program Characterized in that When the computer program is executed by a processor, it implements the high-resolution image reconstruction method according to any one of claims 1 to 7.

Images