Super-resolution imaging method and device

Abstract

The present invention discloses a super-resolution imaging method and device, which relate to the field of imaging processing technology. The super-resolution imaging method provided by the present invention uses an adaptive moment estimation method to perform end-to-end training on an initial network model based on a constructed training data set to obtain a trained network model. Then, a multi-rotating camera is used to obtain a rotated image of the target to be imaged, and the rotated image is input into the trained network model to obtain a high-resolution image of the target to be imaged. It can be seen that the present invention adopts a rotating camera method and introduces a deep neural network model of the imaging physics process, which can effectively encode and decode high-frequency information in low-resolution images and output a high-resolution image that restores more high-frequency information, thereby improving the image resolution while improving the imaging quality.

Description

Technical Field

[0001] The present invention relates to the field of imaging processing technology, and in particular to a super-resolution imaging method and device. Background Art

[0002] Super-resolution imaging involves restoring a high-resolution image from one or more low-resolution images through computational imaging. A high-resolution image means more pixels are used to describe the image, which, for complex scenes, means greater ability to depict image detail. This technology is currently widely used in fields such as video processing, medical imaging, remote sensing, and public security monitoring. The main difficulty with current super-resolution imaging methods is that super-resolution is an ill-posed problem. For a given low-resolution image input, there may be multiple plausible super-resolution predictions, making it difficult to determine the optimal super-resolution image.

[0003] Recent super-resolution computational imaging methods can be categorized into single-image super-resolution and multi-image super-resolution methods, based on the number of low-resolution input images. Single-image super-resolution methods use a single low-resolution image and prior information to restore a high-resolution image. However, these methods face significant challenges due to their severe pathologies caused by insufficient observational information. Existing methods incorporate various prior information to interpolate, reconstruct, or use learning-based methods to obtain super-resolution images. However, the high-resolution image estimated from a single image can produce severe artifacts, which in turn degrades the quality of the high-resolution image and negatively impacts downstream tasks. Multi-image super-resolution methods leverage the non-redundant information between multiple images to increase the amount of information captured in a single image, thereby achieving super-resolution images. Compared to single-image super-resolution methods, multi-image super-resolution methods often produce more reliable super-resolution results. However, most existing multi-image super-resolution methods employ continuous capture with a single camera under slight displacement, or capture with multiple cameras that are horizontally or vertically offset relative to the camera's focal plane. Among them, the former adopts the method of sacrificing temporal resolution in exchange for spatial resolution, which is difficult to apply to tasks with high requirements for imaging efficiency; the latter does not fully utilize the system structure characteristics of multiple cameras.

[0004] For example, the article Bhat G, Danelljan M, Van Gool L, et al. Deep burst super-resolution[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021. By processing multiple noisy low-resolution images in RAW format, a noise-reduced super-resolution RGB image is generated, and the super-resolution effect is good. However, the input low-resolution image in this article comes from the random jitter during continuous shooting of a smartphone, and the non-redundant complementary characteristics of the information between each sub-image cannot be guaranteed. It is not suitable for various application scenarios that require the stability of the imaging device.

[0005] The article Wilburn B, Joshi N, Vaish V, et al. Highperformance imaging using large camera arrays [M] / / ACM SIGGRAPH 2005 Papers. 2005: is an early work that uses camera array methods for high-performance imaging. By using multiple cameras to form a system to approximate a camera with a large aperture, a dense camera array was used. However, the high cost of dense camera arrays has limited the further development of this method to a certain extent.

[0006] The paper Bonchev S, Alexiev K. Improving super-resolution image reconstruction by in-plane camera rotation [C] / / 2010 13th International Conference on Information Fusion. IEEE, 2010. proposes actively controlling the camera's displacement and rotation within the focal plane to obtain higher-frequency information in the original sampling direction, thereby improving the signal-to-noise ratio of the super-resolved image. However, this paper only provides a partial theoretical analysis and does not propose an effective multi-image super-resolution algorithm.

[0007] The article Jiang S, Guan M, Wu J, et al. Frequency-domain diagonal extension imaging [J]. Advanced Photonics, 2020. conducted an in-depth analysis of the problem of grid sampling pattern changes caused by the rotation of the camera in the focal plane, and applied the strategy of rotating the camera to lensless microscopy, and achieved good super-resolution imaging results, but did not further apply this method to super-resolution imaging of ordinary cameras. Summary of the Invention

[0008] In order to solve the above problems existing in the prior art, the present invention provides a super-resolution imaging method and device.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A super-resolution imaging method, comprising:

[0011] Build a training dataset;

[0012] Obtaining an initial network model; the initial network model adopts a Charbonnier loss function; the initial network model includes: a feature extraction module, a feature registration module, a feature fusion module and an image reconstruction module;

[0013] Based on the training data set, the initial network model is end-to-end trained using an adaptive moment estimation method to obtain a trained network model;

[0014] A multi-rotating camera is used to obtain a rotation image of the target to be imaged, and the rotation image is input into the trained network model to obtain a high-resolution image of the target to be imaged.

[0015] Optionally, constructing a training data set specifically includes:

[0016] Obtaining a training data set ZRR, and converting data set images in the training data set ZRR into grayscale images;

[0017] Acquiring an original image value of the grayscale image using an inverse camera imaging model to obtain an original image;

[0018] Performing random operations on the original image to obtain a high-resolution reference image; the random operations include random rotation, random translation, and random cropping;

[0019] Performing multiple rotation transformations on the high-resolution reference image with a set angle as an increment to obtain multiple images at different angles;

[0020] Adding pixels within a preset translation range and random translation and rotation changes within a preset rotation range to the rotationally transformed high-resolution reference image to simulate system calibration errors to obtain a high-resolution image;

[0021] downsampling the high-resolution image multiple times using an anisotropic box filter and adding noise to obtain a low-resolution sub-image;

[0022] forming a high-low resolution image pair based on the low resolution sub-image and the high resolution image;

[0023] A low-resolution sub-image in the high-low resolution image pair having the same angle as the high-resolution image is used as a reference image, and low-resolution sub-images other than the reference image are used as auxiliary images to form a training data set.

[0024] Optionally, the high-resolution reference image is rotated five times with an increment of 15° to obtain six images at different angles.

[0025] Optionally, random translation and rotation changes with a translation range of [-24 pixels, 24 pixels] and a rotation range of [-3°, 3°] are added to the high-resolution reference image that has undergone rotation transformation to simulate the system calibration error to obtain a high-resolution image.

[0026] Optionally, the high-resolution image is downsampled by a factor of 4 using an anisotropic box filter, and noise is added to obtain a low-resolution sub-image.

[0027] Optionally, after performing end-to-end training on the initial network model using an adaptive moment estimation method based on the training data set to obtain a trained network model, the method further includes:

[0028] Modify the dimension of the grayscale image patches fed into the trained network model.

[0029] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0030] The super-resolution imaging method provided by the present invention introduces a deep neural network model of the imaging physics process, which can effectively decode the information in low-resolution images and output more realistic high-resolution images, thereby improving the quality of generated high-resolution images while improving the imaging authenticity.

[0031] In addition, corresponding to the super-resolution imaging method provided above, the present invention also provides a super-resolution imaging device, which includes:

[0032] An image acquisition module, used to obtain a rotating image of the target to be imaged;

[0033] A high-precision rotation module, used for rotating the image acquisition module;

[0034] An image processing module is connected to the image acquisition module and the high-precision rotation module, respectively, and is implanted with a software program for generating control instructions for the image acquisition module and the high-precision rotation module, and is also used to obtain a high-resolution image of the target to be imaged based on the software program and the rotated image; the software program is used to implement the super-resolution imaging method provided above.

[0035] Optionally, the high-precision rotation module includes: a turntable and a connecting member;

[0036] The turntable is electrically connected to the image processing module; the connecting member is used to fix the image acquisition module; the turntable is used to rotate and open and close according to the control instructions generated by the image processing module.

[0037] Optionally, the image acquisition module includes a single industrial camera;

[0038] Each of the industrial cameras is mounted on the turntable via the connecting member; the connecting member enables the optical axis of the industrial camera to be parallel to the rotation axis of the turntable, and enables the distance between the optical axis of the industrial camera and the rotation axis of the turntable to meet a preset distance requirement.

[0039] Since the technical effects achieved by the super-resolution imaging device provided by the present invention are the same as the technical effects achieved by the super-resolution imaging method provided above, they will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A flowchart of a super-resolution imaging method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a network model provided in an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the training data set preparation process provided by an embodiment of the present invention;

[0044] Figure 4 A schematic structural diagram of a super-resolution imaging device provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the placement of a rotating camera provided in an embodiment of the present invention;

[0046] Figure 6 A super-resolution imaging flow chart of a super-resolution imaging device provided by an embodiment of the present invention;

[0047] Figure 7 A comparison chart of the effects of different super-resolution methods on the ZRR dataset provided by the embodiment of the present invention; wherein, Figure 7 (a) is the image truth method effect test picture; Figure 7 (b) is the EDSR effect test image; Figure 7 (c) is the EBSR effect test image; Figure 7 (d) is a test diagram of the effect of the method provided by the present invention;

[0048] Figure 8 The following is a typical output result diagram of different methods provided in the embodiments of the present invention; wherein, Figure 8 (a) is the relationship diagram between amplitude and pixel in the image truth method; Figure 8 (b) is the relationship diagram between amplitude and pixel in bilinear interpolation method; Figure 8 (c) is a graph showing the relationship between amplitude and pixel in the method provided by the present invention; Figure 8 (d) is the output result of the image truth method; Figure 8 (e) is the output result of the bilinear interpolation method; Figure 8 (f) is the output result diagram of the method provided by the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] The purpose of the present invention is to provide a super-resolution imaging method and device, which can obtain spatial high-frequency information and thus achieve better super-resolution effect.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] like Figure 1 As shown, the super-resolution imaging method provided in this embodiment includes:

[0054] Step 100: Build a training dataset.

[0055] Step 101: Obtain an initial network model. The initial network model uses the Charbonnier loss function. Figure 2 As shown in the figure, the initial network model includes: feature extraction module, feature registration module, feature fusion module and image reconstruction module.

[0056] Step 102: Based on the training data set, the initial network model is end-to-end trained using the adaptive moment estimation method to obtain a trained network model.

[0057] Step 103: Use multiple rotating cameras to obtain a rotated image of the target to be imaged, and input the rotated image into the trained network model to obtain a high-resolution image of the target to be imaged.

[0058] Example 2

[0059] In this embodiment, the construction of the training dataset using the public training dataset ZRR (Zurich Raw-to-RGB dataset) is used as an example to illustrate the specific implementation process of the super-resolution imaging method disclosed in the above embodiment 1, which specifically includes:

[0060] Step 1: Build a training dataset

[0061] (1) Figure 3 As shown in the ZRR (Zurich Raw-to-RGB dataset), which contains 46,839 high-quality images, the dataset images are first converted to 8-bit grayscale images, and the inverse camera imaging model is used to obtain the original image values ​​to obtain grayscale images. ZRR is a benchmark proposed by Andrey Ignatov et al. at CVPR 2020 for evaluating the performance of image super-resolution algorithms. It currently contains over 40,000 high-quality images of real scenes captured in various locations, lighting, and weather conditions, and is widely used in the field of image super-resolution.

[0062] (2) A high-resolution reference image is generated by performing random rotation, translation, and multiple random crops on the original image. The translation and rotation values ​​of the high-resolution reference image are random, and the cropped image is 256*256 pixels, and the cropping position is also random.

[0063] (3) Five rotation transformations with an increment of 15° are performed on each high-resolution reference image to obtain six images at different angles. Random translation and rotation transformations with a translation range of [-24, 24] (in pixels) and a rotation range of [-3°, 3°] are added to the five high-resolution reference images that have undergone rotation transformation to simulate the system calibration error.

[0064] (4) Use an anisotropic box filter to downsample the image by a factor of four and add noise to each of them to obtain six low-resolution sub-images, which together with the high-resolution image form a high-low resolution image pair. Among them, the low-resolution image with the same angle as the high-resolution image is used as the reference image, and the remaining images are used as auxiliary images.

[0065] (5) To achieve data expansion, each high-resolution image is randomly flipped horizontally and vertically, and steps (3) to (4) are repeated to generate a dataset of high-resolution and low-resolution image pairs. The probability of each image pair being subjected to the corresponding operation is 0.5.

[0066] Step 2: Network model parameter optimization

[0067] The loss function of the network model adopts the Charbonnier loss function, which is shown in the following formula:

[0068]

[0069] Among them, S represents the total number of images contained in each input image block during network training. Represents the i-th low-resolution image contained in the image block, E() represents the process of inputting the network and outputting the high-resolution image, represents the true value of the corresponding i-th high-resolution image, and ∈ is the bias constant.

[0070] The network model is updated using the adaptive moment estimation method. An end-to-end training mode is used on a dataset generated using the ZRR dataset, which contains multiple sets of high- and low-resolution image pairs (i.e., the training dataset). ∈ is set to 1e-6, 32 sets of image blocks are used for each training, the learning rate is set to 0.0002, and the learning rate is halved every 100 cycles. A total of 500 cycles are trained, and the training phase of the network model is completed.

[0071] Step 3: Super-resolution imaging

[0072] (1) Load the parameters of the trained network model

[0073] Load the parameters of the network model trained in step 2, mainly including the pre-calibrated imaging model. Feed the images collected by the rotating camera imaging device into the trained network model, and keep the trained network model parameters unchanged.

[0074] (2) Image super-resolution reconstruction

[0075] According to the actual application requirements, a part of the captured images is selected as the reference image, and the rest of the images are used as auxiliary images. The rotated image is input to obtain the corresponding high-resolution image.

[0076] Modify the B*N*C*W*H grayscale image block dimensions of the trained network model input, appropriately reducing the first dimension and increasing the last two dimensions, so that the trained network model can process images with resolutions larger than the training resolution. Therefore, when the total network input data volume exceeds B*N*C*W*H, since the trained network model is a fully convolutional neural network, modifying the trained input dimensions and replacing the graphics card with a higher memory capacity can enable the trained network model to process higher resolution input images without retraining.

[0077] Example 3

[0078] Corresponding to the super-resolution imaging method provided above, this embodiment provides a super-resolution imaging device, such as Figure 4 As shown, the device includes:

[0079] The image acquisition module is used to obtain a rotating image of the target to be imaged.

[0080] High-precision rotation module, used to rotate the image acquisition module.

[0081] The image processing module is connected to the image acquisition module and the high-precision rotation module, respectively, and is embedded with a software program for generating control instructions for the image acquisition module and the high-precision rotation module. The software program is also used to obtain a high-resolution image of the target to be imaged based on the software program and the rotated image. The software program is used to implement the super-resolution imaging method provided in the first or second embodiment.

[0082] Specifically, the image acquisition module's main hardware consists of industrial cameras and acquisition cards. Its primary function is to capture images within the field of view in real time, save historical images, and convert and transmit the captured images to the image processing module. The image acquisition module includes multiple industrial cameras.

[0083] Multiple industrial cameras are mounted on a turntable via connectors that align the optical axes of the industrial cameras with the turntable's axis of rotation and ensure that the distance between them meets a predetermined distance requirement.

[0084] The high-precision rotation module can rotate an industrial camera to the desired angle and read the actual angle. The high-precision rotation module includes a turntable and connectors.

[0085] The turntable is electrically connected to the image processing module. The connector is used to fix the image acquisition module. The turntable is used to rotate and open and close according to the control instructions generated by the image processing module.

[0086] The image processing module can use stored public datasets and saved historical images to train and fine-tune the super-resolution reconstruction network parameters and reconstruct high-resolution images.

[0087] Example 4

[0088] This embodiment uses the public training dataset ZRR (Zurich Raw-to-RGB dataset) as an example to construct a training dataset, and describes the specific implementation process of the super-resolution imaging device disclosed in the above embodiment 3, which specifically includes:

[0089] Step 1: Image acquisition

[0090] 1) Solution 1: Rotating Camera Image Acquisition

[0091] Mount the industrial camera on a high-precision turntable with controllable rotation angle. Use connectors to make the camera optical axis parallel to the turntable axis and keep the axis close to the optical axis to ensure a high degree of overlap when rotating the camera. Place the target in front of the camera and rotate the camera 15° each time (the angle can be changed, and multiple times adding up to 90 degrees is sufficient). Capture an image after each rotation, taking a total of six images (not limited to six, can be more or less). The position and rotation angle of the camera at each image capture from the perspective directly in front of the camera are as follows: Figure 5 shown.

[0092] 2) Solution 2: Multiple cameras simultaneously capture images

[0093] Multiple cameras are fixed to a mechanical structure at the same time, and the position and rotation angle of each camera are as follows: Figure 5 As shown, under the premise of no mechanical interference, each camera should be as close as possible to ensure that the cameras have a high degree of field of view overlap.

[0094] Step 2: Network design and implementation

[0095] like Figure 2 As shown in the figure, the network model design includes four major modules: feature extraction, feature registration, feature fusion and image reconstruction.

[0096] Among them, the feature extraction module is a multi-layer feature extraction module based on gradient guidance, which includes an image branch and a gradient branch. It uses the Sobel operator to extract the gradient information in the input image, introduces the attention mechanism to selectively enhance the image features input to the lower-level module, and the module inputs a B*N*C*W*H grayscale image block and outputs a high-dimensional feature block.

[0097] The feature registration module is a feature registration module inspired by physical processes. It uses a pre-calibrated imaging model combined with the optical flow method to achieve feature registration and obtain a registration feature block with the same dimension as the feature block.

[0098] The feature fusion module and image reconstruction module mainly learn high-frequency residual information through cross-image fusion and long-distance cascade network, reconstruct high-resolution images, and finally output B*C*4W*4H grayscale image blocks to achieve multiple super-resolution.

[0099] In this embodiment, the network model is implemented in Python on the PyTorch framework.

[0100] Step 3: Create a training dataset

[0101] (1) Figure 3 As shown in the figure, in the ZRR (Zurich Raw-to-RGB dataset) dataset containing 46,839 high-quality images, the dataset images are first converted into 8-bit grayscale images, and the inverse camera imaging model is used to obtain the original value of the image to obtain a grayscale image.

[0102] (2) A high-resolution reference image is generated by performing random rotation, translation, and multiple random crops on the original image. The translation and rotation values ​​of the high-resolution reference image are random, and the cropped image is 256*256 pixels, and the cropping position is also random.

[0103] (3) Five rotation transformations with an increment of 15° are performed on each high-resolution reference image to obtain six images at different angles. Random translation and rotation transformations with a translation range of [-24, 24] (in pixels) and a rotation range of [-3°, 3°] are added to the five high-resolution reference images that have undergone rotation transformation to simulate the system calibration error.

[0104] (4) Use an anisotropic box filter to downsample the image by a factor of four and add noise to each of them to obtain six low-resolution sub-images, which together with the high-resolution image form a high-low resolution image pair. Among them, the low-resolution image with the same angle as the high-resolution image is used as the reference image, and the remaining images are used as auxiliary images.

[0105] (5) To achieve data expansion, each high-resolution image is randomly flipped horizontally and vertically, and steps (3) to (4) are repeated to generate a dataset of high-resolution and low-resolution image pairs. The probability of each image pair being subjected to the corresponding operation is 0.5.

[0106] Step 4: Network model parameter optimization

[0107] The loss function of the network model adopts the Charbonnier loss function, which is shown in the following formula:

[0108]

[0109] Among them, S represents the total number of images contained in each input image block during network training. Represents the i-th low-resolution image contained in the image block, E() represents the process of inputting the network and outputting the high-resolution image, represents the true value of the corresponding i-th high-resolution image, and ∈ is the bias constant.

[0110] The network model is updated using the adaptive moment estimation method. An end-to-end training mode is used on a dataset generated using the ZRR dataset, which contains multiple sets of high- and low-resolution image pairs (i.e., the training dataset). ∈ is set to 1e-6, 32 sets of image blocks are used for each training, the learning rate is set to 0.0002, and the learning rate is halved every 100 cycles. A total of 500 cycles are trained, and the training phase of the network model is completed.

[0111] Step 5: Super-resolution imaging

[0112] like Figure 4 As shown, this process includes:

[0113] (1) Load the parameters of the trained network model

[0114] Load the parameters of the network model trained in step 4, mainly including the pre-calibrated imaging model. Feed the images captured by the rotating camera into the trained network model, and keep the trained network model parameters unchanged.

[0115] (2) Image super-resolution reconstruction

[0116] According to the actual application requirements, one of the images taken by the industrial camera is selected as the reference image, and the remaining images are used as auxiliary images. The rotated image is input to obtain the corresponding high-resolution image.

[0117] Modify the B*N*C*W*H grayscale image block dimensions of the trained network model input, appropriately reducing the first dimension and increasing the last two dimensions, so that the trained network model can process images with resolutions larger than the training resolution. Therefore, when the total network input data volume exceeds B*N*C*W*H, since the trained network model is a fully convolutional neural network, modifying the trained input dimensions and replacing the graphics card with a higher memory capacity can enable the trained network model to process higher resolution input images without retraining.

[0118] Based on the above description, the super-resolution imaging process of the super-resolution imaging device is as follows: Figure 6 shown.

[0119] The advantages of the solutions provided in the above-mentioned embodiments 1 to 4 are further explained below by means of simulation comparison.

[0120] When evaluating the performance of the super-resolution algorithm, the images in the ZRR dataset are inversely transformed into the imaging model to generate a low-resolution image dataset as the algorithm input, and the original image is used as the true value to evaluate the performance of the super-resolution algorithm.

[0121] The super-resolution algorithm inputs a low-resolution image and generates a high-resolution image. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) calculated using the high-resolution image and the true value image are used to evaluate the algorithm performance. PSNR is an indicator that measures the global similarity between the generated image and the true value. The higher the PSNR, the smaller the difference between the generated image and the true value. SSIM is an indicator that measures the structural similarity between the generated image and the true value. Compared with PSNR, it is more in line with the visual perception of the human eye. The closer SSIM is to 1, the smaller the difference between the generated image and the true value. The performance test results of the scheme provided above in the ZRR data set are shown in Table 1.

[0122] Table 1 Comparison of indicators of different super-resolution methods on ZRR dataset

[0123]

[0124]

[0125] Since other algorithms do not have a training process or do not specifically apply the rotation camera to capture images during training, the performance of the existing methods and this method is only compared on the test dataset. By comparing the current mainstream bicubic interpolation and multi-image super-resolution algorithms EDSR, DEEP-rep and EBSR, it can be seen that the PSNR of this method (the method provided by the present invention) is 43.94, which is higher than that of other algorithms. The generated image effect of this method is as follows Figure 7 shown.

[0126] The performance test results of the method provided by the present invention using a rotating camera imaging device to capture real images and using a deep neural network inspired by physical processes to generate high-resolution images are shown in Table 2.

[0127] Table 2 Performance test results of high-resolution images

[0128]

[0129] The experiment used a rotating camera imaging device to capture a total of 480 sets of high- and low-resolution corresponding rotated images, and used the same scene images to fine-tune the network. It can be seen that the performance of this method in generating high-resolution images is better than the bilinear interpolation method. The typical output results of different methods are as follows: Figure 8 shown.

[0130] Based on the above description, the high performance of this method stems from two main aspects: 1) Using a rotating camera imaging device to acquire images, the anisotropy of CCD pixels is exploited to capture high-frequency information in a specific direction, thereby increasing the information contained in the input image. 2) Using an image processing module with a clear physical imaging process, a physics-inspired deep neural network is constructed. This effectively decodes the additional information introduced by camera rotation, improving the quality of the generated high-resolution images and enhancing their realism.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0132] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A super-resolution imaging method, characterized in that: include: Constructing a training data set, including: obtaining a training data set ZRR and converting data set images in the training data set ZRR into grayscale images; using an inverse camera imaging model to obtain the original image value of the grayscale image to obtain the original image; performing random operations on the original image to obtain a high-resolution reference image; the random operations include random rotation, random translation and random cropping; performing five rotation transformations on the high-resolution reference image with an increment of 15° to obtain six images at different angles; adding random translation and rotation changes with a translation range of [-24 pixels, 24 pixels] and a rotation range of [-3°, 3°] to the high-resolution reference image after the rotation transformation to simulate a system calibration error to obtain a high-resolution image; using an anisotropic box filter to perform multiple downsampling on the high-resolution image and adding noise to obtain a low-resolution sub-image; forming a high-low resolution image pair based on the low-resolution sub-image and the high-resolution image; using the low-resolution sub-image with the same angle as the high-resolution image in the high-low resolution image pair as a reference image, and using the low-resolution sub-image other than the reference image as an auxiliary image to form a training data set; Obtain an initial network model; the initial network model adopts a Charbonnier loss function; the initial network model includes: a feature extraction module, a feature registration module, a feature fusion module and an image reconstruction module; wherein the feature extraction module is a multi-layer feature extraction module based on gradient guidance, which includes an image branch and a gradient branch, uses a Sobel operator to extract gradient information in the input image, introduces an attention mechanism to selectively enhance image features and inputs them to the lower-level module, the module inputs a grayscale image block of B*N*C*W*H, and outputs a high-dimensional feature block; the feature registration module is a feature registration module inspired by physical processes, which uses a pre-calibrated imaging model combined with an optical flow method to achieve feature registration, and obtains a registration feature block with the same dimension as the feature block; the feature fusion module and the image reconstruction module mainly learn high-frequency residual information through cross-image fusion and a long-distance cascade network, reconstruct a high-resolution image, and finally output a grayscale image block of B*C*4W*4H to achieve multiple super-resolution; Based on the training data set, the initial network model is end-to-end trained using an adaptive moment estimation method to obtain a trained network model; the dimension of the B*N*C*W*H grayscale image block input to the trained network model is modified, the first dimension of the image block is appropriately reduced and the last two dimensions are increased, so that the trained network model can process images with a resolution larger than that used for training; A multi-rotation camera is used to obtain a rotation image of the target to be imaged, and the rotation image is input into the trained network model to obtain a high-resolution image of the target to be imaged; the multi-rotation camera is rotated by a set angle each time, and an image is collected after each rotation; the sum of the set angles of multiple rotations is equal to 90 degrees.

2. The super-resolution imaging method according to claim 1, wherein: The high-resolution image is downsampled by a factor of 4 using an anisotropic box filter, and noise is added to obtain a low-resolution sub-image.

3. The super-resolution imaging method according to claim 1, wherein: After performing end-to-end training on the initial network model using the adaptive moment estimation method based on the training data set to obtain a trained network model, the method further includes: Modify the dimension of the grayscale image patches fed into the trained network model.

4. A super-resolution imaging device, characterized in that include: An image acquisition module, used to obtain a rotating image of the target to be imaged; A high-precision rotation module, used for rotating the image acquisition module; An image processing module is connected to the image acquisition module and the high-precision rotation module, respectively, and is implanted with a software program for generating control instructions for the image acquisition module and control instructions for the high-precision rotation module, and is also used to obtain a high-resolution image of the target to be imaged based on the software program and the rotated image; the software program is used to implement the super-resolution imaging method according to any one of claims 1 to 3.

5. The super-resolution imaging device according to claim 4, characterized in that The high-precision rotation module includes: a turntable and a connecting piece; The turntable is electrically connected to the image processing module; the connecting member is used to fix the image acquisition module; the turntable is used to rotate and open and close according to the control instructions generated by the image processing module.

6. The super-resolution imaging device according to claim 5, characterized in that The image acquisition module includes a single industrial camera; Each of the industrial cameras is mounted on the turntable via the connecting member; the connecting member enables the optical axis of the industrial camera to be parallel to the rotation axis of the turntable, and enables the distance between the optical axis of the industrial camera and the rotation axis of the turntable to meet a preset distance requirement.

Images