Image super-resolution reconstruction methods, devices and electronic equipment

Abstract

This application provides an image super-resolution reconstruction method, apparatus, and electronic device, relating to the field of image processing technology. The method includes: acquiring a first high-resolution image and at least two first low-resolution images based on a far-field reflective Fourier stacked imaging system; normalizing the first high-resolution image to obtain a second high-resolution image; performing phase random translation degradation and random phase difference degradation processing on the first low-resolution images to obtain a second low-resolution image; and training a deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain a target deep learning super-resolution network. In the above design, the low-resolution images acquired by the far-field reflective Fourier stacked imaging system significantly reduce data acquisition time; simultaneously, the degradation processing of the low-resolution images can improve the generalization ability of the deep learning super-resolution network.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image super-resolution reconstruction method, apparatus, and electronic device. Background Technology

[0002] Fourier Ptychographic (FP) imaging is a technique that uses a small-aperture camera to acquire target information at different spectral levels by employing methods such as camera shifting and laser source shifting. The target information is then stitched together in the frequency domain using phase retrieval methods to achieve large-scale target spectral synthesis, thereby obtaining a high-resolution image. However, in far-field imaging environments, due to numerous interference factors, traditional FP reconstruction algorithms often face problems such as unstable spectral shifts and complex noise, leading to severe artifacts and reduced resolution in the reconstructed image.

[0003] In recent years, deep learning methods have achieved remarkable results in the field of image processing. By leveraging the powerful computing capabilities of computers, deep learning can optimize Fourier layered images, thereby improving image quality.

[0004] However, most current deep learning-based methods primarily focus on network structure design for microscopic or transmission imaging, failing to adequately consider the physical model of far-field reflective Fourier layered imaging. This leads to problems such as artifacts and insufficient detail processing capabilities when handling far-field reflective Fourier layered imaging. Summary of the Invention

[0005] In order to at least overcome the above-mentioned shortcomings in the prior art, the purpose of this application is to provide an image super-resolution reconstruction method, apparatus and electronic device.

[0006] In a first aspect, embodiments of this application provide an image super-resolution reconstruction method, the image super-resolution reconstruction method comprising:

[0007] Based on a far-field reflective Fourier stacked imaging system, acquire a first high-resolution image and at least two first low-resolution images;

[0008] The first high-resolution image is normalized to obtain a second high-resolution image; the first low-resolution image is degraded to obtain a second low-resolution image, wherein the degradation processing includes phase random translation degradation processing and random phase difference degradation processing.

[0009] The deep learning super-resolution network is trained using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network.

[0010] In one possible implementation, the step of degrading the first low-resolution image to obtain a second low-resolution image includes:

[0011] Based on the first low-resolution image, the amplitude map of the first low-resolution image is calculated, and the amplitude map is subjected to Fourier transform to generate the first phase map.

[0012] Based on the first phase diagram, random phase difference degradation processing and multiple phase random translation degradation processing are performed to calculate the second phase diagram after introducing random perturbation;

[0013] Based on the amplitude map and the second phase map, a second low-resolution image after introducing random perturbations is calculated.

[0014] In one possible implementation, the amplitude diagram A ori With the first phase diagram Φ ori It is calculated in the following way:

[0015]

[0016] Among them, A ori This represents the amplitude graph after the Fourier transform, Φ ori This represents the first phase diagram after the Fourier transform. I represents the Fourier transform. ori This represents the intensity map before the Fourier transform, and n represents the bit depth of the first low-resolution image data.

[0017] The second phase diagram Φ after introducing random perturbation new It is calculated in the following way:

[0018] Φ new =random(Φ Z *random(Φ ori ,x i1 ,y i1 ),x i2 ,y i2 )

[0019] Where, Φ new This represents the second phase diagram after introducing random perturbations, Φ Z Represents the randomly generated Zernike aberration, random(…,x i ,y i ) represents the randomly generated image translation amount (x) i ,y i That is, to move the image horizontally by x. i Vertical movement y i ;

[0020] The second low-resolution image I after introducing random perturbation new It is calculated in the following way:

[0021]

[0022] Among them, I new This represents the second low-resolution image after introducing random perturbations. This represents the inverse Fourier transform, ||…|| 2 This represents taking the square of the modulus.

[0023] In one possible implementation, the deep learning super-resolution network includes a generator network and a discriminator network;

[0024] The generative network includes a multi-scale feature fusion module, a residual module, an upsampling module, and a feature mapping module. The multi-scale feature fusion module extracts multi-scale features from the input second low-resolution image to synthesize a first multi-channel feature map. The residual module adjusts the dimensionality of the feature map to generate a second multi-channel feature map. The upsampling module performs stepwise information decoding on the second multi-channel feature map to generate a third multi-channel feature map. The feature mapping module maps the third multi-channel feature map to a single-channel output image to obtain a first output result.

[0025] The discriminant network includes a feature extraction module, a downsampling module, and a sample classification module; the feature extraction module is used to extract features from the input first output result or the second high-resolution image; the downsampling module is used to reduce the number of parameters in the feature map; and the sample classification module is used to calculate the classification probability to obtain the second output result.

[0026] In one possible implementation, the step of training the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network includes:

[0027] The second low-resolution image is input into the generation network to obtain the first output result;

[0028] The first output result or the second high-resolution image is input into the discrimination network to obtain the second output result;

[0029] Based on the first output result and the second output result, calculate the loss of the generator network and the loss of the discriminator network;

[0030] Based on the loss of the generator network and the loss of the discriminator network, the parameters of the deep learning super-resolution network are updated to determine the target deep learning super-resolution network.

[0031] In one possible implementation, the step of updating the parameters of the deep learning super-resolution network based on the loss of the generator network and the loss of the discriminator network includes:

[0032] Based on the second high-resolution image and the first output result, the reconstruction loss L is calculated. rec ;

[0033] Based on the second output, calculate the adversarial loss L. gan ;

[0034] Based on the second high-resolution image and the first output result, calculate the first-order norm loss L of the Fourier domain phase. Φ ;

[0035] Combined with reconstruction loss L rec , countering losses L gan and the first norm loss L of the Fourier domain phase Φ Calculate the total loss L of the generator network. total ;

[0036] Based on the second output result, the loss L of the discriminant network is calculated. D ;

[0037] Based on the total loss L of the generator network total and the loss L of the discriminant network D Update the parameters of the generator network and the discriminator network until convergence.

[0038] In one possible implementation, the reconstruction loss L rec It is calculated in the following way:

[0039]

[0040] Among them, L rec Let represent the reconstruction loss, gt represent the second high-resolution image, and G(lr) represent the first output result when the second low-resolution image is used as input to the generator network. This indicates taking the expected value of the corresponding multidimensional tensor, and ||…||2 indicates taking the second norm;

[0041] The resistance loss L gan It is calculated in the following way:

[0042]

[0043] Among them, L ganLet D(gt) represent the adversarial loss, and let ||…||1 represent the second output result when the second high-resolution image is used as the input of the discriminant network.

[0044] The first-order norm loss L of the Fourier domain phase Φ It is calculated in the following way:

[0045]

[0046] Among them, L Φ This represents the first-order norm loss of the Fourier domain phase. This represents the Fourier transform, and angle(...) represents the phase diagram of the Fourier spectrum.

[0047] The total loss L of the generator network total It is calculated in the following way:

[0048] L total =L rec +λ×L gan +β×L Φ

[0049] Among them, L total Let λ represent the total loss of the generator network, λ represent the scaling factor of the adversarial loss function, and β represent the scaling factor of the first-order norm loss function of the Fourier domain phase.

[0050] The loss L of the discriminant network D It is calculated in the following way:

[0051]

[0052] Among them, L D This indicates the determination of network loss. D(G(lr)) represents the expected value of the corresponding multidimensional tensor, D(gt) represents the second output result when the first output result is used as the input of the discriminant network, and D(gt) represents the second output result when the second high-resolution image is used as the input of the discriminant network.

[0053] In one possible implementation, after the step of training the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network, the method further includes:

[0054] At least two images to be processed are input into the target deep learning super-resolution network to obtain a reconstructed high-resolution image.

[0055] Secondly, embodiments of this application also provide an image super-resolution reconstruction apparatus, the image super-resolution reconstruction apparatus comprising:

[0056] Acquisition module: used in a far-field reflection-based Fourier layered imaging system to acquire a first high-resolution image and at least two first low-resolution images;

[0057] Image processing module: used to normalize the first high-resolution image to obtain a second high-resolution image; and to degrade the first low-resolution image to obtain a second low-resolution image, wherein the degradation processing includes phase random translation degradation processing and random phase difference degradation processing.

[0058] Training module: used to train the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network.

[0059] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions, which, when executed by the processor, implement the methods described in any of the above aspects.

[0060] Based on any of the above aspects, the image super-resolution reconstruction method, apparatus, and electronic device provided in this application embodiment can achieve zero overlap between adjacent sub-apertures in low-resolution images acquired by a far-field reflective Fourier stacked imaging system. Compared to the traditional Fourier stacked imaging method, which requires at least approximately 35% overlap to achieve image reconstruction, this significantly reduces data acquisition time. Simultaneously, by performing degradation processing on the low-resolution image—namely, phase random translation degradation and random aberration degradation—the random phase translation in the Fourier domain can simulate the random jitter of the far-field reflective Fourier stacked imaging system, and the random aberration degradation can simulate the random interference of complex environmental factors during far-field imaging, thereby improving the generalization ability of deep learning super-resolution networks. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A schematic flowchart illustrating the image super-resolution reconstruction method provided in this application embodiment;

[0063] Figure 2A schematic diagram of the sub-steps of step S200 provided in the embodiments of this application;

[0064] Figure 3 This is a schematic diagram of the structure of the generative network provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the discriminant network provided in an embodiment of this application;

[0066] Figure 5 A schematic diagram of the sub-steps of step S300 provided in the embodiments of this application;

[0067] Figure 6 A schematic diagram of the sub-steps of step S340 provided in the embodiments of this application;

[0068] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0069] Figure 8 This is a schematic diagram of the image super-resolution reconstruction apparatus provided in the embodiments of this application.

[0070] Icons: 110 - Generative Network; 111 - Multi-scale Feature Fusion Module; 112 - Residual Module; 113 - Upsampling Module; 114 - Feature Mapping Module; 120 - Discriminant Network; 121 - Feature Extraction Module; 122 - Downsampling Module; 123 - Sample Classification Module; 800 - Electronic Device; 810 - Processor; 820 - Machine-readable Storage Medium; 830 - Image Super-resolution Reconstruction Device; 831 - Acquisition Module; 832 - Image Processing Module; 833 - Training Module. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0072] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0073] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0074] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0075] It should be noted that, where there is no conflict, different features in the embodiments of this application can be combined with each other.

[0076] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0077] Please refer to Figure 1 , Figure 1 Example: A flowchart of an image super-resolution reconstruction method provided in this embodiment is shown. The method may include the following steps.

[0078] Step S100: Based on a far-field reflective Fourier stacked imaging system, acquire a first high-resolution image and at least two first low-resolution images.

[0079] In this embodiment, the far-field reflective Fourier stacked imaging system is based on the principle of coherent diffraction imaging. This system uses a coherent light beam to illuminate the sample, causing the reflected light from the sample surface to be focused onto a photosensitive detector through an imaging lens, thereby recording a series of intensity images. Far-field reflective means that the distance between the sample and the imaging lens is much greater than the aperture of the imaging lens. The first high-resolution image can be obtained using a large-aperture camera, and the first low-resolution image can be obtained using a small-aperture camera.

[0080] Specifically, each of the first high-resolution images of the same imaging area and at least two corresponding first low-resolution images can be combined to form a sample image group. Each low-resolution image corresponds to a sub-aperture, and the overlap rate between adjacent sub-apertures of the first low-resolution images is 0, that is, there is no overlap between adjacent sub-apertures, and each sub-aperture acquires independent information. This design maximizes the utilization of the spectral information acquired by the imaging system, improving the resolution and accuracy of the Fourier layered imaging system.

[0081] For example, the first low-resolution image can be 25 images. In this embodiment, 25 first low-resolution images with an adjacent sub-aperture overlap rate of 0 and 1 first high-resolution image can be combined.

[0082] It should be noted that in step S100, multiple different first high-resolution images and first low-resolution images may be acquired to form different sample image groups. The scenes, lighting, and angles captured in different sample image groups may differ. In subsequent steps, the deep learning super-resolution network can be trained multiple times using multiple sample image groups, which can improve the generalization processing capability of the deep learning super-resolution network.

[0083] Step S200: Normalize the first high-resolution image to obtain a second high-resolution image; degrade the first low-resolution image to obtain a second low-resolution image, wherein the degradation processing includes phase random translation degradation processing and random phase difference degradation processing.

[0084] In this embodiment, the first high-resolution image in each combination obtained in step S100 can be normalized, that is, divided by the maximum gray value corresponding to the bit depth of the image data. For example, 8-bit data is divided by 255.0, and 16-bit data is divided by 65535.0, so that the pixel values ​​of the image are mapped to the range of [0,1] to obtain the second high-resolution image. This design can give the image data a uniform scale, so as to better perform subsequent processing and analysis.

[0085] By performing multiple phase random translation degradation processes on the first low-resolution image in each combination obtained in step S100, the influence of random jitter in complex environments can be better fitted. Furthermore, random aberration degradation processing based on Zernike polynomials can be performed to fit the influence of factors such as weather changes on image quality during far-field shooting.

[0086] Specifically, by combining a far-field reflective Fourier stacked imaging system with multiple phase random translation degradation processes and random aberration degradation processes based on Zernike polynomials, the dataset can be enriched, the generalization ability of deep learning networks can be enhanced, and the accuracy of image processing in complex environments can be improved.

[0087] It should be noted that during each iterative training of the deep learning super-resolution network, the first high-resolution image needs to be normalized and the first low-resolution image needs to be degraded.

[0088] Step S300: Train the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network.

[0089] In this embodiment, multiple sets of images obtained in step S200 can be input into the deep learning super-resolution network for training, and the Adam (Adaptive Moment Estimation) optimizer can be used for iterative updates to gradually reduce the loss function until the network converges. The parameters at this time are saved as the optimal network parameters, thereby obtaining a deep learning super-resolution network based on a far-field reflection Fourier stacked imaging system.

[0090] In the above design, the first low-resolution image acquired by the far-field reflective Fourier stacked imaging system can have an overlap rate of 0 between adjacent sub-apertures. Compared with the traditional Fourier stacked imaging method, which requires at least about 35% overlap to achieve image reconstruction, this reduces a significant amount of data acquisition time. At the same time, by performing degradation processing on the first low-resolution image, the random jitter of the far-field reflective Fourier stacked imaging system can be simulated, as well as the random interference of complex environmental factors during far-field imaging, thereby improving the generalization ability of the deep learning super-resolution network.

[0091] In one possible implementation, please refer to Figure 2 Step S200 may also include the following sub-steps.

[0092] Step S210: Based on the first low-resolution image, calculate the amplitude map of the first low-resolution image, and perform a Fourier transform on the amplitude map to generate a first phase map.

[0093] In this embodiment, the first low-resolution image obtained in step S100 can be normalized and then square-rooted to obtain the amplitude map of the first low-resolution image. Then, a Fourier transform can be performed to obtain the first phase map.

[0094] Specifically, the amplitude diagram A ori With the first phase diagram Φ ori It can be calculated in the following way:

[0095]

[0096] Among them, A ori This represents the amplitude graph after the Fourier transform, Φ ori This represents the first phase diagram after the Fourier transform. I represents the Fourier transform. ori This represents the intensity map before the Fourier transform, and n represents the bit depth of the first low-resolution image data.

[0097] Step S220: Based on the first phase map, perform random phase difference degradation processing and multiple random phase translation degradation processing to calculate the second phase map after introducing random perturbation.

[0098] In this embodiment, the first phase map obtained in step S210 can be randomly shifted, and then random aberrations based on Zernike polynomials can be added. The phase map is then randomly shifted again to obtain the second phase map after introducing random perturbation. Specifically, the phase map of the image contains high-level semantic information of the image, such as image content and object positions. Multiple random shifts can better fit the influence of random jitter in complex environments.

[0099] The second phase diagram Φ after introducing random perturbation new It is calculated in the following way:

[0100] Φ new =random(Φ Z *random(Φ ori ,x i1 ,y i1 ),x i2 ,y i2 )

[0101] Where, Φ new This represents the second phase diagram after introducing random perturbations, Φ Z Represents the randomly generated Zernike aberration, random(…,x i ,y i ) represents the randomly generated image translation amount (x) i ,y i That is, to move the image horizontally by x. i Vertical movement y i .

[0102] Step S230: Based on the amplitude map and the second phase map, calculate the second low-resolution image after introducing random perturbation.

[0103] In this embodiment, the amplitude map obtained in step S210 and the second phase map obtained in step S220 can be synthesized, and after performing an inverse Fourier transform, the square of the modulus is taken to obtain the second low-resolution image with introduced random perturbation.

[0104] The second low-resolution image I after introducing random perturbation new It is calculated in the following way:

[0105]

[0106] Among them, I new This represents the second low-resolution image after introducing random perturbations. This represents the inverse Fourier transform, ||…|| 2 This represents taking the square of the modulus.

[0107] In one possible implementation, the deep learning super-resolution network includes a generator network 110 and a discriminator network 120.

[0108] In this embodiment, the multiple second low-resolution images obtained in step S230 can be stitched together channel by channel and used as the input tensor of the generator network 110. The input tensor of the discriminator network 120 can be either the first output result of the generator network 110 or the second high-resolution image. The output result of the discriminator network 120 is a probability judgment of the input tensor of the discriminator network 120.

[0109] For example, the deep learning super-resolution network can be built and trained using the deep learning framework PyTorch, or it can be trained using deep learning frameworks such as TensorFlow, Keras, and MXNet. The optimizer can be Adam (Adaptive Moment Estimation), and the learning rate can be dynamically adjusted using cosine annealing. The maximum value of the learning rate is 10. -3 The minimum value is 10 -5 The number of warm-up iterations is 100.

[0110] Please refer to Figure 3 The generator network 110 may include a multi-scale feature fusion module 111, a residual module 112, an upsampling module 113, and a feature mapping module 114. The multi-scale feature fusion module 111 extracts multi-scale features from the input second low-resolution image to synthesize a first multi-channel feature map; the residual module 112 adjusts the dimensionality of the feature map to generate a second multi-channel feature map; the upsampling module 113 performs stepwise information decoding on the second multi-channel feature map to generate a third multi-channel feature map; and the feature mapping module 114 maps the third multi-channel feature map to a single-channel output image to obtain a first output result.

[0111] In this embodiment, the multi-scale feature fusion module 111 can be composed of four parallel branch structures. The first branch can be a 1×1 convolutional layer, a BatchNormal normalization layer, and a PReLU activation layer; the second branch can be a 3×3 average pooling layer, a 1×1 convolutional layer, a BatchNormal normalization layer, and a PReLU activation layer; the third branch can be a 1×1 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 1×5 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 5×1 convolutional layer, a BatchNormal normalization layer, and a PReLU activation layer; the fourth branch can be a 1×1 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 1×3 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 3×1 convolutional layer, a BatchNormal normalization layer, and a PReLU activation layer. The first feature map is passed to the residual module 112 by concatenating the outputs of the four branches channel by channel and then passing them through a 1×1 convolutional layer and a ReLU activation layer. This asymmetric convolutional splitting (1×n) structure, compared to a symmetric convolutional (n×n) structure, reduces the number of parameters while handling more and richer spatial features. The multi-scale feature fusion module 111 extracts information from different scales of the image at the same level using multiple different convolutional kernels, obtaining features from different receptive fields. After fusion, these features are passed to the next layer, allowing for a more flexible balance between computational load and model representation capability.

[0112] The generative network 110 may include multiple residual modules 112. Each residual module 112 may consist of two branches: the first branch is a 1×1 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 3×3 convolutional layer, a BatchNormal normalization layer, a PReLU activation layer, a 1×1 convolutional layer, and a BatchNormal normalization layer; the second branch is a 1×1 convolutional layer and a BatchNormal normalization layer. The outputs of the two branches are summed and then passed through a ReLU activation layer before being passed to the next residual module 112. The output of the last residual module 112 is passed through a 1×1 convolutional layer and then summed with the first feature map output by the multi-scale feature fusion module 111. The resulting second feature map is then passed to the upsampling module 113. The residual modules 112 can expand the model's representational capabilities while avoiding gradient vanishing or gradient exploding problems caused by increasing network depth. For example, the number of residual modules 112 can be 25. Specifically, the number of residual modules 112 can be adjusted according to the difficulty of training the deep learning super-resolution network.

[0113] The generator network 110 may include two upsampling modules 113, each of which may consist of a bicubic interpolation layer, a 3×3 convolutional layer, and an LReLU activation layer. Specifically, the output of the first upsampling module 113 is passed to the second upsampling module 113, and the output of the second upsampling module 113 is passed to the feature mapping module 114. The upsampling module 113 is used to progressively decode the information based on the representation results of the multi-scale feature fusion module 111 and the residual module 112, thereby improving the resolution of the output image of the generator network 110.

[0114] The feature mapping module 114 can be composed of a 7×7 convolutional layer, a ReLU activation layer, a 3×3 convolutional layer, and a tanh activation layer. The output of the tanh activation layer ranges from -1 to 1. Therefore, the output needs to be incremented by 1 and multiplied by 0.5 to obtain the final output of the generator network 110, which is the reconstructed high-resolution image after the deep learning super-resolution network training is completed. The feature mapping module 114 can map the multi-channel feature map generated by the upsampling module 113 into a single-channel output image through multiple weights.

[0115] Please refer to Figure 4The discriminant network 120 may include a feature extraction module 121, a downsampling module 122, and a sample classification module 123. The feature extraction module 121 is used to extract features from the input first output result or the second high-resolution image; the downsampling module 122 is used to reduce the number of parameters in the feature map; and the sample classification module 123 is used to calculate the classification probability to obtain the second output result.

[0116] In this embodiment, the feature extraction module 121 can be composed of a 3×3 convolutional layer, an LReLU activation layer, a 4×4 convolutional layer, a BatchNormal normalization layer, and an LReLU activation layer. The feature extraction module 121 uses a two-layer convolutional structure in series to extract multi-channel feature maps, and then passes the extracted feature maps to the downsampling module 122. For example, the feature map output by the feature extraction module 121 has 64 channels.

[0117] The discriminant network 120 may include three downsampling modules 122, which are connected in series. Specifically, each downsampling module 122 may consist of a 3×3 convolutional layer, a BatchNormal normalization layer, an LReLU activation layer, a 4×4 convolutional layer, a BatchNormal normalization layer, and an LReLU activation layer. The output of the last downsampling module 122 needs to be passed to the sample classification module 123. The downsampling modules 122 gradually reduce the number of parameters of the feature map by using a two-layer convolutional structure connected in series. Each time the feature map passes through a downsampling module 122, the number of channels doubles, that is, the number of channels of the output feature map is 128, 256, and 512 respectively, while the width and height of the feature map are halved.

[0118] The sample classification module 123 can be composed of an adaptive average pooling layer, a 1×1 convolutional layer, an LReLU activation layer, a 1×1 convolutional layer, and a sigmoid activation layer. Specifically, the spatial dimension of the feature map can be reduced to 1×1 using an adaptive average pooling algorithm, that is, the width and height of the feature map are both 1. Then, the number of channels of the feature map is gradually reduced to 1 through a 2-layer convolutional structure, and finally, the classification probability is calculated using the sigmoid function.

[0119] Specifically, the first low-resolution image and the first high-resolution image acquired by the far-field reflective Fourier layered imaging system can be used as a training set. At least two adjacent low-resolution images with zero sub-aperture overlap and one high-resolution image from the same shooting area in the training set form a sample image group. The scenes, lighting, and angles captured in different sample image groups can be different. This training set is used to train the deep learning super-resolution network. The training set may also include a validation set, which is used to evaluate the current network training performance during the training process.

[0120] It should be noted that the network depth, kernel size, and training hyperparameters of the deep learning super-resolution network need to be adjusted according to the difficulty of the training task. The number of input channels of the deep learning super-resolution network also needs to be adjusted based on the number of images acquired by the Fourier layered imaging system. For grayscale images, the number of channels is equal to the number of images.

[0121] In one possible implementation, please refer to Figure 5 Step S300 may also include the following sub-steps.

[0122] Step S310: Input the second low-resolution image into the generation network 110 to obtain the first output result.

[0123] In this embodiment, each time the deep learning super-resolution network is trained, multiple second low-resolution images can be input into the generator network 110 for forward propagation to obtain the first output result.

[0124] Step S320: Input the first output result or the second high-resolution image into the discrimination network 120 to obtain the second output result.

[0125] In this embodiment, the second high-resolution image obtained in step S200 and the first output result of the generator network 110 obtained in step S310 can be input into the discriminator network 120 to obtain the second output result.

[0126] Step S330: Based on the first output result and the second output result, calculate the loss of the generator network 110 and the loss of the discriminator network 120.

[0127] In this embodiment, the loss of the generator network 110 and the loss of the discriminator network 120 can be calculated using different formulas based on the first output result and the second output result.

[0128] Step S340: Based on the loss of the generator network 110 and the loss of the discriminator network 120, update the parameters of the deep learning super-resolution network to determine the target deep learning super-resolution network.

[0129] In this embodiment, the weight parameters of the generator network 110 and the discriminator network 120 can be updated layer by layer using the losses of the generator network 110 and the discriminator network 120. In this way, the weight parameters of the deep learning super-resolution network are continuously adjusted, enabling the generator network 110 to generate more realistic images and the discriminator network 120 to more accurately distinguish between real and generated images.

[0130] In one possible implementation, please refer to Figure 6 Step S340 may also include the following sub-steps.

[0131] Step S341: Based on the second high-resolution image and the first output result, calculate the reconstruction loss L. rec .

[0132] In this embodiment, the reconstruction loss L rec It can be calculated in the following way:

[0133]

[0134] Among them, L rec Let represent the reconstruction loss, gt represent the second high-resolution image, and G(lr) represent the first output result when the second low-resolution image is used as input to the generator network 110. represents taking the expected value of the corresponding multidimensional tensor, and ||…||2 represents taking the second norm.

[0135] Step S342: Based on the second output result, calculate the adversarial loss L. gan .

[0136] In this embodiment, the adversarial loss L gan It can be calculated in the following way:

[0137]

[0138] Among them, L gan Let represent the adversarial loss, D(gt) represent the second output result when the second high-resolution image is used as the input of the discriminant network 120, and ||…||1 represent taking the first norm.

[0139] Step S343: Based on the second high-resolution image and the first output result, calculate the first-order norm loss L of the Fourier domain phase. Φ .

[0140] In this embodiment, the first-order norm loss L of the Fourier domain phase Φ It can be calculated in the following way:

[0141]

[0142] Among them, L Φ This represents the first-order norm loss of the Fourier domain phase. This represents the Fourier transform, and angle(...) represents the phase diagram of the Fourier spectrum.

[0143] Step S344, combining reconstruction loss L rec , countering losses L gan and the first norm loss L of the Fourier domain phase Φ Calculate the total loss L of the generator network 110. total .

[0144] In this embodiment, the total loss L of the generator network 110 total It can be calculated in the following way:

[0145] L total =L rec +λ×L gan +β×L Φ

[0146] Among them, L total Let λ represent the total loss of the generator network 110, where λ and β are weight parameters, λ represents the scaling factor of the adversarial loss function, and β represents the scaling factor of the first-order norm loss function of the Fourier domain phase.

[0147] For example, the scaling factor λ of the adversarial loss function can be 0.01, and the scaling factor β of the first norm loss function of the Fourier domain phase can be 0.1.

[0148] In the above design, the loss function of the generator network 110 can be designed to improve the deep learning super-resolution network's ability to reconstruct high-level semantic information of images, such as image content and object location.

[0149] Step S345: Based on the second output result, calculate the loss L of the discriminant network 120. D .

[0150] In this embodiment, the loss L of the discrimination network 120 D It can be calculated in the following way:

[0151]

[0152] Among them, LD This indicates the determination of network 120 losses. D(G(lr)) represents the expected value of the corresponding multidimensional tensor, D(gt) represents the second output result when the first output result is used as the input of the discriminant network 120, and D(gt) represents the second output result when the second high-resolution image is used as the input of the discriminant network 120.

[0153] Step S346, based on the total loss L of the generator network 110 total and the loss L of the discriminant network 120 D Update the parameters of the generator network 110 and the discriminator network 120 until convergence.

[0154] In this embodiment, the total loss L of the generator network 110 is calculated. total and the loss L of the discriminant network 120 D The gradient of the loss function with respect to the weight parameters can be obtained, and the parameters of the generator network 110 and the discriminator network 120 can be updated using the optimizer. Training is complete when the predetermined number of training iterations is reached or the loss converges.

[0155] In one possible implementation, after training the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network, at least two images to be processed can be input into the target deep learning super-resolution network to obtain the reconstructed high-resolution image.

[0156] In this embodiment, the image to be processed can be a low-resolution image acquired by a far-field reflective Fourier stacked imaging system. The image to be processed is stitched together channel by channel to form an input tensor with the same number of input channels as the generator network 110. Through reconstruction by the target deep learning super-resolution network, a reconstructed high-resolution image can be obtained, thereby realizing image super-resolution reconstruction of sparsely sampled Fourier stacked imaging.

[0157] This embodiment also provides an electronic device 800, please refer to... Figure 7 , Figure 7The example is a block diagram of the electronic device 800. The electronic device includes a processor 810, a machine-readable storage medium 820, and an image super-resolution reconstruction apparatus 830. The machine-readable storage medium 820 and the processor 810 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The image super-resolution reconstruction apparatus 830 includes multiple software function modules that can be stored as software or firmware in the machine-readable storage medium 820 or embedded in the operating system (OS) of the image super-resolution reconstruction apparatus 830. The processor 810 is used to execute executable modules stored in the machine-readable storage medium 820, such as the software function modules and computer programs included in the image super-resolution reconstruction apparatus 830.

[0158] The machine-readable storage medium 820 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The machine-readable storage medium 820 is used to store a program, which the processor 810 executes upon receiving an execution instruction.

[0159] The processor 810 may be an integrated circuit chip with signal processing capabilities. The processor 810 may be a general-purpose processor 810, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor 810 may be a microprocessor 810, or any conventional processor 810, etc.

[0160] Please refer to Figure 8This application also provides an image super-resolution reconstruction apparatus 830, which includes multiple functional modules that can be stored in machine-readable storage medium 820 in software form. Functionally, the image super-resolution reconstruction apparatus 830 may include an acquisition module 831, an image processing module 832, and a training module 833. Wherein:

[0161] The acquisition module 831 is used in a far-field reflective Fourier layered imaging system to acquire a first high-resolution image and at least two first low-resolution images.

[0162] In this embodiment, the acquisition module 831 can be used to perform... Figure 1 For a detailed description of the acquisition module 831, please refer to the description of step S100 shown.

[0163] The image processing module 832 is used to normalize the first high-resolution image to obtain a second high-resolution image; and to degrade the first low-resolution image to obtain a second low-resolution image. The degradation processing includes phase random translation degradation processing and random phase difference degradation processing.

[0164] In this embodiment, the image processing module 832 can be used to perform... Figure 1 For a detailed description of the image processing module 832, please refer to the description of step S200 shown.

[0165] The training module 833 is used to train the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network.

[0166] In this embodiment, the training module 833 can be used to execute Figure 1 For a detailed description of the training module 833, please refer to the description of step S300 shown.

[0167] In summary, this application provides an image super-resolution reconstruction method, apparatus, and electronic device. The first low-resolution image acquired through a far-field reflective Fourier stacked imaging system can have an adjacent sub-aperture overlap rate of 0. Compared to traditional Fourier stacked imaging methods that require at least approximately 35% overlap for image reconstruction, this significantly reduces data acquisition time. Furthermore, by performing degradation processing on the first low-resolution image—namely, phase random translation degradation and random aberration degradation—the random phase translation in the Fourier domain can simulate the random jitter of the far-field reflective Fourier stacked imaging system, and the random aberration degradation can simulate the random interference from complex environmental factors during far-field imaging, thereby improving the generalization ability of deep learning super-resolution networks.

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image super-resolution reconstruction method, characterized in that, The method includes: Based on a far-field reflective Fourier stacked imaging system, acquire a first high-resolution image and at least two first low-resolution images; The first high-resolution image is normalized to obtain a second high-resolution image; the first low-resolution image is degraded to obtain a second low-resolution image, wherein the degradation processing includes phase random translation degradation processing and random phase difference degradation processing. The degradation treatment steps include: Based on the first low-resolution image, the amplitude map of the first low-resolution image is calculated, and the amplitude map is subjected to Fourier transform to generate the first phase map. The amplitude diagram With the first phase map It is calculated in the following way: ； in, This represents the magnitude plot after the Fourier transform. This represents the first phase diagram after the Fourier transform. Indicates Fourier transform, This represents the intensity map before the Fourier transform. Indicates the bit depth of the first low-resolution image data; The second phase diagram after introducing random perturbation It is calculated in the following way: ； in, This represents the second phase diagram after introducing random perturbations. This represents the randomly generated Zernike aberration. Represents the amount of random image translation. That is, to move the image horizontally. Vertical movement ; The second low-resolution image after introducing random perturbation It is calculated in the following way: ； in, This represents the second low-resolution image after introducing random perturbations. This represents the inverse Fourier transform. This indicates taking the square of the modulus. Based on the first phase diagram, random phase difference degradation processing and multiple phase random translation degradation processing are performed to calculate the second phase diagram after introducing random perturbation; Based on the amplitude map and the second phase map, calculate the second low-resolution image after introducing random perturbation; The deep learning super-resolution network is trained using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network. The deep learning super-resolution network includes a generator network and a discriminator network; The generative network includes a multi-scale feature fusion module, a residual module, an upsampling module, and a feature mapping module. The multi-scale feature fusion module extracts multi-scale features from the input second low-resolution image to synthesize a first multi-channel feature map. The residual module adjusts the dimensionality of the feature map to generate a second multi-channel feature map. The upsampling module performs stepwise information decoding on the second multi-channel feature map to generate a third multi-channel feature map. The feature mapping module maps the third multi-channel feature map to a single-channel output image to obtain a first output result. The discriminant network includes a feature extraction module, a downsampling module, and a sample classification module; the feature extraction module is used to extract features from the input first output result or the second high-resolution image; the downsampling module is used to reduce the number of parameters in the feature map; and the sample classification module is used to calculate the classification probability to obtain the second output result.

2. The image super-resolution reconstruction method according to claim 1, characterized in that, The step of training the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network includes: The second low-resolution image is input into the generation network to obtain the first output result; The first output result or the second high-resolution image is input into the discrimination network to obtain the second output result; Based on the first output result and the second output result, calculate the loss of the generator network and the loss of the discriminator network; Based on the loss of the generator network and the loss of the discriminator network, the parameters of the deep learning super-resolution network are updated to determine the target deep learning super-resolution network.

3. The image super-resolution reconstruction method according to claim 2, characterized in that, The step of updating the parameters of the deep learning super-resolution network based on the loss of the generator network and the loss of the discriminator network includes: Based on the second high-resolution image and the first output result, the reconstruction loss is calculated. ; Based on the second output, calculate the adversarial loss. ; Based on the second high-resolution image and the first output result, calculate the first-order norm loss of the Fourier domain phase. ; Combined with reconstruction losses Combating losses and the first norm loss of the Fourier domain phase Calculate the total loss of the generator network. ; Based on the second output result, the loss of the discriminant network is calculated. ; Based on the total loss of the generating network and the loss of the discriminant network Update the parameters of the generator network and the discriminator network until convergence.

4. The image super-resolution reconstruction method according to claim 3, characterized in that, The reconstruction loss It is calculated in the following way: ； in, Indicates the losses incurred during reconstruction. Indicates the second highest resolution image. This represents the first output result when the second low-resolution image is used as input to the generator network. This indicates taking the expected value of the corresponding multidimensional tensor. This indicates taking the second norm; The resistance loss It is calculated in the following way: ；； in, Indicating resistance to loss, This represents the second output result when the second high-resolution image is used as input to the discriminant network. This indicates taking the first-order norm; The first-order norm loss of the Fourier domain phase It is calculated in the following way: ； in, This represents the first-order norm loss of the Fourier domain phase. Indicates Fourier transform, This represents the phase diagram obtained from the Fourier spectrum. The total loss of the generator network It is calculated in the following way: ； in, This represents the total loss of the generator network. This represents the scaling factor of the adversarial loss function. The scaling factor representing the first-order norm loss function of the Fourier domain phase; The loss of the discriminant network It is calculated in the following way: ； in, This indicates the determination of network loss. This indicates taking the expected value of the corresponding multidimensional tensor. This represents the second output result when the first output result is used as input to the discriminant network. This represents the second output result when the second high-resolution image is used as input to the discriminant network.

5. The image super-resolution reconstruction method according to claim 1, characterized in that, After the step of training the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network, the method further includes: At least two images to be processed are input into the target deep learning super-resolution network to obtain a reconstructed high-resolution image.

6. An image super-resolution reconstruction apparatus, employing the image super-resolution reconstruction method according to any one of claims 1 to 5, characterized in that, include: Acquisition module: used in a far-field reflection-based Fourier layered imaging system to acquire a first high-resolution image and at least two first low-resolution images; Image processing module: used to normalize the first high-resolution image to obtain a second high-resolution image; The first low-resolution image is subjected to degradation processing to obtain a second low-resolution image. The degradation processing includes phase random translation degradation processing and random phase difference degradation processing. Training module: used to train the deep learning super-resolution network using the second low-resolution image and the second high-resolution image to obtain the target deep learning super-resolution network.

7. An electronic device, characterized in that, The electronic device includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions, which, when executed by the processor, implement the method according to any one of claims 1-5.

Images