Far-field reflective imaging system and image super-resolution reconstruction method and device

Abstract

The embodiment of the invention provides a far-field reflective imaging system and an image super-resolution reconstruction method and device, and relates to the technical field of image processing. The far-field reflective imaging system comprises an illumination unit, an image acquisition unit and an image processing unit, the image acquisition unit comprises a plurality of imaging subunits arranged in an array, and different imaging subunits are used for capturing non-overlapping view fields of the same scene. A plurality of different frequency spectrum images of the same scene are captured at a time through a plurality of imaging subunits arranged in an array, so that the workload of repeated imaging can be reduced, the data acquisition efficiency can be improved, more frequency spectrum information can be captured, more feature information can be provided for image reconstruction of an image processing unit, and ultrahigh-resolution reconstruction is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a far-field reflective imaging system, an image super-resolution reconstruction method and a device. Background Art

[0002] Fourier Ptychographic (FP) imaging technology is a method that uses a small-aperture camera to collect target information under different spectra through camera shift, laser light source shift, etc., and then splices the target information in the frequency domain through the phase recovery method to synthesize a large range of target spectra to achieve the imaging effect of a large-aperture camera. However, the existing technology requires multiple acquisitions of the same scene to obtain enough image acquisition data, and the data acquisition efficiency is not high. Summary of the invention

[0003] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a far-field reflective imaging system, an image super-resolution reconstruction method and an apparatus.

[0004] In a first aspect, an embodiment of the present application provides a far-field reflective imaging system, comprising an illumination unit, an image acquisition unit and an image processing unit, wherein the image acquisition unit comprises a plurality of imaging subunits arranged in an array, wherein different imaging subunits are used to capture non-overlapping fields of view of the same scene;

[0005] The lighting unit is used to provide lighting for the target object;

[0006] The image acquisition unit acquires multiple different spectrum images of the same scene at one time through multiple imaging subunits arranged in an array;

[0007] The image processing unit is used to splice a plurality of different spectrum images and reconstruct the spliced ​​spectrum images to obtain an image of target resolution.

[0008] In a second aspect, an embodiment of the present application further provides an image super-resolution reconstruction method, which is applied to an image processing unit in the far-field reflective imaging system in the first aspect, and the method comprises:

[0009] Acquire multiple different spectrum images of the same scene at one time through multiple array-arranged imaging subunits in the image acquisition unit;

[0010] splicing a plurality of the spectrum images to obtain a target input tensor;

[0011] The target input tensor is input into a pre-trained image super-resolution reconstruction model to obtain a target resolution image, wherein the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module.

[0012] In a possible implementation, the step of inputting the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image includes:

[0013] The first pyramid feature extraction module performs feature extraction on the target input tensor to obtain a first multi-scale feature map;

[0014] Calculating the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain a corresponding multi-channel feature map;

[0015] The second pyramid feature extraction module performs feature extraction on the multi-channel feature map to obtain a second multi-scale feature map;

[0016] The upsampling decoding module performs feature mapping on the second multi-scale feature map to obtain a target resolution image.

[0017] In a possible implementation, before the step of inputting the target input tensor into a pre-trained image super-resolution reconstruction model, the method further includes a step of training the image super-resolution reconstruction model, which step includes:

[0018] Acquire a training image set including a high-quality image of the same scene and a plurality of spectrum image groups, and divide the plurality of spectrum image groups in the training image set into a training image subset and a test image subset, wherein each of the spectrum image groups includes different spectrum images of the same scene acquired simultaneously by a plurality of the imaging subunits;

[0019] splicing the spectrum images in the training image subset to obtain a training input tensor;

[0020] Inputting the training input tensor into a deep learning network to be trained, wherein the deep learning network includes a first pyramid feature extraction module to be trained, a multi-scale residual module to be trained, a second pyramid feature extraction module to be trained, and an upsampling decoding module to be trained;

[0021] The first pyramid-shaped feature extraction module to be trained performs feature extraction on the training input tensor to obtain a first training multi-scale feature map;

[0022] Calculating the mean feature of the first training multi-scale feature map by the multi-scale residual module to be trained to obtain a corresponding multi-channel feature map;

[0023] The second pyramid-shaped feature extraction module to be trained performs feature extraction on the multi-channel feature map to obtain a second training multi-scale feature map;

[0024] The upsampling decoding module to be trained performs feature mapping on the second training multi-scale feature map to obtain a training target resolution image;

[0025] The loss function value of the deep learning network is calculated based on the training image set and the training target resolution image, and the model parameters in the deep learning network are iteratively updated according to the loss function value until the iteration termination condition is met, and the updated deep learning network is used as the trained image super-resolution reconstruction model.

[0026] In a possible implementation, before the step of splicing the spectrum images in the training image subset to obtain the training input tensor, the method further includes a step of preprocessing the training image set, which step includes:

[0027] Extracting regions of interest from the high-quality image and the spectrum images in the spectrum image group, and matching the high-quality image with the spectrum image group based on the extracted regions of interest;

[0028] The high-quality image and the spectrum images in the spectrum image group are normalized.

[0029] In a possible implementation, after the step of splicing the spectrum images in the training image subset in the spatial domain to obtain a training input tensor, the method further includes:

[0030] The training input tensor is spliced ​​with the spectrum image in the test image set, and a corresponding mean feature map is obtained based on the spliced ​​feature map.

[0031] In a possible implementation, the step of calculating the loss function value of the deep learning network based on the training image set and the training target resolution image, iteratively updating the model parameters in the deep learning network according to the loss function value until an iteration termination condition is met, and using the updated deep learning network as a trained image super-resolution reconstruction model includes:

[0032] Calculate the reconstruction loss function value based on the training target resolution image and the high-quality image;

[0033] Calculate and generate a loss function value based on the multi-channel feature map and the mean feature map;

[0034] An overall loss function value is obtained based on the reconstruction loss function value and the generation loss function value, and based on the overall loss function value, the model parameters in the deep learning network are iteratively updated until the iteration termination condition is met, and the updated deep learning network is used as the trained image super-resolution reconstruction model.

[0035] In a third aspect, an embodiment of the present application further provides an image super-resolution reconstruction device, characterized in that the image super-resolution reconstruction device is applied to an image processing unit in the far-field reflective imaging system described in the first aspect, and the device comprises:

[0036] An acquisition module, used for acquiring a plurality of different spectrum images of the same scene at one time through the image acquisition unit;

[0037] A splicing module, used for splicing a plurality of the spectrum images to obtain a target input tensor;

[0038] The reconstruction module is used to input the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image.

[0039] In a possible implementation, the reconstruction module further includes:

[0040] An input module, used for inputting the target input tensor into a pre-trained image super-resolution reconstruction model, wherein the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module;

[0041] A first feature extraction module, configured to extract features from the target input tensor by using the first pyramid feature extraction module to obtain a first multi-scale feature map;

[0042] A residual module, used for calculating the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain a corresponding multi-channel feature map;

[0043] A second feature extraction module, configured to extract features from the multi-channel feature map by using the second pyramid feature extraction module to obtain a second multi-scale feature map;

[0044] A mapping module is used to perform feature mapping on the second multi-scale feature map by the upsampling decoding module to obtain a target resolution image.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of the second aspects is implemented.

[0046] Based on any of the above aspects, the far-field reflective imaging system, image super-resolution reconstruction method and device provided by the embodiments of the present application include an illumination unit, an image acquisition unit and an image processing unit, wherein the image acquisition unit includes a plurality of imaging sub-units arranged in an array, and different imaging sub-units are used to capture non-overlapping fields of view of the same scene. By capturing multiple different spectral images of the same scene at one time through multiple imaging sub-units arranged in an array, not only can the workload of repeated imaging be reduced and the efficiency of data acquisition be improved, but more spectral information can also be captured, providing more feature information for the image processing unit to perform image reconstruction, thereby achieving ultra-high resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A schematic diagram of a module of a far-field reflective imaging system provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of an arrangement of imaging subunits provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a process for super-resolution reconstruction of an image provided by an embodiment of the present application;

[0051] Figure 4 for Figure 3 Schematic diagram of the sub-step flow chart of step S230;

[0052] Figure 5 for Figure 3 A possible flow chart before step S210;

[0053] Figure 6 A comparison diagram of the depth model image reconstruction before and after provided in the embodiment of the present application Figure 1 ;

[0054] Figure 7 A comparison diagram of the depth model image reconstruction before and after provided in the embodiment of the present application Figure 2 ;

[0055] Figure 8 for Figure 5 A schematic diagram of a sub-step flow chart of step S183;

[0056] Fig. 9A functional module diagram of an image super-resolution reconstruction model provided in an embodiment of the present application;

[0057] Fig.10 A functional module diagram of an image super-resolution reconstruction model provided in an embodiment of the present application;

[0058] Fig.11 A schematic diagram of the hardware structure of the image processing unit provided in this application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0062] In the description of this application, it should be noted that the terms "upper", "lower", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the application is usually placed when in use, which is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0063] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0064] It should be noted that, in the absence of conflict, different features in the embodiments of the present application may be combined with each other.

[0065] To solve the problems in the prior art, please refer to Figure 1 and Figure 2 , the embodiment of the present application provides a far-field reflective imaging system 100, including an illumination unit 110, an image acquisition unit 120 and an image processing unit 130, wherein the image acquisition unit 120 includes a plurality of imaging subunits 121 arranged in an array, wherein different imaging subunits 121 are used to capture non-overlapping fields of view of the same scene, that is, the scene portions captured by each imaging subunit 121 have no spatial intersection. For example, please refer to Figure 2 The image acquisition unit 120 may include nine imaging subunits 121, and the nine imaging subunits 121 are arranged in an array.

[0066] The lighting unit 110 can provide illumination for the target object, and the image acquisition unit 120 obtains multiple different spectral images of the same scene at one time through multiple array-arranged imaging subunits 121, wherein different imaging subunits 121 can simultaneously obtain different spectral images of the same scene, that is, each imaging subunit 121 can simultaneously capture spectral images of different light rays at different angles, providing more feature information for subsequent image reconstruction.

[0067] The image processing unit 130 can splice multiple different spectrum images and reconstruct the spliced ​​spectrum images. That is, the image processing unit 130 can combine the feature information in different spectrum images to provide richer feature information for image reconstruction, obtain the target resolution image, and achieve ultra-high resolution reconstruction.

[0068] In this step, multiple different spectral images of the same scene are captured at one time by using multiple array-arranged imaging subunits 121, which can not only reduce the workload of repeated imaging and improve the efficiency of data acquisition, but also capture more spectral information, provide more feature information for the image processing unit 130 to perform image reconstruction, and achieve ultra-high resolution reconstruction.

[0069] Based on the same inventive concept, the present application also provides an image super-resolution reconstruction method for the image processing unit 130 in the far-field reflective imaging system 100. Figure 3 , Figure 3 Schematic diagram of the process of image super-resolution reconstruction method. Figure 3 Each step of the image super-resolution reconstruction method is described in detail.

[0070] Step S210: acquiring a plurality of different spectrum images of the same scene at one time through a plurality of array-arranged imaging subunits in the image acquisition unit.

[0071] In this step, multiple different spectral images of the same scene are acquired at one time through multiple array-arranged imaging subunits 121, which can not only capture spectral images of different light rays and angles to provide more feature information for subsequent image reconstruction, but also reduce the workload of repeated imaging and improve the efficiency of data acquisition.

[0072] Step S220: splicing multiple spectrum images to obtain a target input tensor.

[0073] In this step, multiple spectrum images may be spliced ​​batch by batch to integrate feature information of the multiple spectrum images.

[0074] Step S230: input the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image.

[0075] In this step, the pre-trained image super-resolution reconstruction model can obtain the feature information of the target input tensor, and reconstruct it based on the feature information of the target input tensor to obtain a super-resolution reconstructed image, that is, a target resolution image.

[0076] It should be noted that the image super-resolution reconstruction model can fit the sampling process of the array imaging system, and it can also effectively enhance the generalization ability of the model when the overlap rate of the spectrum image is low or even zero. In addition, in step S220, batch-by-batch splicing of multiple spectrum images can decouple the array arrangement of the imaging subunit 121, so that the image super-resolution reconstruction model can perform super-resolution reconstruction on multiple frames of spectrum images.

[0077] In this embodiment, by capturing multiple different spectrum images of the same scene at one time through multiple array-arranged imaging subunits 121 in the image acquisition unit 120, not only can the workload of repeated imaging be reduced, the efficiency of data acquisition be improved, but more spectrum information can also be captured. At the same time, by splicing multiple spectrum images, not only can the array arrangement of the imaging subunits 121 be decoupled, but also more abundant feature information can be provided for subsequent image reconstruction, effectively reducing the background noise of the image and achieving ultra-high resolution reconstruction.

[0078] The inventors found that the traditional Fourier stack imaging method failed to combine the physical model of array-type far-field reflection Fourier stack imaging for network structure design, and faced problems such as insufficient model generalization ability. In addition, the traditional Fourier stack imaging method has certain requirements for image overlap rate, requiring at least about 35% overlap rate to achieve image reconstruction, and 60%-80% overlap rate to obtain better reconstruction results, which greatly increases the workload and time of data collection.

[0079] To solve the above problems, the image super-resolution reconstruction model provided by the present application may include a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module. Figure 4 , Figure 4 This is a possible sub-step flow chart of step S230. Figure 4 Step S230 will be described in detail.

[0080] Step S231: The first pyramid feature extraction module performs feature extraction on the target input tensor to obtain a first multi-scale feature map.

[0081] In this step, the first pyramid feature extraction module is composed of five partially overlapping image block embedding modules and five transformer modules alternating, so that low-level to high-level feature representations can be effectively extracted, that is, the first multi-scale feature map includes global features and local features of the target input tensor.

[0082] Step S232: Calculate the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain a corresponding multi-channel feature map.

[0083] In this step, the multi-scale residual module is composed of five double upsampling convolutional layers and five residual modules alternately. Residual learning is used to extract and optimize features and fuse feature maps of different scales.

[0084] Step S233: The second pyramid feature extraction module performs feature extraction on the multi-channel feature map to obtain a second multi-scale feature map.

[0085] In this step, the second pyramid-type feature extraction module is composed of five partially overlapping image block embedding modules and five transformer modules alternatingly, which can further extract and refine features. Among them, the stacking depth of the five transformer modules in the second pyramid-type feature extraction module can be different from the stacking depth of the five transformer modules in the first pyramid-type feature extraction module to adapt to different levels of feature extraction requirements and improve the quality of the reconstructed image.

[0086] Step S234: The upsampling decoding module performs feature mapping on the second multi-scale feature map to obtain a target resolution image.

[0087] In this step, the upsampling decoding module consists of a multi-scale residual module, an upsampling module and a feature mapping module, which is used to gradually increase the resolution of the feature map and map the multi-channel feature map into a single-channel high-resolution image, that is, the target resolution image.

[0088] In this embodiment, the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module. The design of these modules fully combines the physical characteristics and data characteristics of the array-arranged imaging subunits 121, thereby effectively fitting the sampling process of the array imaging system, so that it has a strong generalization ability even when the spectrum image overlap rate is low or even zero, thereby reducing the difficulty of data acquisition and improving acquisition efficiency.

[0089] In some preferred embodiments, the step of training the image super-resolution reconstruction model may be included before step S210. Figure 5 , Figure 5 Schematic diagram of the process of image super-resolution reconstruction method. Figure 5 The steps of training the image super-resolution reconstruction model are described in detail.

[0090] Step S110: obtaining a training image set including a high-quality image of the same scene and a plurality of spectrum image groups, and dividing the plurality of spectrum image groups in the training image set into a training image subset and a test image subset.

[0091] In this step, different scenes may be first captured by a large aperture lens (whose aperture is at least twice that of the imaging subunit 121) to obtain high-quality images, and then multiple low-resolution spectrum image groups of different scenes may be acquired by the image acquisition unit 120. The imaging result of multiple array-arranged imaging subunits 121 at one time may be used as a spectrum image group, that is, each spectrum image group may include different spectrum images of the same scene acquired simultaneously by multiple imaging subunits 121.

[0092] Then, according to the parameters calibrated by the image acquisition unit 120, the region of interest of the high-quality image and the spectrum image in the spectrum image group is extracted by an edge detection algorithm (such as Canny, Sobel, Prewitt), and the high-quality image and the spectrum image group are paired based on the region of interest at the extraction location. Next, the high-quality image and the spectrum image in the spectrum image group are normalized, that is, divided by the maximum grayscale value corresponding to the bit depth of the image data, for example, 8-bit data is divided by 255.0, and 16-bit data is divided by 65535.0, so that the pixel value of the image is mapped to the range of [0,1]. Such a design can make the image data have a unified scale, so as to better perform subsequent processing and analysis.

[0093] Finally, the multiple spectrum image groups are divided into training image subsets, test image subsets and verification image subsets in the training image set. The verification image subset is used to judge the current network training effect during the training process. Specifically, a group of spectrum image groups in the training image set can be arbitrarily selected as the training image subset, and a group of spectrum image groups can be arbitrarily selected from other spectrum image groups as the verification image subset, and the other spectrum image groups can be used as the training image subset.

[0094] Exemplarily, when the image acquisition unit 120 includes nine imaging subunits 121 arranged in an array, the same scene can be acquired ten times to obtain ten spectrum image groups, each spectrum image group including nine different spectrum images. One spectrum image group can be arbitrarily selected as a training image subset, and eight spectrum image groups can be arbitrarily selected from the other nine spectrum image groups as training image subsets, and finally one spectrum image group is left as a verification image subset.

[0095] Step S120: splicing the spectrum images in a training image subset to obtain a training input tensor.

[0096] In this step, in order to integrate the feature information of different spectrum images in the training image subset, the spectrum images in the training image subset can be spliced ​​batch by batch to obtain the training input tensor.

[0097] Next, the training input tensor and the spectrum image in the test image set can be concatenated batch by batch and channel by channel, and the corresponding mean feature map can be obtained based on the concatenated feature map.

[0098] Step S130: Input the training input tensor into the deep learning network to be trained.

[0099] In this step, the deep learning network includes a first pyramid feature extraction module to be trained, a multi-scale residual module to be trained, a second pyramid feature extraction module to be trained, and an upsampling decoding module to be trained.

[0100] Step S140: The first pyramid feature extraction module to be trained performs feature extraction on the training input tensor to obtain a first training multi-scale feature map.

[0101] In this step, the first pyramid feature extraction module to be trained is composed of five partially overlapping image block embedding modules and five transformer modules alternately, each partially overlapping image block embedding module is composed of a double downsampling convolution layer and a LayerNormal normalization layer, and each transformer module is composed of a plurality of multi-head attention mechanism layers stacked, and the stacking depths of the five transformer modules are 3, 3, 6, 8, and 3, respectively, and each multi-head attention mechanism layer uses a pre-normalization layer to reduce signal blocking. Among them, in order to reduce the computational amount and memory overhead of the network, the cosine attention mechanism and the space reduction attention mechanism can be introduced when the transformer module calculates Q (Queries), K (keys), and V (values).

[0102] Step S150: Calculate the mean feature of the first training multi-scale feature map by the multi-scale residual module to be trained to obtain a corresponding multi-channel feature map.

[0103] In this step, the multi-scale residual module to be trained is composed of five double upsampling convolutional layers and five residual modules alternately, which are used to gradually increase the spatial resolution of the image, and use residual learning to extract and optimize features. Among them, the input of the first double upsampling convolutional layer can be the output tensor of the last transformer module in the first pyramid feature extraction module, and the output tensors of the remaining transformer modules in the first pyramid feature extraction module will be superimposed with the output tensors of the residual modules with the same spatial scale of the multi-scale residual module as the input of the next double upsampling convolutional layer. In addition, the output tensor of the last residual module will also be averaged in the batch dimension and channel dimension to obtain the mean features across the batch dimension and channel dimension.

[0104] Step S160: The second pyramid feature extraction module to be trained performs feature extraction on the multi-channel feature map to obtain a second training multi-scale feature map.

[0105] In this step, the second pyramid feature extraction module to be trained is composed of five partially overlapping image block embedding modules and five transformer modules alternating, and the stacking depths of the five transformer modules are 3, 3, 4, 6, and 3, respectively. Super-resolution reconstruction is achieved while reducing the number of module parameters, redundant network calculations, and memory overhead.

[0106] Step S170: The upsampling decoding module to be trained performs feature mapping on the second training multi-scale feature map to obtain a training target resolution image.

[0107] In this step, the upsampling decoding module to be trained is composed of a multi-scale residual module, an upsampling module and a feature mapping module. The spatial dimension of the multi-scale residual module is consistent with the spatial dimension of the output tensor of the corresponding transformer module of the transformer module in the second pyramid feature extraction module, and the output tensor of the multi-scale residual module is upsampled twice by the width and height of the image through the upsampling module. The feature mapping module is composed of a residual module, a convolutional layer and a Tanh activation layer. The module maps the multi-channel feature map generated by the upsampling module into a single-channel high-resolution image, i.e., the training target resolution image, through multiple layers of weights, and compresses the value range to [0,1].

[0108] Step S180: Calculate the loss function value of the deep learning network based on the training image set and the training target resolution image, and iteratively update the model parameters in the deep learning network according to the loss function value until the iteration termination condition is met, and use the updated deep learning network as the trained image super-resolution reconstruction model.

[0109] In this step, when the model parameters in the deep learning model are iteratively updated according to the loss function value, the iteration can be stopped when the loss function value is less than the preset loss function threshold value or when the number of iterations reaches the preset number. Exemplarily, when the model parameters in the deep learning model are iteratively updated according to the loss function value, the iteration can be stopped when the number of iterations reaches 2000, where please refer to Figure 6 and Figure 7 , Figure 6 and Figure 7 The left side is a low-resolution image captured by the imaging subunit 121, and the right side is a reconstruction effect diagram obtained after the model is trained 2000 times.

[0110] It should be noted that this application can use deep learning network frameworks such as PyTorch, TensorFlow, Keras, and MXNet for training, and use the optimizer of the adaptive learning rate optimization algorithm for iterative updates, gradually reducing the loss function value until the network converges. Exemplarily, the optimizer includes but is not limited to the Adam (Adaptive Moment Estimation) optimizer. The learning rate is dynamically adjusted using the cosine annealing method, and the range of the learning rate can be (10 -5 ,10 -3 ), the number of warm-up iterations is 100.

[0111] For further information, please refer to Figure 8, step S180 can be implemented by the following method.

[0112] Step S181: Calculate the reconstruction loss function value based on the training target resolution image and the high-quality image.

[0113] Specifically, the reconstruction loss function value can be calculated by the following formula:

[0114]

[0115] Among them, L rec represents the reconstruction loss function value, hr represents the high-quality image, D(lr N ) represents the training target resolution image output by the upsampling decoding module, E[*] represents the expected value of the corresponding multidimensional tensor, and ||x||2 represents the second-order norm.

[0116] Step S182: Calculate and generate a loss function value based on the multi-channel feature map and the mean feature map.

[0117] Among them, the loss function value can be calculated by the following formula:

[0118]

[0119] Among them, L g Indicates the generation of loss function value, concat(lr N ,lr NX8 ) represents the mean feature map, R(lr N ) represents the multi-channel feature map output by the multi-scale residual module, It means taking the expected value of the corresponding multidimensional tensor, and ||*||2 means taking the second-order norm.

[0120] Step S183: Obtain an overall loss function value based on the reconstruction loss function value and the generation loss function value, and iteratively update the model parameters in the deep learning network based on the overall loss function value until the iteration termination condition is met, and use the updated deep learning network as the trained image super-resolution reconstruction model.

[0121] Among them, the overall loss function value can be calculated by the following formula:

[0122] L total =L rec +λ*L g

[0123] Among them, L total represents the overall loss function value, λ represents the scaling factor of the generation loss function of the spectrum image collected by the imaging subunit 121, and preferably, the scaling factor is 0.01.

[0124] In this embodiment, the model of the deep learning network is iteratively updated through the composite overall loss function value, so that the network can automatically extract and enhance the feature information of the spectrum image, effectively fit the array sampling mechanism of the imaging subunit 121, thereby enhancing the generalization ability of the model and realizing super-resolution image reconstruction.

[0125] Based on the same inventive concept, the present application also provides an image super-resolution reconstruction device 200, please refer to 9, Fig. 9 A schematic diagram of the functional modules of the image super-resolution reconstruction device 200 provided in the embodiment of the present application. The embodiment of the present application can divide the functional modules of the image super-resolution reconstruction device 200 according to the method embodiment executed by the computer device, that is, the following functional modules corresponding to the image super-resolution reconstruction device 200 can be used to execute the above-mentioned method embodiments. Among them, the image super-resolution reconstruction device 200 can be applied to the image processing unit 130 in the far-field reflective imaging system 100, including an acquisition module 210, a stitching module 220 and a reconstruction module 230. The functions of each functional module of the image super-resolution reconstruction device 200 are described in detail below.

[0126] The acquisition module 210 is used to acquire a plurality of different spectrum images of the same scene at one time through the image acquisition unit 120 .

[0127] Acquiring multiple different spectral images of the same scene at one time through multiple array-arranged imaging subunits 121 can not only capture spectral images of different light rays at different angles to provide more feature information for subsequent image reconstruction, but also reduce the workload of repeated imaging and improve the efficiency of data acquisition.

[0128] In this embodiment, the acquisition module 210 may be used to execute the above step S210. The detailed implementation of the acquisition module 210 may refer to the above detailed description of step S210.

[0129] The splicing module 220 is used to splice the multiple spectrum images to obtain a target input tensor.

[0130] Multiple spectrum images can be spliced ​​batch by batch to integrate the feature information of multiple spectrum images.

[0131] In this embodiment, the splicing module 220 can be used to execute the above step S220. The detailed implementation of the splicing module 220 can refer to the detailed description of the above step S220.

[0132] The reconstruction module 230 is used to input the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image.

[0133] The pre-trained image super-resolution reconstruction model can obtain the feature information of the target input tensor, and reconstruct it based on the feature information of the target input tensor to obtain a super-resolution reconstructed image, that is, a target resolution image.

[0134] In this embodiment, the reconstruction module 230 may be used to execute the above step S230. The detailed implementation of the reconstruction module 230 may refer to the above detailed description of step S230.

[0135] In this embodiment, the acquisition module 210 captures multiple different spectrum images of the same scene at one time through multiple array-arranged imaging subunits 121 in the image acquisition unit 120, which can not only reduce the workload of repeated imaging, improve the efficiency of data acquisition, but also capture more spectrum information. At the same time, the splicing module 220 splices multiple spectrum images, which can not only decouple the array arrangement of the imaging subunits 121, but also provide richer feature information when the reconstruction module 230 reconstructs the image, effectively reduce the background noise of the image, and achieve ultra-high resolution reconstruction.

[0136] Furthermore, the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module. Fig.10 The reconstruction module 230 may further include a first feature extraction module 231 , a residual module 232 , a second feature extraction module 233 and a mapping module 234 .

[0137] The first feature extraction module 231 is used to perform feature extraction on the target input tensor by using the first pyramid feature extraction module to obtain a first multi-scale feature map.

[0138] The first pyramid feature extraction module is composed of five partially overlapping image block embedding modules and five transformer modules alternating, so that low-level to high-level feature representations can be effectively extracted, that is, the first multi-scale feature map includes global features and local features of the target input tensor.

[0139] In this embodiment, the first feature extraction module 231 may be used to execute the above step S231 . The detailed implementation of the first feature extraction module 231 may refer to the above detailed description of step S231 .

[0140] The residual module 232 is used to calculate the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain the corresponding multi-channel feature map.

[0141] The multi-scale residual module is composed of five double upsampling convolutional layers and five residual modules alternatingly. It uses residual learning to extract and optimize features and fuse feature maps of different scales.

[0142] In this embodiment, the residual module 232 can be used to execute the above step S232. The detailed implementation of the residual module 232 can refer to the detailed description of the above step S232.

[0143] The second feature extraction module 233 is used to perform feature extraction on the multi-channel feature map by using the second pyramid feature extraction module to obtain a second multi-scale feature map.

[0144] The second pyramid-shaped feature extraction module is composed of five partially overlapping image block embedding modules and five transformer modules alternating with each other, which can further extract and refine features. The stacking depth of the five transformer modules in the second pyramid-shaped feature extraction module can be different from the stacking depth of the five transformer modules in the first pyramid-shaped feature extraction module to adapt to different levels of feature extraction requirements and improve the quality of the reconstructed image.

[0145] In this embodiment, the second feature extraction module 233 may be used to execute the above step S233. The detailed implementation of the second feature extraction module 233 may refer to the above detailed description of step S233.

[0146] The mapping module 234 is configured to perform feature mapping on the second multi-scale feature map by the upsampling decoding module to obtain a target resolution image.

[0147] The upsampling decoding module consists of a multi-scale residual module, an upsampling module and a feature mapping module, which is used to gradually increase the resolution of the feature map and map the multi-channel feature map into a single-channel high-resolution image, that is, the target resolution image.

[0148] In this embodiment, the mapping module 234 may be used to execute the above step S234. The detailed implementation of the mapping module 234 may refer to the above detailed description of step S234.

[0149] It should be noted that it should be understood that the division of the various modules in the above device or system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software (for example, open source software) that can be called by the processor; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by the processor, and some modules can be implemented in the form of hardware. As an example, the mapping module 234 can be implemented by running a separate processor, and can be stored in the memory of the above device or system in the form of program code. The functions of the above mapping module 234 are called and executed by a processor of the above device or system. The implementation of other modules is similar to it, and it will not be repeated here. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processor described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or each module in the above technical solution can be completed by an integrated logic circuit in the processor or by executing a software program.

[0150] Please refer to Fig.11 , Fig.11 FIG. 1 is a schematic diagram showing the hardware structure of the image processing unit 130 for implementing the above-mentioned image super-resolution reconstruction method provided by the embodiment of the present disclosure. Fig.11 As shown, the image processing unit 130 may include a processor 131 , a computer-readable storage medium 132 , a bus 133 , and a communication unit 134 .

[0151] In a specific implementation process, the processor 131 executes computer-executable instructions stored in a computer-readable storage medium (eg, Fig.10 The various modules in the image quality enhancement device shown in ), so that the processor 131 can execute the image quality enhancement method of the above method embodiment, wherein the processor 131, the computer readable storage medium 132 and the communication unit 134 can be connected through the bus 133.

[0152] The specific implementation process of the processor 131 can be found in the various method embodiments executed by the above-mentioned computer device. The implementation principles and technical effects are similar, and the embodiments of the present application will not be repeated here.

[0153] The computer readable storage medium 132 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store programs or data.

[0154] The bus 133 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

[0155] In the interaction scenario provided in the embodiment of the present application, the communication unit 134 may be used to communicate with other devices to implement data interaction between the image processing unit 130 and other devices.

[0156] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned image super-resolution reconstruction method is implemented.

[0157] In summary, the embodiments of the present application provide a far-field reflective imaging system, an image super-resolution reconstruction method and device, wherein the far-field reflective imaging system includes an illumination unit, an image acquisition unit and an image processing unit, wherein the image acquisition unit includes a plurality of imaging sub-units arranged in an array, and different imaging sub-units are used to capture non-overlapping fields of view of the same scene. By capturing multiple different spectral images of the same scene at one time through a plurality of imaging sub-units arranged in an array, not only can the workload of repeated imaging be reduced and the efficiency of data acquisition be improved, but more spectral information can also be captured, providing more feature information for the image processing unit to perform image reconstruction, thereby achieving ultra-high resolution reconstruction.

[0158] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A far-field reflective imaging system, characterized in that: It comprises an illumination unit, an image acquisition unit and an image processing unit, wherein the image acquisition unit comprises a plurality of imaging subunits arranged in an array, wherein different imaging subunits are used to capture non-overlapping fields of view of the same scene; The lighting unit is used to provide lighting for the target object; The image acquisition unit acquires multiple different spectrum images of the same scene at one time through multiple imaging subunits arranged in an array; The image processing unit is used to splice a plurality of different spectrum images and reconstruct the spliced ​​spectrum images to obtain an image of target resolution.

2. A method for image super-resolution reconstruction, characterized in that: The image processing unit applied to the far-field reflective imaging system according to any one of claims 1, the method comprising: Acquire multiple different spectrum images of the same scene at one time through multiple array-arranged imaging subunits in the image acquisition unit; splicing a plurality of the spectrum images to obtain a target input tensor; The target input tensor is input into a pre-trained image super-resolution reconstruction model to obtain a target resolution image, wherein the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module.

3. The image super-resolution reconstruction method according to claim 2, characterized in that: The step of inputting the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image comprises: The first pyramid feature extraction module performs feature extraction on the target input tensor to obtain a first multi-scale feature map; Calculating the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain a corresponding multi-channel feature map; The second pyramid feature extraction module performs feature extraction on the multi-channel feature map to obtain a second multi-scale feature map; The upsampling decoding module performs feature mapping on the second multi-scale feature map to obtain a target resolution image.

4. The image super-resolution reconstruction method according to claim 3, characterized in that: Before the step of inputting the target input tensor into a pre-trained image super-resolution reconstruction model, the method further includes a step of training the image super-resolution reconstruction model, which step includes: Acquire a training image set including a high-quality image of the same scene and a plurality of spectrum image groups, and divide the plurality of spectrum image groups in the training image set into a training image subset and a test image subset, wherein each of the spectrum image groups includes different spectrum images of the same scene acquired simultaneously by a plurality of the imaging subunits; splicing the spectrum images in the training image subset to obtain a training input tensor; Inputting the training input tensor into a deep learning network to be trained, wherein the deep learning network includes a first pyramid feature extraction module to be trained, a multi-scale residual module to be trained, a second pyramid feature extraction module to be trained, and an upsampling decoding module to be trained; The first pyramid-shaped feature extraction module to be trained performs feature extraction on the training input tensor to obtain a first training multi-scale feature map; Calculating the mean feature of the first training multi-scale feature map by the multi-scale residual module to be trained to obtain a corresponding multi-channel feature map; The second pyramid-shaped feature extraction module to be trained performs feature extraction on the multi-channel feature map to obtain a second training multi-scale feature map; The upsampling decoding module to be trained performs feature mapping on the second training multi-scale feature map to obtain a training target resolution image; The loss function value of the deep learning network is calculated based on the training image set and the training target resolution image, and the model parameters in the deep learning network are iteratively updated according to the loss function value until the iteration termination condition is met, and the updated deep learning network is used as the trained image super-resolution reconstruction model.

5. The image super-resolution reconstruction method according to claim 4, characterized in that: Before the step of splicing the spectrum images in the training image subset to obtain the training input tensor, the method further includes the step of preprocessing the training image set, which step includes: Extracting regions of interest from the high-quality image and the spectrum images in the spectrum image group, and matching the high-quality image with the spectrum image group based on the extracted regions of interest; The high-quality image and the spectrum images in the spectrum image group are normalized.

6. The image super-resolution reconstruction method according to claim 5, characterized in that: After the step of splicing the spectrum images in the training image subset in the spatial domain to obtain the training input tensor, the method further includes: The training input tensor is spliced ​​with the spectrum image in the test image set, and a corresponding mean feature map is obtained based on the spliced ​​feature map.

7. The image super-resolution reconstruction method according to claim 6, characterized in that: The step of calculating the loss function value of the deep learning network based on the training image set and the training target resolution image, iteratively updating the model parameters in the deep learning network according to the loss function value until an iteration termination condition is met, and using the updated deep learning network as the trained image super-resolution reconstruction model includes: Calculate the reconstruction loss function value based on the training target resolution image and the high-quality image; Calculate and generate a loss function value based on the multi-channel feature map and the mean feature map; An overall loss function value is obtained based on the reconstruction loss function value and the generation loss function value, and based on the overall loss function value, the model parameters in the deep learning network are iteratively updated until the iteration termination condition is met, and the updated deep learning network is used as the trained image super-resolution reconstruction model.

8. An image super-resolution reconstruction device, characterized in that: The image super-resolution reconstruction device is applied to the image processing unit in the far-field reflective imaging system according to claim 1, and the device comprises: An acquisition module, used for acquiring a plurality of different spectrum images of the same scene at one time through the image acquisition unit; A splicing module, used for splicing a plurality of the spectrum images to obtain a target input tensor; The reconstruction module is used to input the target input tensor into a pre-trained image super-resolution reconstruction model to obtain a target resolution image.

9. The image super-resolution reconstruction device according to claim 8, characterized in that: The reconstruction module also includes: An input module, used for inputting the target input tensor into a pre-trained image super-resolution reconstruction model, wherein the image super-resolution reconstruction model includes a first pyramid feature extraction module, a multi-scale residual module, a second pyramid feature extraction module and an upsampling decoding module; A first feature extraction module, configured to extract features from the target input tensor by using the first pyramid feature extraction module to obtain a first multi-scale feature map; A residual module, used for calculating the mean feature of the first multi-scale feature map by the multi-scale residual module to obtain a corresponding multi-channel feature map; A second feature extraction module, configured to extract features from the multi-channel feature map by using the second pyramid feature extraction module to obtain a second multi-scale feature map; A mapping module is used to perform feature mapping on the second multi-scale feature map by the upsampling decoding module to obtain a target resolution image.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of claims 2 to 8 is implemented.

Images