An embedded micro-scanning super-resolution real-time processing system and method

Through the combination of high-speed microsweep mechanism and NVIDIA-Jetson embedded computing module, subpixel displacement image sequence acquisition and iterative solution super-resolution reconstruction of the photoelectric/infrared imaging system are realized, and poor robustness and high-frequency information in the existing technology are solved. The problem of insufficient recovery improves the resolution and processing efficiency of the image.

CN114363517BActive Publication Date: 2024-11-05IAE IND GRP CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202210019340.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-07
Publication Date
2024-11-05
Estimated Expiration
2042-01-07

AI Technical Summary

Technical Problem

The existing microscan super-resolution technology has poor real-time processing in photoelectric imaging systems, is susceptible to noise and motion errors, cannot fully recover the aliased high-frequency information, and has limited improvement to MTF transmission.

Method used

The high-speed micro-sweep mechanism is used to drive the lens/lens group/detector of the photoelectric/infrared imaging system to perform regular micro-moving, combined with the NVIDIA-Jetson embedded computing module for micro-scan super-resolution real-time processing, and restore high resolution through iterative solution algorithm. Image, realize image sequence acquisition of subpixel displacement and real-time iterative solution super-resolution reconstruction.

Benefits of technology

The MTF of the image is improved to 200%, real-time output of 30Hz super-resolution video is achieved, and the image delay is less than 60ms, which enhances the system's robustness and recovery ability of high-frequency signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114363517B_ABST
    Figure CN114363517B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of high-resolution image processing, and specifically discloses an embedded micro-scanning super-resolution real-time processing system and method, including a high-speed micro-scanning mechanism, an optoelectronic / infrared imaging system, an embedded image processing board and a subsequent device, wherein the high-speed micro-scanning mechanism includes a piezoelectric high-speed micro-scanning platform and its controller, and the piezoelectric high-speed micro-scanning platform is installed inside the imaging system; the embedded image processing board includes at least one FPGA, a GPU embedded computing module and a power management module, and the FPGA and GPU embedded computing modules are transmitted in parallel at high speed via PCIE. The present invention uses an embedded image processing board with NVIDIA‑Jetson embedded computing module as the core to perform micro-scanning super-resolution real-time processing method and system, which realizes real-time iterative solution of super-resolution reconstruction and obtains a true super-resolution image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of high-resolution image processing, and in particular to an embedded micro-scanning super-resolution real-time processing system and method. Background Art

[0002] High-resolution images have high practical value in many image application fields, such as medical diagnosis, military and defense fields, and have very important research value. In the physical world, optoelectronic imaging systems are often limited by the pixel size of the sensor and are under-sampled systems, so a lot of spatial information is submerged. Image super-resolution technology uses multiple frames of images with sub-pixel displacements to reconstruct high-resolution images. Its essence is to increase the sampling rate, that is, to increase the phase information, to improve the imaging system transfer function.

[0003] There are two ways to obtain image sequences with sub-pixel displacement: one is to use the movement or jitter of the platform itself. Taking an airplane as an example, its own flight, the jitter caused by the engine and the atmospheric turbulence will cause relative motion between image sequences; the other is to introduce a micro-scanning system of dynamic optical components to actively control the position change of the image point on the imaging detector, so that the optoelectronic imaging system can capture image sequences with sub-pixel displacement.

[0004] At present, micro-scanning super-resolution technology is widely used in foreign military / civilian optoelectronic imaging systems. However, the real-time super-resolution processing of these optoelectronic systems all uses an algorithm based on non-uniformity interpolation. The advantages of this algorithm are small computational complexity and easy real-time processing; the disadvantages are also obvious: poor robustness, easy to be affected by noise and motion errors, and limited improvement on the MTF transfer function, which cannot fully restore the aliased high-frequency information. Summary of the invention

[0005] The object of the present invention is to provide an embedded micro-scanning super-resolution real-time processing system and method to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention uses a high-speed micro-scanning mechanism to drive the lens / lens group / detector in the optoelectronic / infrared imaging system to make regular micro-movements, so that the optoelectronic / infrared imaging system captures a controllable low-resolution image sequence with sub-pixel displacement, and combines an embedded image processing board with the NVIDIA-Jetson embedded computing module as the core to perform a method and system for micro-scanning super-resolution real-time processing, thereby realizing real-time iterative solution of super-resolution reconstruction and obtaining a true super-resolution image.

[0007] The present invention provides the following technical solution: an embedded micro-scanning super-resolution real-time processing system, including a high-speed micro-scanning mechanism, a photoelectric / infrared imaging system, an embedded image processing board and a subsequent device,

[0008] The high-speed micro-scanning mechanism includes a piezoelectric high-speed micro-scanning platform and a controller thereof, the piezoelectric high-speed micro-scanning platform is installed inside the imaging system, the load of the piezoelectric high-speed micro-scanning platform includes but is not limited to the lens, lens group, reflector or detector of the photoelectric / infrared imaging system, and the piezoelectric high-speed micro-scanning platform drives the load to make regular high-speed micro-movements, so that the imaging system can collect an image sequence with sub-pixel displacement;

[0009] The controller is used for driving and high-precision position control of the high-speed micro-scanning platform, and realizes high-speed micro-movement as required under the control of the embedded image processing board;

[0010] The embedded image processing board includes at least one FPGA, one GPU embedded computing module and a power management module. The FPGA and the GPU embedded computing module are transmitted in parallel via PCIE at high speed.

[0011] FPGA is used to collect camera video. The FPGA transmits image data and instructions from the host computer to the GPU embedded computing module, and outputs the super-resolution image data processed by the algorithm sent back by the GPU embedded computing module to the downstream module;

[0012] The GPU embedded computing module can be a domestic GPU or an embedded GPU module of NVIDIA, and has a memory that can store multiple frames of images. The GPU embedded computing module performs super-resolution reconstruction processing on the input multiple frames of micro-scanning images based on a processing algorithm to obtain a high-resolution image without aliasing.

[0013] The specific algorithm processing of the GPU embedded computing module includes:

[0014] a: Sub-pixel precision and high-accuracy image registration; Micro-scanning super-resolution technology uses the sub-pixel phase difference between image sequences to restore the aliased high-frequency signals. In fact, due to platform movement or jitter, the actual motion relationship (or phase difference) between frames is not the standard rectangle followed by micro-scanning, but an irregular shape after the platform movement or jitter is superimposed on the rectangle. In order to estimate the actual displacement between image frames, it is necessary to introduce image registration technology.

[0015] b: Real-time multi-frame super-resolution reconstruction based on iterative solution method;

[0016] c: Other image enhancement techniques, including image contrast enhancement and atmospheric disturbance suppression processing.

[0017] Preferably, the FPGA serves as the main control unit of the system, responsible for the overall management of the system, the control of the high-speed micro-scanning mechanism, the control of the optoelectronic / infrared imaging system, and the synchronization of the high-speed micro-scanning mechanism and the optoelectronic / infrared imaging system.

[0018] Preferably, the interfaces of the FPGA include but are not limited to Cameralink, SDI, network port and USB.

[0019] Preferably, the specific functions of FPGA are:

[0020] (1) Camera video acquisition;

[0021] (2) Video output;

[0022] (3) System comprehensive management: a: system self-check, b: self-repair, c: enabling and disabling the GPU embedded computing module, d: monitoring the working status of the GPU embedded computing module, e: monitoring the temperature of the embedded board and the GPU embedded computing module, f: outputting a test image when there is no image output from the GPU embedded computing module;

[0023] (4) High-speed micro-scanning mechanism control;

[0024] (5) Imaging detector control;

[0025] (6) Synchronization of the high-speed micro-scanning mechanism and the imaging detector.

[0026] Preferably, the subsequent device is used to output the high-resolution image without aliasing processed by the embedded image processing board to the back-end device for further image data analysis or display.

[0027] Preferably, in order to ensure real-time performance, a large number of optimization processes are performed on the image algorithm according to the hardware characteristics of the embedded system, and the amount of calculation is reduced to achieve real-time performance while ensuring performance. The present invention also provides an embedded micro-scanning super-resolution real-time processing method, comprising the steps of:

[0028] S1: Image registration, using the pyramid optical flow method to estimate the motion of the reference image and multiple target images to obtain the displacement deviation relative to the reference image;

[0029] S2: construct an initial super-resolution image, and combine it with the relative motion information calculated in step S1 to obtain a non-uniformly interpolated high-resolution image;

[0030] S3: construct an observation model and use an iterative image reconstruction algorithm to restore the aliasing-free super-resolution image;

[0031] S4: image contrast enhancement, calculating the overall histogram characteristics of the image and determining the specific method and degree of image contrast enhancement;

[0032] S5: Remove atmospheric disturbance from the image, build an atmospheric turbulence model, and remove the distortion and blurring of the image caused by atmospheric turbulence;

[0033] S6: Output image.

[0034] Preferably, the observation model in S3 is: k =DB k M k x+n k , where B k is the image deformation matrix, M k is the blur matrix, D is the downsampling matrix, n k is the noise matrix, x is the ideal high-resolution, non-degraded image obtained by sampling continuous scenes, and y k is the k-frame low-resolution image.

[0035] Preferably, the above-mentioned embedded micro-scanning super-resolution real-time processing method realizes real-time output of 30Hz super-resolution video, and the image delay is less than 60ms.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention proposes an embedded micro-scanning super-resolution real-time processing system and method, which drives the lens / lens group / detector in the optoelectronic / infrared imaging system to make regular micro-movements through a high-speed micro-scanning mechanism, so that the optoelectronic / infrared imaging system captures a controllable low-resolution image sequence with sub-pixel displacement. The method and system for micro-scanning super-resolution real-time processing are combined with an embedded image processing board with the NVIDIA-Jetson embedded computing module as the core, realizing real-time iterative solution of super-resolution reconstruction and obtaining a true super-resolution image.

[0038] 2. The present invention adopts an iterative solution super-resolution reconstruction algorithm. This method regards super-resolution reconstruction as an inverse solution problem. First, the observation process is modeled, and important degradation factors such as deformation, blur, downsampling and noise are comprehensively considered. Then, the high-resolution image is treated as an unknown number to solve; commonly used parameter estimation methods include MLE and MAP. According to the form of the objective function, a suitable optimization method is selected, such as gradient descent method, steepest descent method and conjugate gradient method. The advantage of this method is that it can break through the detector aperture limitation and equivalently reduce the pixel aperture. Taking 2×2 micro-scanning super-resolution as an example, the algorithm can increase the MTF to 200% (assuming that it is not limited by the optical diffraction limit). Moreover, the iterative solution super-resolution reconstruction method has a very large amount of calculation, which is generally dozens or even hundreds of times that of non-uniform interpolation, and is not easy to be embedded in real time. According to the survey, there is currently no precedent for single-board embedded platform real-time processing of "image registration + iterative solution super-resolution reconstruction". Especially for the domestic market, this part is currently blank, so the applicability of the present invention in the field of micro-scanning is relatively strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a system block diagram of the present invention;

[0040] Figure 2 It is a logic block diagram of system processing in an embodiment of the present invention;

[0041] Figure 3 A circuit board architecture diagram of an embedded image processing board in an embodiment of the present invention;

[0042] Figure 4 It is a processing flow chart of FPGA and GPU embedded computing modules in an embodiment of the present invention;

[0043] Figure 5 is a uniformly interpolated high-resolution image obtained by S2 in an embodiment of the present invention;

[0044] Figure 6 This is an image enhancement benefit diagram in an embodiment of the present invention;

[0045] Figure 7 This is a diagram of the real-time processing effect in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Embodiment 1-5: The present invention provides a technical solution: an embedded micro-scanning super-resolution real-time processing system, comprising a high-speed micro-scanning mechanism, an optoelectronic / infrared imaging system, an embedded image processing board and a subsequent device, wherein the high-speed micro-scanning mechanism comprises a piezoelectric high-speed micro-scanning platform and its controller, the piezoelectric high-speed micro-scanning platform is installed inside the imaging system, the load of the piezoelectric high-speed micro-scanning platform includes but is not limited to the lens, lens group, reflector or detector of the optoelectronic / infrared imaging system, the piezoelectric high-speed micro-scanning platform drives the load to make regular high-speed micro-movements, so that the imaging system can collect an image sequence with sub-pixel displacement; the controller is used for driving and high-precision position control of the high-speed micro-scanning platform, and high-precision position control is achieved as required under the control of the embedded image processing board. The embedded image processing board includes at least one FPGA, one GPU embedded computing module and a power management module. The FPGA and the GPU embedded computing module are transmitted in parallel at high speed via PCIE. The FPGA is used to collect camera video. The FPGA transmits image data and instructions from the host computer to the GPU embedded computing module, and outputs the super-resolution image data processed by the algorithm sent back by the GPU embedded computing module to the downstream module. The GPU embedded computing module can be a domestic GPU or an embedded GPU module of NVIDIA, and has a memory that can store multiple frames of images. The GPU embedded computing module performs super-resolution reconstruction processing on the input multiple frames of micro-scanning images based on the processing algorithm to obtain high-resolution images without aliasing.

[0048] In this embodiment, a piezoelectric high-speed micro-scanning platform MS17 is used to drive the lens / lens group / detector in the optoelectronic / infrared imaging system to make regular micro-movements, so that the optoelectronic / infrared imaging system captures an image sequence with sub-pixel displacement.

[0049] In this embodiment, the high-speed micro-scanning mechanism can realize the following three types of micro-scanning, namely 2×2, 3×3 and 4×4, and the corresponding displacements between adjacent images are 1 / 2 pixel, 1 / 3 pixel and 1 / 4 pixel.

[0050] In this embodiment, the embedded image processing board uses an FPGA and a GPU embedded computing module to perform image acquisition, real-time micro-scanning super-resolution algorithm processing and video output;

[0051] In this embodiment, the GPU embedded computing module adopts NVIDIA Jetson TX2, which is an ARM+GPU supercomputing module based on Pascal architecture launched by NVIDIA.

[0052] In this embodiment, the FPGA is a Xilinx FPGA, model XC7K325T (pin-to-pin compatible with the domestic Fudan Microelectronics FPGA, model JFM7K325T); the preprocessing and forwarding of the original Cameralink image data are completed; an NVIDIA-JetsonTX2 embedded computing module completes the image data processing and Cameralink image transmission and reception.

[0053] The specific algorithm processing of the GPU embedded computing module includes:

[0054] a: Sub-pixel precision and high-accuracy image registration; Micro-scanning super-resolution technology uses the sub-pixel phase difference between image sequences to restore the aliased high-frequency signals. In fact, due to platform movement or jitter, the actual motion relationship (or phase difference) between frames is not the standard rectangle followed by micro-scanning, but an irregular shape after the platform movement or jitter is superimposed on the rectangle. In order to estimate the actual displacement between image frames, it is necessary to introduce image registration technology.

[0055] b: Real-time multi-frame super-resolution reconstruction based on iterative solution method;

[0056] c: Other image enhancement techniques, including image contrast enhancement and atmospheric disturbance suppression processing.

[0057] In this embodiment, FPGA serves as the main control unit of the system and is responsible for system comprehensive management, control of the high-speed micro-scanning mechanism, control of the optoelectronic / infrared imaging system, and synchronization of the high-speed micro-scanning mechanism and the optoelectronic / infrared imaging system.

[0058] In this embodiment, the interfaces of the FPGA include but are not limited to Cameralink, SDI, network port and USB.

[0059] In this embodiment, NVIDIA Jetson TX2 is used to process the image data sent by FPGA and transmit the processed data back to FPGA.

[0060] In this embodiment, the FPGA is responsible for receiving the image data of the detector through Cameralink, forwarding the data to the TX2 module through the PCIe interface for image data processing, and sending the image data processed by the TX2 module to the subsequent device.

[0061] In this embodiment, the subsequent device is used to output the high-resolution image without aliasing processed by the embedded image processing board to the back-end device for further image data analysis or display.

[0062] In this embodiment, the Cameralink image transceiver module equipped with the above-mentioned embedded image processing board includes a receiver and a transmitter. The image data is converted into parallel data by the receiver and directly sent to the FPGA. The receiver uses a domestic chip, the differential clock supports up to 85MHz, and the total data rate is up to 1575Mbps. The output video is output by the FPGA through the transmission converter. The transmitter uses a domestic chip, the differential clock supports up to 85MHz, and the total data rate is up to 2380Mbps.

[0063] In this embodiment, in order to ensure real-time performance, a large number of optimization processes are performed on the image algorithm according to the hardware characteristics of the embedded system, and the amount of calculation is reduced to achieve real-time performance while ensuring performance. The present invention also provides an embedded micro-scanning super-resolution real-time processing method, including the steps of:

[0064] S1: Image registration, using the pyramid optical flow method to estimate the motion of the reference image and multiple target images to obtain the displacement deviation relative to the reference image;

[0065] S2: Construct the initial super-resolution image, and combine it with the relative motion information calculated in step S1 to obtain a non-uniformly interpolated high-resolution image (such as Figure 2 );

[0066] S3: construct an observation model and use an iterative image reconstruction algorithm to restore the aliasing-free super-resolution image;

[0067] S4: image contrast enhancement, calculating the overall histogram characteristics of the image and determining the specific method and degree of image contrast enhancement;

[0068] S5: Remove atmospheric disturbance from the image, build an atmospheric turbulence model, and remove the distortion and blurring of the image caused by atmospheric turbulence;

[0069] S6: Output image.

[0070] In this embodiment, the observation model in S3 is: k =DB k M k x+n k , where B k is the image deformation matrix, M k is the blur matrix, D is the downsampling matrix, n k is the noise matrix, x is the ideal high-resolution, non-degraded image obtained by sampling continuous scenes, and y k is the k-frame low-resolution image.

[0071] See also Figure 6 , Figure 6For the image enhancement benefit of the embodiment, a high-speed micro-scanning mechanism is introduced to actively control the position change of the image point on the imaging detector, so that the collected image sequence has a controllable, regular, and effective sub-pixel displacement. The effective image information is artificially increased, so that the image super-resolution processing has a better effect. The iterative solution super-resolution reconstruction algorithm of the present invention is adopted, and the MTF can be increased to 200% for 2×2 micro-scanning super-resolution without being restricted by the optical diffraction limit.

[0072] See also Figure 7 , Figure 7 To achieve the real-time effect of the embodiment and to ensure the real-time performance of the embedded system, this embodiment has made a lot of optimizations to the image algorithm based on the hardware characteristics of the embedded system, and finally achieved real-time output of 30Hz super-resolution video with an image delay of less than 60ms.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An embedded micro-scanning super-resolution real-time processing system, characterized in that: Including high-speed micro-scanning mechanism, optoelectronic / infrared imaging system, embedded image processing board and back-end equipment, The high-speed micro-scanning mechanism includes a piezoelectric high-speed micro-scanning platform and a controller thereof, the piezoelectric high-speed micro-scanning platform is installed inside the imaging system, the load of the piezoelectric high-speed micro-scanning platform includes but is not limited to the lens, lens group, reflector or detector of the photoelectric / infrared imaging system, and the piezoelectric high-speed micro-scanning platform drives the load to make regular high-speed micro-movements, so that the imaging system can collect an image sequence with sub-pixel displacement; The controller is used for driving and high-precision position control of the high-speed micro-scanning platform, and realizes high-speed micro-movement as required under the control of the embedded image processing board; The embedded image processing board includes at least one FPGA, one GPU embedded computing module and a power management module, and the FPGA and the GPU embedded computing module are transmitted in parallel via PCIE at high speed; The FPGA is used to collect camera video, transmit image data and instructions from the host computer to the GPU embedded computing module, and output the super-resolution image data processed by the algorithm sent back by the GPU embedded computing module to the downstream module; The GPU embedded computing module has a memory capable of storing multiple frames of images. The GPU embedded computing module performs super-resolution reconstruction processing on the input multiple frames of micro-scanning images based on a processing algorithm to obtain a high-resolution image without aliasing. The specific algorithm processing of the GPU embedded computing module includes: a: Sub-pixel precision and high-accuracy image registration; micro-scanning super-resolution technology uses the sub-pixel phase difference between image sequences to restore the aliased high-frequency signals; b: Real-time multi-frame super-resolution reconstruction based on iterative solution method; c: Image enhancement technology, including image contrast enhancement and atmospheric disturbance suppression processing.

2. The embedded micro-scanning super-resolution real-time processing system according to claim 1, characterized in that: The FPGA serves as the main control unit of the system and is responsible for system comprehensive management, control of the high-speed micro-scanning mechanism, control of the optoelectronic / infrared imaging system, and synchronization of the high-speed micro-scanning mechanism and the optoelectronic / infrared imaging system.

3. The embedded micro-scanning super-resolution real-time processing system according to claim 1, characterized in that: The interfaces of the FPGA include but are not limited to Cameralink, SDI, network port and USB.

4. The embedded micro-scanning super-resolution real-time processing system according to claim 1, characterized in that: The subsequent device is used to output the high-resolution image without aliasing processed by the embedded image processing board to the back-end device for further image data analysis or display.

Citation Information

Patent Citations

  • Thermal analysis test system based on infrared sequence image super-resolution reconstruction method

    CN101320072A

  • Super-resolution imaging method based on spherical aberration aliasing observation

    CN102186016A

  • Infrared super-resolution image reconstruction method and system based on multi-core processor

    CN110111258A

  • Pixel-level multispectral and pixel-level multi-polarization-state detection resolution enhancement technology

    CN111537067A

  • Image stabilization, non-uniform correction and super-resolution system based on micro-scanning platform

    CN111664944A