A multi-channel image fusion system and method based on inertial space alignment

Abstract

The present invention relates to a multi-channel image fusion system and method based on inertial space alignment. The system comprises: at least two optoelectronic imaging devices, an inertial measurement unit (IMU), and an image synchronization acquisition and processing device. The IMU is used to measure the dynamic change in the optical axis orientation of the corresponding optoelectronic imaging device. The image synchronization acquisition and processing device sends a synchronized external trigger signal to each optoelectronic imaging device. After receiving the external trigger signal, each optoelectronic imaging device synchronously acquires an image and transmits the acquired image and information collected by the corresponding IMU back to the image synchronization acquisition and processing device. The image synchronization acquisition and processing device performs inertial space alignment on each image based on the information collected by the IMU and then fuses the images. The system can effectively solve the problems of inconsistent dynamic and time bases between multiple devices. The background synchronously acquires multiple detector images and then performs image fusion to improve the signal-to-noise ratio.

Description

Technical Field

[0001] The present invention relates to the field of photoelectric detection technology, and in particular to a multi-channel image fusion system and method based on inertial space alignment. Background Art

[0002] Infrared and visible light optoelectronic detection equipment is widely used in defense and industry. Their ability to detect and reconnaissance incoming aircraft, drones, missiles, and guided bombs in electromagnetically silent environments is crucial. This is especially true for high-speed, low-altitude targets. The longer the detection distance, the more time defensive weapons have to maneuver, increasing the probability of a defensive response.

[0003] Given a certain level of detector performance, there are several ways to improve target detection range and probability. One approach is hardware: increasing the optical system's aperture to improve light collection capabilities. However, a larger aperture increases manufacturing complexity. Experience shows that the current manufacturing cost of an optical system is roughly proportional to the cube of the aperture. Another approach is software: fusion processing of images from multiple sensors can improve detection performance. However, current patents and research focus primarily on image fusion across different wavelengths.

[0004] Regarding the hardware approach, patent number CN 201903705 U, for example, discloses a large-aperture, fully reflective optical synthetic aperture imaging system. This system uses multiple small-aperture optical systems to collect light and synthesize it onto a single detector to form an image. This approach uses only a single detector and requires precise phase modulation of the light waves collected by the different small apertures. Otherwise, interference fringes will form on the detector target surface. This requires extremely high equipment assembly precision and stability, and the hardware is complex and expensive to manufacture.

[0005] In the software pipeline, publication numbers CN 1545064A, CN 101714251A, CN 101799915A, and CN101853492A, for example, disclose various image fusion methods between infrared bands and between infrared and visible light bands. These methods focus on the fusion of images from different bands and employ various filtering and scale matching techniques in image processing. These methods can only be used for matching and fusion when the target image can be extracted from the original image. Summary of the Invention

[0006] The present invention addresses the technical problems existing in the prior art and provides a multi-channel image fusion system and method based on inertial space alignment. This system does not require the use of complex optical systems or modifications to photoelectric detection equipment. For multiple similar image detection devices installed separately, without changing the hardware, the system first performs pixel-level distortion calibration on the image to achieve pixel alignment between multiple channels. Synchronous imaging is then performed through collaborative scanning and external triggering, and the inertial devices carried by the detection device itself are used to align the image in inertial space. Image fusion is performed after synchronously acquiring multiple detector images in the background to improve the signal-to-noise ratio. This system can effectively solve the problems of inconsistent dynamic and time bases between multiple devices. Ultimately, it achieves information enhancement of small and weak targets, improves the detection range and detection probability, and is used to enhance the detection capabilities of photoelectric warning and reconnaissance equipment or long-range target tracking equipment. This system is particularly suitable for the coordinated use of multiple similar photoelectric detection devices installed in a dispersed manner on a moving platform.

[0007] According to a first aspect of the present invention, there is provided a multi-channel image fusion system based on inertial spatial alignment, comprising: at least two optoelectronic imaging devices of the same type installed at different locations, an inertial measurement unit arranged in a one-to-one correspondence with the optoelectronic imaging devices, and an image synchronization acquisition and processing device;

[0008] The inertial measurement unit is used to measure the dynamic change of the optical axis direction of the corresponding optoelectronic imaging device;

[0009] The image synchronous acquisition and processing device sends a synchronous external trigger signal to each of the photoelectric imaging devices;

[0010] After receiving the external trigger signal, each of the photoelectric imaging devices synchronously acquires an image, and transmits the acquired image and the corresponding information acquired by the inertial measurement unit back to the image synchronous acquisition and processing device;

[0011] The image synchronous acquisition and processing device performs inertial space alignment on each image according to the information acquired by the inertial measurement unit and then fuses the images.

[0012] On the basis of the above technical solution, the present invention can also make the following improvements.

[0013] Optionally, when installing the optoelectronic imaging device, the intrinsic parameter matrix and the extrinsic parameter matrix of each optoelectronic imaging channel are obtained by an image calibration method, and the distortion parameters are solved; the scanning axis system and the optical axis of the corresponding optoelectronic imaging device after installation are calibrated according to the distortion parameters to obtain its kinematic parameter model; and the pointing direction and image of the optoelectronic imaging device are corrected according to the kinematic parameter model.

[0014] Optionally, before each of the photoelectric imaging devices performs synchronous acquisition, each of the photoelectric imaging devices is directed to the same direction or the same target for synchronous scanning.

[0015] Optionally, after each of the photoelectric imaging devices transmits the distorted image, servo information and information measured by the inertial measurement unit to the image synchronization acquisition and processing device, the image synchronization acquisition and processing device processes the servo information and inertial information of each of the photoelectric imaging devices, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average direction of the optical axis of each image at the same moment.

[0016] Optionally, the image synchronous acquisition and processing device performs pixel superposition on each image to achieve image fusion.

[0017] Optionally, after the image synchronous acquisition and processing device performs pixel superposition on the image, the method further includes: obtaining directional information of the target or performing type recognition.

[0018] Optionally, the multi-channel image fusion system further includes an image display unit for displaying the fused image.

[0019] According to a second aspect of the present invention, a multi-channel image fusion method based on inertial space alignment is provided, comprising:

[0020] Step 1: Install at least two optoelectronic imaging devices of the same type at different locations and calibrate image distortion and installation references;

[0021] Step 2, performing synchronous scanning by the tracking servo system of each of the optoelectronic imaging devices so that each of the optoelectronic imaging devices points to the same direction or the same target;

[0022] Step 3: After each of the optoelectronic imaging devices has entered a stable scanning or tracking state, each of the optoelectronic imaging devices acquires an image based on an external trigger pulse sent by the image synchronous acquisition and processing device. After acquisition is complete, each of the optoelectronic imaging devices transmits the distortion-corrected image, servo information, and information acquired by the inertial measurement unit to the image synchronous acquisition and processing device.

[0023] Step 4: The image synchronous acquisition and processing device processes the servo information and inertial information of each of the optoelectronic imaging devices, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average orientation of the optical axes of each image at the same moment;

[0024] Step 5: Perform pixel superposition on the image with the distortion and inertial coordinate error eliminated.

[0025] On the basis of the above technical solution, the present invention can also make the following improvements.

[0026] Optionally, the process of calibrating the image distortion and the installation reference in step 1 includes:

[0027] The intrinsic and extrinsic parameter matrices of each optoelectronic imaging channel are obtained through an image calibration method, and the distortion parameters are solved. The scanning axis system and optical axis of the corresponding optoelectronic imaging device after installation are calibrated based on the distortion parameters to obtain its kinematic parameter model. Based on the kinematic parameter model, the pointing direction and image of the optoelectronic imaging device are corrected.

[0028] Optionally, after step 5, the following steps may be further performed:

[0029] Step 6: Obtain the target's direction information or perform type recognition.

[0030] The embodiment of the present invention provides a multi-channel image fusion system and method based on inertial space alignment. For a plurality of similar image detection devices installed separately, without changing the hardware, the system first performs pixel-level distortion calibration on the image to achieve pixel alignment between multiple channels; then synchronizes imaging through collaborative scanning and external triggering, and uses the inertial device carried by the detection device itself to align the image in the inertial space; after synchronously acquiring multiple detector images in the background, the system performs image fusion to improve the signal-to-noise ratio; the system can effectively solve the problems of inconsistent dynamic references and inconsistent time references between multiple devices; the image fusion also directly acts on pixel matching, without the need for multi-scale filtering, and without the need for target and background image segmentation after the target is detected; and finally effectively improves the target signal-to-noise ratio (theoretically, it can improve n 1 / 2 times, where n is the number of channels), thereby improving the target detection distance and detection probability, and is particularly suitable for the coordinated use of multiple photoelectric detection devices of the same type installed in a dispersed manner on a moving platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic structural diagram of a multi-channel image fusion system based on inertial space alignment provided by an embodiment of the present invention;

[0032] Figure 2 A flowchart of a multi-channel image fusion method based on inertial space alignment provided by an embodiment of the present invention;

[0033] In the accompanying drawings, the components represented by the reference numerals are as follows:

[0034] 1. Photoelectric imaging equipment, 2. Inertial measurement unit, 3. Image synchronous acquisition and processing equipment, 4. Image display unit. DETAILED DESCRIPTION

[0035] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0036] Figure 1 A schematic diagram of the structure of a multi-channel image fusion system based on inertial space alignment provided by the present invention is shown in FIG. Figure 1 As shown, the multi-channel image fusion system includes: at least two optoelectronic imaging devices 1 of the same type installed at different positions, an inertial measurement unit 2 corresponding to the optoelectronic imaging device 1, and an image synchronous acquisition and processing device 3.

[0037] In a specific implementation, the optoelectronic imaging device 1 can be an optoelectronic warning, reconnaissance or tracking device of the same type.

[0038] The inertial measurement unit 2 can be an inertial component directly installed in the optoelectronic imaging device 1. Optoelectronic devices installed on moving platforms (such as ships, vehicles, and aircraft) are generally equipped with an inertial measurement unit 2 (such as a gyroscope, inertial navigation system, etc.) to eliminate errors caused by their own motion because they need to exchange target coordinate information with the outside world.

[0039] The inertial measurement unit 2 is used to measure the dynamic change of the direction of the optical axis of the corresponding optoelectronic imaging device 1.

[0040] The image synchronous acquisition and processing device 3 sends a synchronous external trigger signal to each optoelectronic imaging device 1 .

[0041] After receiving the external trigger signal, each optoelectronic imaging device 1 performs synchronous image acquisition. This ensures that the image acquisition time nodes of each device are synchronized. In this way, the captured images are time-aligned, ensuring that the corresponding image acquisition targets are at the same position in space. The captured images and the information collected by the corresponding inertial measurement unit 2 are then transmitted back to the image synchronization acquisition and processing device 3.

[0042] The image synchronous acquisition and processing device 3 performs inertial space alignment on each image based on the information collected by the inertial measurement unit and then fuses the images.

[0043] The present invention provides a multi-channel image fusion system based on inertial space alignment. This system eliminates the need for complex optical systems or modifications to photoelectric detection equipment. For multiple similar image detection devices installed separately, without changing the hardware, the system first performs pixel-level distortion calibration on the image to achieve pixel alignment between multiple channels. Synchronous imaging is then performed through collaborative scanning and external triggering, using the inertial devices carried by the detection devices themselves to align the images in inertial space. Image fusion is then performed after synchronously acquiring multiple detector images in the background to improve the signal-to-noise ratio. This system effectively resolves the issues of inconsistent dynamic and time bases between multiple devices, ultimately enhancing information on small and weak targets, improving detection range and detection probability, and enhancing the detection capabilities of photoelectric warning and reconnaissance equipment or long-range target tracking equipment. The system is particularly suitable for the coordinated use of multiple similar photoelectric detection devices installed separately on a moving platform.

[0044] Example 1

[0045] The embodiment 1 provided by the present invention is an embodiment of a multi-channel image fusion system based on inertial space alignment provided by the present invention, combined with Figure 1 It can be seen that the embodiment of the multi-channel image fusion system includes: at least two optoelectronic imaging devices 1 of the same type installed at different positions, an inertial measurement unit 2 arranged in one-to-one correspondence with the optoelectronic imaging device 1, and an image synchronous acquisition and processing device 3.

[0046] In one possible embodiment, when installing the optoelectronic imaging device 1, the intrinsic parameter matrix and the extrinsic parameter matrix of each optoelectronic imaging channel are obtained through an image calibration method (such as Zhang Zhengyou calibration method, etc.), and the distortion parameters are solved; the scanning axis system and the optical axis of the corresponding optoelectronic imaging device 1 after installation are calibrated according to the distortion parameters to obtain its kinematic parameter model; according to the kinematic parameter model, the pointing direction and image of the optoelectronic imaging device 1 are corrected.

[0047] The inertial measurement unit 2 is used to measure the dynamic change of the direction of the optical axis of the corresponding optoelectronic imaging device 1.

[0048] The image synchronous acquisition and processing device 3 sends a synchronous external trigger signal to each optoelectronic imaging device 1 .

[0049] In a possible embodiment, each optoelectronic imaging device 1 is made to point to the same direction as much as possible at the same time, or when there is a clear target, the same target is tracked.

[0050] After receiving the external trigger signal, each optoelectronic imaging device 1 performs synchronous image acquisition and transmits the acquired image and the corresponding information acquired by the inertial measurement unit 2 back to the image synchronous acquisition and processing device 3 .

[0051] The image synchronous acquisition and processing device 3 performs inertial space alignment on each image based on the information collected by the inertial measurement unit and then fuses the images.

[0052] In one possible embodiment, after each optoelectronic imaging device 1 transmits the corrected distortion image, servo information, and information measured by the inertial measurement unit 2 to the image synchronization acquisition and processing device 3, the image synchronization acquisition and processing device 3 processes the servo information and inertial information of each optoelectronic imaging device 1, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average direction of the optical axis of each image at the same time.

[0053] In a possible embodiment, the image synchronous acquisition and processing device 3 performs pixel superposition on each image to achieve image fusion.

[0054] By performing pixel superposition on images that have been free of distortion and inertial coordinate errors, the target signal strength will theoretically increase by a factor of n, while the noise signal strength will increase by a factor of n½. Therefore, theoretically, an image with an n½-fold signal-to-noise ratio enhancement can be obtained.

[0055] In a possible embodiment, after the image synchronous acquisition and processing device 3 performs pixel superposition on the image, the device further includes: further processing the enhanced image to obtain the directional information of the target or perform type recognition and other operations.

[0056] In a possible embodiment, the multi-channel image fusion system further includes an image display unit 4 for displaying the fused image.

[0057] Example 2

[0058] The embodiment 2 provided by the present invention is an embodiment of a multi-channel image fusion method based on inertial space alignment provided by the present invention, such as Figure 2 The flowchart of a multi-channel image fusion method based on inertial space alignment provided by an embodiment of the present invention is shown. Figure 2 It can be seen that the embodiment of the multi-channel image fusion method includes:

[0059] Step 1: Install at least two optoelectronic imaging devices of the same type at different locations and calibrate image distortion and installation reference.

[0060] In one possible embodiment, the process of calibrating the image distortion and the installation reference in step 1 includes:

[0061] The intrinsic and extrinsic parameter matrices of each optoelectronic imaging channel are obtained through image calibration methods (such as the Zhang Zhengyou calibration method), and the distortion parameters are solved. The scanning axis system and optical axis of the corresponding optoelectronic imaging device after installation are calibrated based on the distortion parameters to obtain its kinematic parameter model. Based on the kinematic parameter model, the pointing direction and image of the optoelectronic imaging device are corrected.

[0062] Step 2: Perform synchronous scanning through the tracking servo systems of the various optoelectronic imaging devices so that the various optoelectronic imaging devices point to the same direction or the same target.

[0063] Step 3: After the scanning or tracking of each optoelectronic imaging device enters a stable state, each optoelectronic imaging device performs image acquisition according to the external trigger pulse sent by the image synchronization acquisition and processing device; after the acquisition is completed, each optoelectronic imaging device transmits the corrected distortion image, servo information and information collected by the inertial measurement unit to the image synchronization acquisition and processing device.

[0064] In step 4, the image synchronous acquisition and processing device processes the servo information and inertial information of each optoelectronic imaging device, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average direction of the optical axis of each image at the same moment.

[0065] Step 5: Perform pixel superposition on the image with the distortion and inertial coordinate error eliminated.

[0066] In a possible embodiment, step 5 further includes: step 6, obtaining the directional information of the target or performing type identification.

[0067] It can be understood that the multi-channel image fusion method based on inertial space alignment provided by the present invention corresponds to the multi-channel image fusion system based on inertial space alignment provided in the aforementioned embodiments. The relevant technical features of the multi-channel image fusion method based on inertial space alignment can refer to the relevant technical features of the multi-channel image fusion system based on inertial space alignment, and will not be repeated here.

[0068] The embodiment of the present invention provides a multi-channel image fusion system and method based on inertial space alignment. For a plurality of similar image detection devices installed separately, without changing the hardware, the system first performs pixel-level distortion calibration on the image to achieve pixel alignment between multiple channels; then synchronizes imaging through collaborative scanning and external triggering, and uses the inertial device carried by the detection device itself to align the image in the inertial space; after synchronously acquiring multiple detector images in the background, the system performs image fusion to improve the signal-to-noise ratio; the system can effectively solve the problems of inconsistent dynamic references and inconsistent time references between multiple devices; the image fusion also directly acts on pixel matching, without the need for multi-scale filtering, and without the need for target and background image segmentation after the target is detected; and finally effectively improves the target signal-to-noise ratio (theoretically, it can improve n 1 / 2 times, where n is the number of channels), thereby improving the target detection distance and detection probability, and is particularly suitable for the coordinated use of multiple photoelectric detection devices of the same type installed in a dispersed manner on a moving platform.

[0069] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0070] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0074] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0075] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multi-channel image fusion system based on inertial space alignment, characterized in that: The multi-channel image fusion system includes: at least two optoelectronic imaging devices of the same type installed at different positions, an inertial measurement unit and an image synchronous acquisition and processing device arranged in a one-to-one correspondence with the optoelectronic imaging devices; The inertial measurement unit is used to measure the dynamic change of the optical axis direction of the corresponding optoelectronic imaging device; The image synchronous acquisition and processing device sends a synchronous external trigger signal to each of the photoelectric imaging devices; After receiving the external trigger signal, each of the photoelectric imaging devices synchronously acquires an image, and transmits the acquired image and the corresponding information acquired by the inertial measurement unit back to the image synchronous acquisition and processing device; The image synchronous acquisition and processing device performs inertial space alignment on each image based on the information collected by the inertial measurement unit and then fuses the images; When installing the optoelectronic imaging device, an intrinsic parameter matrix and an extrinsic parameter matrix of each optoelectronic imaging channel are obtained by an image calibration method, and distortion parameters are obtained by solving the problem; a scanning axis system and an optical axis of the corresponding optoelectronic imaging device after installation are calibrated according to the distortion parameters to obtain a kinematic parameter model; and a pointing direction and an image of the optoelectronic imaging device are corrected according to the kinematic parameter model; After each of the photoelectric imaging devices transmits the distortion-corrected image, servo information, and information measured by the inertial measurement unit to the image synchronization acquisition and processing device, the image synchronization acquisition and processing device processes the servo information and inertial information of each of the photoelectric imaging devices, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average direction of the optical axis of each image at the same moment.

2. The multi-channel image fusion system according to claim 1, characterized in that: Before each of the photoelectric imaging devices performs synchronous acquisition, each of the photoelectric imaging devices is directed to the same direction or the same target for synchronous scanning.

3. The multi-channel image fusion system according to claim 1, characterized in that: The image synchronous acquisition and processing device performs pixel superposition on each image to achieve image fusion.

4. The multi-channel image fusion system according to claim 1, characterized in that: After the image synchronous acquisition and processing device performs pixel superposition on the image, the method further includes: obtaining the directional information of the target or performing type recognition.

5. The multi-channel image fusion system according to claim 1, characterized in that: The multi-channel image fusion system further includes an image display unit for displaying the fused image.

6. A multi-channel image fusion method based on inertial space alignment, characterized in that: The multi-channel image fusion method comprises: Step 1: Install at least two optoelectronic imaging devices of the same type, an inertial measurement unit and an image synchronous acquisition and processing device corresponding to the optoelectronic imaging devices at different locations; The inertial measurement unit is used to measure the dynamic change of the optical axis direction of the corresponding optoelectronic imaging device; Step 2: the image synchronous acquisition and processing device sends a synchronous external trigger signal to each of the optoelectronic imaging devices; Step 3: After receiving the external trigger signal, each of the photoelectric imaging devices synchronously acquires an image, and transmits the acquired image and the corresponding information acquired by the inertial measurement unit back to the image synchronous acquisition and processing device; Step 4: the image synchronous acquisition and processing device performs inertial space alignment on each image based on the information collected by the inertial measurement unit and then fuses the images; When installing the optoelectronic imaging device in step 1, an intrinsic parameter matrix and an extrinsic parameter matrix of each optoelectronic imaging channel are obtained by an image calibration method, and distortion parameters are obtained by solving; the scanning axis system and the optical axis of the corresponding optoelectronic imaging device after installation are calibrated according to the distortion parameters to obtain its kinematic parameter model; and the orientation and image of the optoelectronic imaging device are corrected according to the kinematic parameter model; After each of the photoelectric imaging devices transmits the distortion-corrected image, servo information, and information measured by the inertial measurement unit to the image synchronization acquisition and processing device, the image synchronization acquisition and processing device processes the servo information and inertial information of each of the photoelectric imaging devices, calculates the inertial space coordinates corresponding to each image, and then crops and / or rotates each image based on the average direction of the optical axis of each image at the same moment.

7. The multi-channel image fusion method according to claim 6, characterized in that: After step 4, the following steps are also included: Step 5: Obtain the target's direction information or perform type recognition.

Images