An electronic device and method for infrared and visible light target fusion

Abstract

This invention relates to an electronic device and method for infrared and visible light target fusion. The method involves acquiring infrared and visible light images in real time and preprocessing them; extracting infrared latent targets from the infrared images; identifying and marking ground features using the visible light images to extract visible light latent targets; using the marked ground feature background from the visible light images to remove false alarms from the extracted infrared and visible light latent targets; performing track matching or creating new tracks for the infrared latent targets after false alarm removal; performing track matching or creating new tracks for the visible light latent targets after false alarm removal; and performing target fusion and classification based on track features and target characteristics for the infrared and visible light targets after track addition, outputting the final target information. This invention focuses on target information in infrared and visible light images, reduces redundant background interference, improves computational efficiency, and effectively enhances the ability to detect, identify, and classify targets.

Description

Technical Field

[0001] This invention relates to the fields of multi-source information fusion and target processing technology, and to an electronic device and method for infrared and visible light target fusion. Background Technology

[0002] Infrared and visible light fusion is an important research area in multi-source information fusion, aiming to complement the advantages of infrared and visible light images. The development of infrared and visible light fusion has been rapid, evolving from traditional image processing methods to modern deep learning approaches, greatly promoting the application and advancement of multimodal data fusion technology, particularly in fields such as autonomous driving and security monitoring.

[0003] However, current infrared and visible light fusion strategies are generally image-level fusion methods, lacking target-level fusion approaches. Image-level fusion suffers from problems such as background interference, information aliasing, high computational complexity, and decreased target detection accuracy. In certain tasks (such as target detection, recognition, and tracking), these drawbacks may limit its application. Therefore, in more complex scenes or tasks, target-level fusion methods are generally more effective, reducing background interference, improving computational efficiency, and performing better when handling differences between images of different modalities. Summary of the Invention

[0004] The purpose of this invention is to provide an electronic device and method for infrared and visible light target fusion, which solves the problem that current infrared and visible light fusion strategies are generally image-level fusions, lacking target-level fusion methods. Image-level fusion suffers from problems such as background interference, information aliasing, high computational complexity, and decreased target detection accuracy.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] An electronic device for fusion of infrared and visible light targets includes an infrared image processing board, a visible light image processing board, an integrated control board, and a computer program stored therein;

[0007] The infrared image processing board integrates an infrared image acquisition module, an infrared image preprocessing module, an infrared latent target extraction module, and an infrared track establishment and matching module. The infrared image acquisition module acquires infrared image data in real time. The infrared image preprocessing module performs preprocessing on the infrared images, including two-point correction, scene correction, and image enhancement. The infrared latent target extraction module performs target detection on the infrared images, acquiring target position information, time information, target size, target shape, target area, and target brightness. The infrared track establishment and matching module matches the extracted infrared latent targets with existing target tracks, calculating the target's velocity information, acceleration information, motion pattern information, and track length. It also establishes new tracks for targets that do not match existing tracks and periodically manages all target tracks.

[0008] The visible light image processing board integrates a visible light image acquisition module, a visible light image preprocessing module, a visible light latent target extraction module, a ground feature recognition module, and a visible light track establishment and matching module. Specifically, the visible light image acquisition module acquires visible light image data in real time; the visible light image preprocessing module performs preprocessing on the visible light images, including image denoising and image enhancement; the visible light latent target extraction module performs target detection on the visible light images, acquiring target location information, time information, target size, target shape, target area, and target brightness; the ground feature recognition module uses a deep learning-based target detection method to identify and distinguish ground features from the background; and the visible light track establishment and matching module matches the extracted visible light latent targets with existing target tracks, calculates the target's velocity information, acceleration information, motion pattern information, and track length; establishes new tracks for targets that do not match, and periodically manages all target tracks.

[0009] The integrated control board integrates a target fusion module; the target fusion module performs target fusion and classification identification on infrared and visible light targets after the addition of flight tracks based on flight track characteristics and target characteristics, and outputs the final target information.

[0010] The infrared image processing board adopts an FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data.

[0011] The visible light image processing board adopts an FPGA + image processor architecture, storing FPGA and image processor programs for processing visible light image data.

[0012] The infrared image processing board and the visible light image processing board communicate via optical fiber, and the infrared image processing board and the visible light image processing board communicate with the integrated control board via PCIe.

[0013] A method for fusing infrared and visible light targets using the aforementioned electronic device, the method comprising the following steps:

[0014] Step 1: The infrared image acquisition module and the visible light image acquisition module acquire infrared and visible light images in real time, respectively; the infrared image preprocessing module performs preprocessing on the infrared image, including two-point correction, scene correction, and image enhancement; the visible light image preprocessing module performs image denoising and image enhancement on the visible light image.

[0015] Step 2: The infrared latent target extraction module extracts infrared latent targets from the infrared image; the ground feature recognition module uses the visible light image to identify and mark ground features and backgrounds; and the visible light latent target extraction module extracts visible light latent targets.

[0016] Step 3: Use the background of ground features marked in the visible light image to remove false alarms from the extracted infrared and visible light latent targets;

[0017] Step 4: The infrared track establishment and matching module performs track matching or establishes new tracks for infrared latent targets after false alarm elimination; the visible light track establishment and matching module performs track matching or establishes new tracks for visible light latent targets after false alarm elimination.

[0018] Step 5: The target fusion module performs target fusion and classification identification on the infrared and visible light targets after the addition of the track, based on the track characteristics and target features, and outputs the final target information.

[0019] The infrared and visible light images in step one are local images at the same azimuth and elevation angle; the infrared and visible light images acquired in real time are linear scan images or area scan images.

[0020] Image enhancement in step one of the preprocessing steps refers to enhancing the edges, brightness, and details of objects in the image, making the object edges clearer and the brightness higher.

[0021] In step two, the ground feature background identification and labeling uses a deep learning-based object detection method to identify and distinguish ground features from the background.

[0022] False alarms in step three refer to identifying areas that do not conform to the characteristics of infrared and visible light latent targets and the ground background by analyzing the features of infrared and visible light latent targets and eliminating the background areas. The ground background clearly marked in the visible light image can be used as a reference for eliminating false alarms. If the latent target in the infrared image does not match the ground type in the visible light image, then the area may be a false alarm.

[0023] The trajectory features mentioned in step five include position information, velocity information, acceleration information, motion pattern information, trajectory length, and time information; target characteristics include target size, target shape, target area, and target brightness.

[0024] The advantages of this invention are as follows: 1. The infrared and visible light target fusion method of this invention focuses more on the extraction, matching, and synthesis of information about the target itself; 2. The method of this invention can effectively solve the problems of background interference, information aliasing, high computational complexity, and decreased target detection accuracy in image-level fusion; 3. In target detection, recognition, tracking, and classification, target-level fusion methods are generally more effective, reducing background interference, improving computational efficiency, effectively enhancing the ability to detect, recognize, and classify targets, and performing better when dealing with differences in images of different modalities, especially in applications with clear targets in complex scenes; 4. The electronic device of this invention for infrared and visible light target fusion adopts an embedded processor architecture, which has the advantages of strong processing power, low latency, and high real-time performance, and the data processing time can reach the millisecond level. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the electronic device structure for the infrared and visible light target fusion method of the present invention. Figure 1 ;

[0026] Figure 2 This is a schematic diagram of the electronic device structure for the infrared and visible light target fusion method of the present invention. Figure 2 ;

[0027] Figure 3 This is a schematic flowchart of the infrared and visible light target fusion method of the present invention.

[0028] In the diagram: 201-Infrared image acquisition module; 202-Visible light image acquisition module; 203-Infrared image preprocessing module; 204-Visible light image preprocessing module; 205-Infrared latent target extraction module; 206-Visible light latent target extraction module; 207-Ground feature identification module; 208-Infrared track establishment and matching module; 209-Visible light track establishment and matching module; 210-Target fusion module; 301-Infrared image processing board; 302-Visible light image processing board; 303-Integrated control board. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings. The drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0030] To simplify the description of this embodiment, some components that are well-known to those skilled in the art but are not related to the main content of this invention may be omitted in the accompanying drawings or description. Additionally, for ease of description, some components in the drawings may be omitted, enlarged, or reduced, but these do not represent the actual product dimensions or the complete structure.

[0031] This invention discloses an electronic device for a method of fusion of infrared and visible light targets, such as... Figure 1 , Figure 2 As shown, it includes an infrared image processing board 301, a visible light image processing board 302, an integrated control board 303, and a computer program stored therein; the computer program runs on each circuit board to jointly realize the aforementioned infrared and visible light target fusion method.

[0032] like Figure 2 As shown, the infrared image processing board 301 integrates an infrared image acquisition module 201, an infrared image preprocessing module 203, an infrared latent target extraction module 205, and an infrared track establishment and matching module 208.

[0033] The infrared image acquisition module 201 acquires infrared image data in real time; the infrared image preprocessing module 203 performs preprocessing on the infrared image, including two-point correction, scene correction, and image enhancement; the infrared latent target extraction module 205 performs target detection on the infrared image, acquiring the target's position information, time information, target size, target shape, target area, and target brightness; the infrared track establishment and matching module 208 matches the extracted infrared latent targets with existing target tracks, calculates the target's velocity information, acceleration information, motion pattern information, and track length; establishes new tracks for targets that do not match tracks, and periodically manages all target tracks.

[0034] The infrared image processing board 301 adopts an FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data. The infrared image acquisition module 201, the infrared image preprocessing module 203, the infrared latent target extraction module 205, and the infrared track establishment and matching module 208 all run on the infrared image processing board 301.

[0035] The FPGA and DSP chips in the infrared image processing board 301 can be selected according to the amount of input data. For example, in some applications with high real-time requirements, the SMQ325T-FFG900 FPGA from Shenzhen Guowei and the FT-6678 8-core DSP processor from the National University of Defense Technology can be combined. This combination can achieve a data processing capacity of 300MB / s and a target detection output time of ≤3ms.

[0036] The visible light image processing board 302 integrates a visible light image acquisition module 202, a visible light image preprocessing module 204, a visible light latent target extraction module 206, a ground feature recognition module 207, and a visible light track establishment and matching module 209;

[0037] The visible light image acquisition module 202 acquires visible light image data in real time; the visible light image preprocessing module 204 performs preprocessing on the visible light image, including image denoising and image enhancement; the visible light latent target extraction module 206 performs target detection on the visible light image, acquiring the target's location information, time information, target size, target shape, target area, and target brightness; the ground feature recognition module 207 uses a deep learning-based target detection method to identify and distinguish ground features and background; the visible light track establishment and matching module 209 matches the extracted visible light latent targets with existing target tracks, calculates the target's velocity information, acceleration information, motion pattern information, and track length; establishes new tracks for targets that do not match tracks, and periodically manages all target tracks.

[0038] The visible light image processing board 302 adopts an FPGA + image processor architecture, storing FPGA and image processor programs for processing visible light image data. The visible light image acquisition module 202, the visible light image preprocessing module 204, the visible light latent target extraction module 206, the ground feature identification module 207, and the visible light track establishment and matching module 209 all run on the visible light image processing board 302.

[0039] The FPGA chip and image processing chip in the visible light image processing board 302 can be selected according to the amount of input data. For example, the FPGA can be the SMQ325T-FFG900 from Shenzhen Guowei or a similar product, and the image processing chip can be the HI3559 from HiSilicon Semiconductor, the FT-6678 8-core DSP processor from the National University of Defense Technology, or other processors suitable for image processing.

[0040] The integrated control board 303 integrates the target fusion module 210.

[0041] The target fusion module 210 performs target fusion and classification identification on infrared and visible light targets after the addition of track features and target characteristics, and outputs the final target information.

[0042] The integrated control board 303 includes a processor and memory, with the memory storing computer programs. The integrated control board 303 is used to achieve infrared and visible light target fusion.

[0043] The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other conventional processor.

[0044] To meet the requirements of high real-time processing of big data, in some embodiments, high-speed fiber optic interfaces and high-speed PCIe (Peripheral Component Interconnect Express) interfaces are used between the circuit boards.

[0045] The infrared image processing board 301 and the visible light image processing board 302 communicate via optical fiber.

[0046] The infrared image processing board 301 and the visible light image processing board 302 communicate with the integrated control board (303) via PCIe.

[0047] A method for fusing infrared and visible light targets using the aforementioned electronic equipment, such as Figure 3 As shown, the method includes the following steps:

[0048] Step 1: The infrared image acquisition module 201 and the visible light image acquisition module 202 acquire infrared and visible light images in real time, respectively; the infrared image preprocessing module 203 performs preprocessing on the infrared image, including two-point correction, scene correction, and image enhancement; the visible light image preprocessing module 204 performs image denoising and image enhancement on the visible light image.

[0049] Among them, the real-time acquired infrared and visible light images are local images at the same azimuth and elevation angle at the same time; the real-time acquired infrared and visible light images are linear scan images or area scan images.

[0050] Specifically, infrared and visible light images can be original grayscale images with a pixel size of 640*512 or 1024*1024, or other sizes. The pixel size of the infrared and visible light images does not need to be the same, but the elevation and azimuth angles of the infrared and visible light images must be the same.

[0051] Image enhancement in preprocessing refers to enhancing the edges, brightness, and details of objects in an image, making the edges of objects clearer and the brightness higher.

[0052] Step 2: Infrared latent target extraction module 205 extracts infrared latent targets from infrared images; ground feature recognition module 207 uses visible light images to identify and mark ground features and backgrounds; and visible light latent target extraction module 206 extracts visible light latent targets.

[0053] Specifically, it extracts specific targets from infrared and visible light images, such as passenger planes, drones in the air, or cars and pedestrians on the ground; at the same time, it uses visible light images to identify and mark ground backgrounds such as buildings, roads, trees, and distant mountains, eliminating potential targets that are misidentified in the ground background.

[0054] Ground feature background identification and labeling uses deep learning-based object detection methods to identify and distinguish ground features from the background, such as convolutional neural networks or deep residual networks.

[0055] Step 3: Use the background of ground features marked in the visible light image to remove false alarms from the extracted infrared and visible light latent targets;

[0056] Specifically, background information of ground features in visible light images, such as buildings, roads, trees, and distant mountains, typically possesses stable and structured characteristics, while potential targets generally exhibit features significantly different from the background. Based on this, false alarms refer to identifying areas that do not conform to the characteristics of infrared and visible light potential targets relative to the ground feature background by analyzing the features of these targets, thus eliminating these background areas. Clearly marked ground feature backgrounds in visible light images can serve as a reference for eliminating false alarms; if a potential target in an infrared image does not match the type of ground feature in a visible light image, that area may be a false alarm.

[0057] Step 4: The infrared track establishment and matching module 208 performs track matching or establishes a new track for the infrared latent targets after false alarm elimination; the visible light track establishment and matching module 209 performs track matching or establishes a new track for the visible light latent targets after false alarm elimination.

[0058] Specifically, track matching is primarily based on the target's position, time, size, shape, area, and brightness. Within a set tolerance range, previously appearing tracks are matched one by one, and targets that meet the criteria are added to the existing tracks. Potential targets that do not match are considered to be appearing for the first time, and a new track is created for them.

[0059] Step 5: The target fusion module 210 performs target fusion and classification identification on the infrared and visible light targets after the addition of the track, based on the track characteristics and target features, and outputs the final target information.

[0060] The track features include position information, velocity information, acceleration information, motion pattern information, track length, and time information; target characteristics include target size, target shape, target area, and target brightness.

[0061] In some embodiments, different weights can be assigned to the importance and uncertainty of information in track features and target characteristics according to application scenario requirements. The similarity between infrared and visible light targets is then calculated through weighted summation. Furthermore, by setting a similarity threshold, targets meeting the threshold range are identified as the same target and classified accordingly.

[0062] The classification and identification of the aforementioned targets can be performed using different classification and identification algorithms depending on the application scenario. For example, the YOLO model, a deep learning-based target detection method, can be chosen to identify and classify aircraft, drones, pedestrians, and vehicles.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. All equivalent changes and modifications made within the scope of the claims of this invention should be considered within the technical scope of this invention.

Claims

1. An electronic device for fusion of infrared and visible light targets, comprising an infrared image processing board (301), a visible light image processing board (302), an integrated control board (303), and a computer program stored therein; Its features are: The infrared image processing board (301) integrates an infrared image acquisition module (201), an infrared image preprocessing module (203), an infrared latent target extraction module (205), and an infrared track establishment and matching module (208); wherein, the infrared image acquisition module (201) acquires infrared image data in real time; the infrared image preprocessing module (203) performs preprocessing on the infrared image including two-point correction, scene correction, and image enhancement; the infrared latent target extraction module (205) performs target detection on the infrared image and acquires the target's position information, time information, target size, target shape, target area, and target brightness; The visible light image processing board (302) integrates a visible light image acquisition module (202), a visible light image preprocessing module (204), a visible light latent target extraction module (206), a ground feature recognition module (207), and a visible light track establishment and matching module (209); wherein, the visible light image acquisition module (202) acquires visible light image data in real time; the visible light image preprocessing module (204) performs preprocessing on the visible light image, including image denoising and image enhancement; the visible light latent target extraction module (206) performs target detection on the visible light image and acquires the target's location information, time information, target size, target shape, target area, and target brightness; the ground feature recognition module (207) uses a deep learning-based target detection method to identify and distinguish ground features and background; The electronic device uses the background of ground features marked in the visible light image to eliminate false alarms of extracted infrared and visible light latent targets. False alarms refer to the identification of areas that do not conform to the characteristics of infrared and visible light latent targets and the ground feature background by analyzing the characteristics of these targets, and then eliminating these background areas. The clearly marked ground feature background in the visible light image serves as a reference for eliminating false alarms. If the latent target in the infrared image does not match the type of ground feature in the visible light image, then that area is a false alarm and is eliminated. The infrared track establishment and matching module (208) matches the infrared potential targets after false alarm elimination with existing target tracks, calculates the target's speed information, acceleration information, motion pattern information and track length; establishes new tracks for targets that do not match tracks, and periodically manages all target tracks. The visible light track establishment and matching module (209) matches the visible light latent targets after false alarm elimination with existing target tracks, calculates the target's speed information, acceleration information, motion pattern information and track length; establishes new tracks for targets that do not match tracks, and periodically manages all target tracks. The integrated control board (303) integrates a target fusion module (210); the target fusion module (210) performs target fusion and classification identification on infrared targets and visible light targets after the addition of the track based on the track characteristics and target characteristics, and outputs the final target information.

2. The electronic device according to claim 1, characterized in that: The infrared image processing board (301) adopts an FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data; the visible light image processing board (302) adopts an FPGA+image processor architecture and stores FPGA and image processor programs for processing visible light image data.

3. The electronic device according to claim 1, characterized in that: The infrared image processing board (301) and the visible light image processing board (302) communicate via optical fiber, and the infrared image processing board (301) and the visible light image processing board (302) communicate with the integrated control board (303) via PCIe.

4. A method for fusing infrared and visible light targets using the electronic device according to any one of claims 1-3, characterized in that: The method includes the following steps: Step 1: The infrared image acquisition module (201) and the visible light image acquisition module (202) acquire infrared images and visible light images in real time, respectively; the infrared image preprocessing module (203) performs preprocessing on the infrared image including two-point correction, scene correction and image enhancement; the visible light image preprocessing module (204) performs image denoising and image enhancement on the visible light image. Step 2: Extract infrared latent targets from the infrared image using the infrared latent target extraction module (205); The ground feature recognition module (207) uses a deep learning-based target detection method to identify and distinguish ground features and background, and uses visible light images to identify and mark ground feature backgrounds. The visible light latent target extraction module (206) extracts visible light latent targets. Step 3: Use the background of ground features marked in the visible light image to remove false alarms from the extracted infrared and visible light latent targets. False alarms refer to the identification of areas that do not conform to the characteristics of infrared and visible light latent targets by analyzing the features of the ground feature background and removing the background areas. The clearly marked ground feature background in the visible light image serves as a reference for removing false alarms. If the latent target in the infrared image does not match the type of ground feature in the visible light image, then that area may be a false alarm. Step 4: The infrared track establishment and matching module (208) performs track matching or establishes a new track for the infrared latent target after false alarm elimination; the visible light track establishment and matching module (209) performs track matching or establishes a new track for the visible light latent target after false alarm elimination. Step 5: The target fusion module (210) performs target fusion and classification identification on the infrared and visible light targets after the track is added, based on the track characteristics and target characteristics, and outputs the final target information.

5. The method according to claim 4, characterized in that: The infrared and visible light images in step one are local images at the same azimuth and elevation angle; the infrared and visible light images acquired in real time are linear scan images or area scan images.

6. The method according to claim 4, characterized in that: Image enhancement in step one of the preprocessing steps refers to enhancing the edges, brightness, and details of objects in the image, making the object edges clearer and the brightness higher.

7. The method according to claim 4, characterized in that: The trajectory features mentioned in step five include position information, velocity information, acceleration information, motion pattern information, trajectory length, and time information; target characteristics include target size, target shape, target area, and target brightness.

Images