Electronic device and method for infrared and visible light target fusion

Abstract

The invention relates to an electronic device and method for infrared and visible light target fusion. The method comprises the steps that an infrared image and a visible light image are obtained in real time and preprocessed; performing infrared latent target extraction on the infrared image; carrying out ground object background identification marking by using the visible light image, and carrying out visible light latent target extraction; performing false alarm elimination on the extracted infrared latent target and the visible light latent target by using the ground object background marked by the visible light image; performing track matching on the infrared latent target after false alarm elimination or establishing a new track; carrying out track matching or establishing a new track on the visible light latent target after the false alarm is eliminated; and carrying out target fusion and classification identification on the infrared target and the visible light target which are added with the track according to track characteristics and target characteristics, and outputting final target information. According to the method, target information in infrared and visible light images is focused, redundant background interference is reduced, the calculation efficiency is improved, and the target detection, recognition and classification capabilities can be effectively improved.

Description

Technical Field

[0001] The invention relates to the technical field of multi-source information fusion and target processing, and relates to an electronic device and a method for fusing infrared and visible light targets. Background Art

[0002] Infrared and visible light fusion is an important research field of multi-source information fusion, which aims to complement the advantages of infrared images and visible light images. Infrared and visible light fusion has developed rapidly from traditional image processing methods to modern deep learning methods, which has greatly promoted the application and progress of multimodal data fusion technology, especially in the fields of autonomous driving, security monitoring, etc.

[0003] However, current infrared and visible light fusion strategies are generally aimed at image-level fusion, lacking target-level fusion methods. Image-level fusion has problems such as background interference, information aliasing, high computational complexity, and reduced target detection accuracy. In some specific tasks (such as target detection, recognition, and tracking), these shortcomings may limit its application effect. Therefore, in more complex scenes or tasks, target-level fusion methods are usually more effective, which can reduce background interference, improve computational efficiency, and perform better when dealing with differences in images of different modalities. Summary of the invention

[0004] The purpose of the present invention is to provide an electronic device and method for infrared and visible light target fusion, which solves the problem that the current infrared and visible light fusion strategy is generally aimed at image-level fusion, but lacks a target-level fusion method. Image-level fusion has problems such as background interference, information aliasing, high computational complexity, and reduced target detection accuracy.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] An electronic device for infrared and visible light target fusion, comprising an infrared image processing board, a visible light image processing board, an integrated control board and a computer program stored therein;

[0007] The infrared image processing board integrates an infrared image acquisition module, an infrared image preprocessing module, an infrared latent target extraction module, and an infrared track establishment and matching module; wherein the infrared image acquisition module acquires infrared image data in real time; the infrared image preprocessing module performs preprocessing of infrared images including two-point correction, scene correction, and image enhancement; the infrared latent target extraction module performs target detection on infrared images to obtain the target's position information, time information, target size, target shape, target area, and target brightness; the infrared track establishment and matching module matches the extracted infrared latent target with the existing target track, and calculates the target's speed information, acceleration information, motion mode information, and track length; establishes a new track for the target that does not match the track, and regularly manages the track of all target tracks;

[0008] The visible light image processing board integrates a visible light image acquisition module, a visible light image preprocessing module, a visible light potential target extraction module, a ground object recognition module and a visible light track establishment and matching module; wherein, the visible light image acquisition module acquires visible light image data in real time; the visible light image preprocessing module performs preprocessing of visible light images including image denoising and image enhancement; the visible light potential target extraction module performs target detection on visible light images to obtain the target's position information, time information, target size, target shape, target area and target brightness; the ground object recognition module adopts a target detection method based on deep learning to identify and distinguish ground objects and background; the visible light track establishment and matching module matches the extracted visible light potential target with the existing target track, and calculates the target's speed information, acceleration information, motion mode information and track length; establishes a new track for the target that does not match the track, and regularly manages the track of all target tracks;

[0009] The integrated control panel integrates a target fusion module; the target fusion module performs target fusion and classification identification on the infrared target and visible light target after adding the track according to the track characteristics and target properties, and outputs the final target information.

[0010] The infrared image processing board adopts FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data;

[0011] The visible light image processing board adopts FPGA+image processor architecture and stores FPGA and image processor programs for processing visible light image data.

[0012] The infrared image processing board and the visible light image processing board communicate with each other through optical fiber, and the infrared image processing board, the visible light image processing board and the integrated control board communicate with each other through PCIe.

[0013] A method for using the aforementioned electronic device to fuse infrared and visible light targets, the method comprising the following steps:

[0014] Step 1: The infrared image acquisition module and the visible light image acquisition module respectively acquire the infrared image and the visible light image in real time; the infrared image preprocessing module performs preprocessing including two-point correction, scene correction, and image enhancement on the infrared image; the visible light image preprocessing module performs image denoising and image enhancement preprocessing on the visible light image;

[0015] Step 2: The infrared potential target extraction module extracts infrared potential targets from the infrared image; the ground object recognition module uses the visible light image to identify the ground object background, and the visible light potential target extraction module extracts visible light potential targets;

[0016] Step 3, using the ground object background marked by the visible light image to remove false alarms of the extracted infrared latent targets and visible light latent targets;

[0017] Step 4: The infrared track establishment and matching module performs track matching or establishes a new track for the infrared latent target after false alarms are eliminated; the visible light track establishment and matching module performs track matching or establishes a new track for the visible light latent target after false alarms are eliminated;

[0018] Step 5: The target fusion module performs target fusion and classification identification on the infrared target and visible light target after adding the track according to the track characteristics and target characteristics, and outputs the final target information.

[0019] The infrared and visible light images in step 1 are local images at the same azimuth and elevation angles; the infrared and visible light images acquired in real time are line scan images or area scan images.

[0020] Image enhancement in step 1 preprocessing refers to enhancing the edge, brightness and details of the target in the image to make the edge of the object clearer and the brightness higher.

[0021] The ground object and background recognition marking in step 2 uses a deep learning-based target detection method to identify and distinguish ground objects and background.

[0022] The false alarm in step three refers to identifying areas that do not meet the characteristics of infrared latent targets and visible light latent targets by analyzing the characteristics of the ground background. The ground background clearly marked in the visible light image can be used as a reference for eliminating false alarms. If the latent target in the infrared image does not match the ground type in the visible light image, the area may be a false alarm.

[0023] The track characteristics described in step five include position information, speed information, acceleration information, motion mode information, track length and time information; the target characteristics include target size, target shape, target area and target brightness.

[0024] The advantages of the present invention are: 1. The infrared and visible light target fusion method of the present invention pays more attention to the information extraction, matching and integration of the target itself; 2. The method of the present invention can effectively solve the problems of background interference, information aliasing, high computational complexity, reduced target detection accuracy, etc. in image-level fusion; 3. In target detection, recognition, tracking and classification, the target-level fusion method is usually more effective, which can reduce background interference, improve computational efficiency, effectively enhance the ability of target detection, recognition and classification, and perform better when processing the differences between images of different modalities, especially in applications with clear targets in complex scenes; 4. The electronic device for infrared and visible light target fusion of the present invention adopts an embedded processor architecture, which has the advantages of strong processing capability, low latency and high real-time performance, and the data processing time can reach the millisecond level requirement. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The electronic device structure diagram of the infrared and visible light target fusion method of the present invention is shown in FIG. Figure 1 ;

[0026] Figure 2 The electronic device structure diagram of the infrared and visible light target fusion method of the present invention is shown in FIG. Figure 2 ;

[0027] Figure 3 It is a flow chart of the infrared and visible light target fusion method of the present invention.

[0028] In the figure: 201-infrared image acquisition module; 202-visible light image acquisition module; 203-infrared image preprocessing module; 204-visible light image preprocessing module; 205-infrared potential target extraction module; 206-visible light potential target extraction module; 207-ground object recognition module; 208-infrared track establishment and matching module; 209-visible light track establishment and matching module; 210-target fusion module; 301-infrared image processing board; 302-visible light image processing board; 303-integrated control board. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings, which are only used for exemplary description and cannot be understood as limiting the present invention.

[0030] In order to more concisely describe the present embodiment, some parts known to those skilled in the art but not related to the main content of the present invention are omitted in the drawings or descriptions. In addition, for the convenience of description, some parts in the drawings are omitted, enlarged or reduced, but they do not represent the size or entire structure of the actual product.

[0031] The present invention discloses an electronic device for a method of fusing infrared and visible light targets, such as Figure 1 , Figure 2 As shown, it includes an infrared image processing board 301, a visible light image processing board 302, an integrated control board 303 and a computer program stored therein; when the computer program runs on each circuit board, the aforementioned infrared and visible light target fusion method is jointly implemented.

[0032] like Figure 2 As shown, the infrared image processing board 301 integrates an infrared image acquisition module 201 , an infrared image preprocessing module 203 , an infrared potential target extraction module 205 and an infrared track establishment and matching module 208 .

[0033] The infrared image acquisition module 201 acquires infrared image data in real time; the infrared image preprocessing module 203 performs preprocessing on the infrared image, including two-point correction, scene correction and image enhancement; the infrared latent target extraction module 205 performs target detection on the infrared image, and obtains the target's position information, time information, target size, target shape, target area and target brightness; the infrared track establishment and matching module 208 matches the extracted infrared latent target with the existing target track, calculates the target's speed information, acceleration information, motion mode information and track length; establishes a new track for the target that does not match the track, and regularly manages the track of all target tracks;

[0034] The infrared image processing board 301 adopts FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data. The infrared image acquisition module 201, infrared image preprocessing module 203, infrared potential target extraction module 205 and infrared track establishment and matching module 208 all run on the infrared image processing board 301.

[0035] Among them, the FPGA chip and DSP chip in the infrared image processing board 301 can select different models of products according to the amount of input data. For example, in some occasions with high real-time requirements, the SMQ325T-FFG900 FPGA of Shenzhen Guowei and the FT-6678 8-core DSP processor of the National University of Defense Technology can be selected for combination. The data processing capacity of this combination can reach 300MB / s, and the target detection output time consumption is ≤3ms.

[0036] The visible light image processing board 302 integrates the visible light image acquisition module 202, the visible light image preprocessing module 204, the visible light potential target extraction module 206, the ground object recognition module 207 and the visible light track establishment and matching module 209;

[0037] The visible light image acquisition module 202 acquires visible light image data in real time; the visible light image preprocessing module 204 performs preprocessing on the visible light image including image denoising and image enhancement; the visible light potential target extraction module 206 performs target detection on the visible light image to obtain the target's position information, time information, target size, target shape, target area and target brightness; the ground object recognition module 207 uses a target detection method based on deep learning to identify and distinguish ground objects and background; the visible light track establishment and matching module 209 matches the extracted visible light potential target with the existing target track, calculates the target's speed information, acceleration information, motion mode information and track length; establishes a new track for the target that does not match the track, and regularly manages the track of all target tracks;

[0038] The visible light image processing board 302 adopts an FPGA+image processor architecture and stores FPGA and image processor programs for processing visible light image data. The visible light image acquisition module 202, the visible light image preprocessing module 204, the visible light potential target extraction module 206, the ground object recognition module 207 and the visible light track establishment and matching module 209 all run on the visible light image processing board 302.

[0039] The FPGA chip and the image processing chip in the visible light image processing board 302 can be selected from different models according to the amount of input data. For example, the FPGA can be SMQ325T-FFG900 of Shenzhen Guowei or a similar product, and the image processing chip can be HI3559 produced by HiSilicon Semiconductor or FT-6678 8-core DSP processor of National University of Defense Technology or other processors suitable for image processing.

[0040] The integrated control board 303 integrates the target fusion module 210 .

[0041] The target fusion module 210 performs target fusion and classification identification on the infrared target and visible light target after adding the track according to the track characteristics and target properties, and outputs the final target information.

[0042] The integrated control board 303 includes a processor and a memory, and the memory stores a computer program. The integrated control board 303 is used to realize the fusion of infrared and visible light targets.

[0043] The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or any other conventional processor.

[0044] In order to meet the requirements of high real-time processing of big data, in some embodiments, high-speed fiber optic interfaces and high-speed PCIe (Peripheral Component Interconnect Express) interfaces are used between the circuit boards.

[0045] The infrared image processing board 301 and the visible light image processing board 302 communicate with each other via optical fiber.

[0046] The infrared image processing board 301 and the visible light image processing board 302 communicate with the integrated control board (303) via PCIe.

[0047] A method for fusing infrared and visible light targets using the aforementioned electronic device, such as Figure 3 As shown, the method comprises the following steps:

[0048] Step 1: The infrared image acquisition module 201 and the visible light image acquisition module 202 respectively acquire the infrared image and the visible light image in real time; the infrared image preprocessing module 203 performs preprocessing including two-point correction, scene correction, and image enhancement on the infrared image; the visible light image preprocessing module 204 performs image denoising and image enhancement preprocessing on the visible light image;

[0049] The infrared and visible light images acquired in real time are local images at the same azimuth and elevation angle at the same time; the infrared and visible light images acquired in real time are line scan images or area scan images.

[0050] Specifically, the infrared and visible light images can be original grayscale images with a pixel size of 640*512 or 1024*1024, or other sizes. It is not required that the pixel sizes of the infrared and visible light images are consistent, but the pitch angles and azimuth angles of the infrared and visible light images need to be consistent.

[0051] Image enhancement in preprocessing refers to enhancing the edges, brightness and details of targets in the image to make the edges of objects clearer and the brightness higher.

[0052] Step 2: The infrared potential target extraction module 205 extracts infrared potential targets from the infrared image; the ground object recognition module 207 uses the visible light image to identify the ground object background, and the visible light potential target extraction module 206 extracts visible light potential targets;

[0053] Specifically, specific targets are extracted from infrared and visible light images, such as passenger planes and drones in the sky or cars and pedestrians on the ground. At the same time, visible light images are used to identify and mark ground backgrounds such as buildings, roads, trees, and distant mountains, and potential targets that are misidentified in the ground background are eliminated.

[0054] Object and background recognition tagging uses deep learning-based target detection methods to identify and distinguish objects from background, such as convolutional neural networks or deep residual networks.

[0055] Step 3, using the ground object background marked by the visible light image to remove false alarms of the extracted infrared latent targets and visible light latent targets;

[0056] Specifically, the ground object background information in the visible light image, such as buildings, roads, trees, and distant mountains, usually has stable and structured characteristics, while the potential target generally presents characteristics that are significantly different from the background. Based on this, false alarms refer to identifying areas that do not meet the characteristics of infrared potential targets and visible light potential targets by analyzing the characteristics of the ground object background and the potential target of visible light, and eliminating the background area; the ground object background clearly marked in the visible light image can be used as a reference for eliminating false alarms. If the potential target in the infrared image does not match the ground object type in the visible light image, then the area may be a false alarm.

[0057] Step 4: the infrared track establishment and matching module 208 performs track matching or establishes a new track for the infrared latent target after false alarms are eliminated; the visible light track establishment and matching module 209 performs track matching or establishes a new track for the visible light latent target after false alarms are eliminated;

[0058] Specifically, track matching is mainly based on the target's location information, time information, target size, target shape, target area and target brightness. The previously appeared tracks are matched one by one within the set tolerance range, and the targets that meet the conditions are added to the previous tracks. Potential targets that are not matched are considered to be the first appearing targets, and new tracks are established for them.

[0059] Step 5: The target fusion module 210 performs target fusion and classification recognition on the infrared target and visible light target after adding the track according to the track characteristics and target properties, and outputs the final target information.

[0060] The track characteristics include position information, speed information, acceleration information, motion mode information, track length and time information; the target characteristics include target size, target shape, target area and target brightness.

[0061] In some embodiments, different weights can be assigned to the importance and uncertainty of information in track features and target characteristics according to application scenario requirements. The similarity between infrared targets and visible light targets is obtained through weighted calculation. Furthermore, by setting a similarity threshold, targets that meet the similarity threshold range are determined to be the same target, and the targets are classified and identified.

[0062] Different classification and recognition algorithms can be used to classify and identify the above targets according to different application scenarios. For example, the YOLO model, a target detection method based on deep learning, can be used to classify and identify aircraft, drones, pedestrians, and vehicles.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. That is, any equivalent changes and modifications made according to the content of the patent application scope of the present invention should be within the technical scope of the present invention.

Claims

1. An electronic device for infrared and visible light target fusion, comprising an infrared image processing board (301), a visible light image processing board (302), an integrated control board (303) and a computer program stored therein; Features: The infrared image processing board (301) integrates an infrared image acquisition module (201), an infrared image preprocessing module (203), an infrared latent target extraction module (205) and an infrared track establishment and matching module (208); wherein the infrared image acquisition module (201) acquires infrared image data in real time; the infrared image preprocessing module (203) performs preprocessing including two-point correction, scene correction and image enhancement on the infrared image; the infrared latent target extraction module (205) performs target detection on the infrared image to obtain the target's position information, time information, target size, target shape, target area and target brightness; the infrared track establishment and matching module (208) matches the extracted infrared latent target with the existing target track, calculates the target's speed information, acceleration information, motion mode information and track length; establishes a new track for the target that does not match the track, and regularly manages the track of all target tracks; The visible light image processing board (302) integrates a visible light image acquisition module (202), a visible light image preprocessing module (204), a visible light potential target extraction module (206), a ground object recognition module (207) and a visible light track establishment and matching module (209); wherein the visible light image acquisition module (202) acquires visible light image data in real time; the visible light image preprocessing module (204) performs preprocessing including image denoising and image enhancement on the visible light image; and the visible light potential target extraction module (206) performs Target detection is performed to obtain the target's location information, time information, target size, target shape, target area and target brightness; a ground object recognition module (207) uses a deep learning-based target detection method to identify and distinguish ground objects from backgrounds; a visible light track establishment and matching module (209) matches the extracted visible light potential target with the existing target track, and calculates the target's speed information, acceleration information, motion mode information and track length; a new track is established for the target that does not match the track, and track management is performed on all target tracks on a regular basis; The integrated control panel (303) integrates a target fusion module (210); the target fusion module (210) performs target fusion and classification identification on the infrared target and the visible light target after adding the track according to the track characteristics and target characteristics, and outputs the final target information.

2. The electronic device according to claim 1, characterized in that: The infrared image processing board (301) adopts an FPGA+DSP architecture and stores FPGA and DSP programs for processing infrared image data; The visible light image processing board (302) adopts an FPGA+image processor architecture and stores FPGA and image processor programs for processing visible light image data.

3. The electronic device according to claim 1, characterized in that: The infrared image processing board (301) and the visible light image processing board (302) communicate with each other via optical fiber, and the infrared image processing board (301) and the visible light image processing board (302) communicate with the integrated control board (303) via PCIe.

4. A method for performing infrared and visible light target fusion using the electronic device according to claims 1-3, characterized in that: The method comprises the following steps: Step 1: The infrared image acquisition module (201) and the visible light image acquisition module (202) respectively acquire the infrared image and the visible light image in real time; the infrared image preprocessing module (203) performs preprocessing including two-point correction, scene correction and image enhancement on the infrared image; and the visible light image preprocessing module (204) performs image denoising and image enhancement preprocessing on the visible light image; Step 2: extract infrared potential targets from the infrared image using the infrared potential target extraction module (205); the ground object recognition module (207) uses the visible light image to identify the ground object background, and the visible light potential target extraction module (206) extracts visible light potential targets; Step 3, using the ground object background marked by the visible light image to remove false alarms of the extracted infrared latent targets and visible light latent targets; Step 4: the infrared track establishment and matching module (208) performs track matching or establishes a new track for the infrared latent target after false alarms are eliminated; the visible light track establishment and matching module (209) performs track matching or establishes a new track for the visible light latent target after false alarms are eliminated; Step 5: The target fusion module (210) performs target fusion and classification identification on the infrared target and visible light target after adding the track according to the track characteristics and target properties, and outputs the final target information.

5. The method according to claim 4, characterized in that: The infrared and visible light images in step 1 are local images at the same azimuth and elevation angles; the infrared and visible light images acquired in real time are line scan images or area scan images.

6. The method according to claim 4, characterized in that: Image enhancement in step 1 preprocessing refers to enhancing the edge, brightness and details of the target in the image to make the edge of the object clearer and the brightness higher.

7. The method according to claim 4, characterized in that: The ground object and background recognition marking in step 2 uses a deep learning-based target detection method to identify and distinguish ground objects and background.

8. The method according to claim 4, characterized in that: The false alarm in step three refers to identifying areas that do not meet the characteristics of infrared latent targets and visible light latent targets by analyzing the characteristics of the ground background. The ground background clearly marked in the visible light image can be used as a reference for eliminating false alarms. If the latent target in the infrared image does not match the ground type in the visible light image, the area may be a false alarm.

9. The method according to claim 4, characterized in that: The track characteristics described in step five include position information, speed information, acceleration information, motion mode information, track length and time information; the target characteristics include target size, target shape, target area and target brightness.

Images