Real-time fusion method for multi-source heterogeneous image information, data information and geographic information

Abstract

The application relates to the field of embedded computer video processing technology and discloses a real-time fusion method for multi-source heterogeneous image information, data information and geographic information, which comprises the following steps: information detection and classification, image preprocessing, data preprocessing, time alignment and space alignment of the data information, data fusion, first image information, secondary alignment and fusion of the first image information and the preprocessed image information after color space unification, and the final fused image. The first image information simultaneously contains real-time data information and geographic information. The data fusion result can compensate for the large pure image fusion error, the inability to detect the target of interest behind the shielding object and the easy misjudgment, improve the fusion precision and accuracy, improve the reliability and robustness of the system, enhance the credibility and accuracy of the data, expand the time and space coverage of the system, and increase the real-time performance and visual effect of the system.

Description

Technical Field

[0001] This invention relates to the field of embedded computer video processing technology, and discloses a method for real-time fusion of multi-source heterogeneous image information, data information and geographic information. Background Technology

[0002] With the rapid development of sensor technology, the acquired multimodal images and data possess richer details, making information fusion a crucial research topic in embedded video processing. There are numerous types of sensors, such as visible light image sensors, infrared thermal imagers, laser imaging radar, millimeter-wave radar, synthetic aperture radar imagers, and multispectral / hyperspectral imagers. Each sensor has its own physical principles and imaging characteristics, and even for the same scene, different sensors will acquire different information. In practical applications, different sensor combinations must be used based on the characteristics and complexity of the specific application scenario to improve the accuracy and comprehensiveness of the acquired information.

[0003] For example, in complex battlefield environments, visible light images are insufficient for effectively detecting camouflaged enemy targets, while infrared thermal imagers can clearly show people and objects with higher temperatures. Thus, by fusing infrared and visible light images, camouflaged targets can be effectively identified. Furthermore, with the development of GIS technology, the real-time fusion of multi-source heterogeneous image information and data with geographic information, and the visualization of this data, is playing an increasingly important role in fields such as assisted navigation, approach guidance, and low-altitude search and rescue.

[0004] Information fusion is a process that uses algorithms to synthesize multi-source heterogeneous image information, data information, and geographic information of the same target scene into a single image containing rich information content. The fused image contains all the important information of the original image. As the types of sensors increase, the information obtained also becomes more diverse and complex. Information fusion technology can integrate this complex information, greatly enhancing the information content, making it easier for users to discover, identify, and determine targets, and increasing users' contextual awareness of the information content.

[0005] While numerous research findings exist regarding multi-source information fusion technology, most fusion methods involve pre-collecting information from multiple sensors and then performing fusion calculations on a high-performance PC. This results in high algorithm complexity, massive computational load, and poor real-time performance, making them unsuitable for embedded systems. Furthermore, the inconsistent installation positions and angles of various sensors lead to variations in resolution, frame rate, and other parameters of the resulting image and data information. Existing information fusion methods lack effective temporal and spatial alignment, resulting in large errors and low reliability compared to reality. Moreover, current fusion algorithms are not real-time processes with significant latency, and the lack of post-fusion image enhancement leads to low brightness and contrast in the displayed image, resulting in poor visualization. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time fusion method for multi-source heterogeneous image information, data information, and geographic information. This method can overcome the shortcomings of pure image fusion, such as large errors, inability to detect targets of interest behind occlusions, susceptibility to misjudgment, long algorithm processing delays, and low reliability. It improves the fusion accuracy and precision, thereby ensuring high system reliability and robustness, enhancing data credibility and accuracy, expanding the system's temporal and spatial coverage, and increasing the system's real-time performance and visualization effects.

[0007] To achieve the above-mentioned technical effects, the technical solution adopted by the present invention is as follows:

[0008] Real-time fusion methods for multi-source heterogeneous image information, data information, and geographic information, including:

[0009] Establish a sensor information classification database, and detect and classify multi-source heterogeneous sensor information based on the sensor information classification database to distinguish between image information and data information;

[0010] The image information and data information are preprocessed separately to obtain preprocessed image information and preprocessed data information;

[0011] After aligning the preprocessed data and geographic information in time, they are then aligned in space. The preprocessed data and geographic information with the same time and space are then fused together, and the fused preprocessed data and geographic information are used to create the first image information.

[0012] The preprocessed image information and the first image information are processed in a unified color space, then time-aligned, and then spatially aligned.

[0013] Target detection is performed on preprocessed image information and first image information with the same time and space to obtain common features between the preprocessed image information and first image information with the same time and space.

[0014] Based on the common features obtained, the preprocessed image information and the first image information with the same time and space are registered. Then, the preprocessed image information and the first image information with the same time and space are fused and enhanced. Finally, the fused and enhanced image is output to the display device for display.

[0015] Furthermore, in the multi-source heterogeneous sensor information, the image information includes, but is not limited to, information acquired by CMOS cameras, infrared thermal imagers, and laser imaging radars; the data information includes, but is not limited to, information acquired by synthetic aperture radars and millimeter-wave radars.

[0016] Furthermore, the methods for preprocessing image information include filtering the image information to obtain preprocessed image information; the methods for preprocessing data information include format conversion of the data information to obtain preprocessed data information containing the latitude, longitude, and altitude information of the target.

[0017] Furthermore, the time alignment method includes: unifying the time parameters of the information to be aligned, wherein the time parameters include the frame rate.

[0018] Furthermore, the spatial alignment method includes aligning the spatial parameters of the information to be aligned, the spatial parameters including viewpoint and resolution.

[0019] Furthermore, methods for performing target detection on preprocessed image information and first image information at the same time and space to obtain common features between the preprocessed image information and first image information at the same time and space include:

[0020] An algorithm is used for target detection to detect common features between preprocessed image information and first image information at the same time and space. The algorithm includes neural network algorithm, YOLO algorithm, SSD algorithm or RetinaNet algorithm.

[0021] Furthermore, methods for image fusion and image enhancement of preprocessed image information and first image information that are registered and have a unified time and space include:

[0022] An image fusion algorithm is used to fuse the registered preprocessed image information and the first image information in a unified time and space. One or more of nonlocal mean filtering, brightness adjustment or contrast adjustment are used to enhance the fused image information.

[0023] Compared with the prior art, the beneficial effects of this invention are:

[0024] 1. This invention effectively improves the accuracy and reliability of fused information through time alignment and spatial alignment, avoiding the problem that the time and spatial parameters of the obtained image and data information are different due to the inconsistent installation positions and angles of various sensors and their different performance indicators; and by enhancing the fused image information, it improves the brightness and contrast of the displayed screen and enhances the visualization effect.

[0025] 2. This invention fully utilizes the heterogeneous characteristics of multi-source sensors and the data advantages of geographic information systems to provide strong information support for real-time fusion decision-making. It can improve the reliability and robustness of the system, enhance the credibility and accuracy of the data, expand the temporal and spatial coverage of the system, and increase the real-time performance and visualization effect of the system.

[0026] 3. This invention first aligns the data information in time and space before performing data fusion to obtain the first image information. Then, the first image information and the preprocessed image information are aligned and fused again after color space unification to obtain the final fused image. The first image information contains both real-time data information and geographic information. The data fusion result can make up for the shortcomings of pure image fusion, such as large error, inability to detect the target of interest behind occlusion, and easy misjudgment, thereby improving the fusion accuracy and precision. Attached Figure Description

[0027] Figure 1 This is a flowchart of the real-time fusion method of multi-source heterogeneous image information, data information and geographic information in Example 1 or 2. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the above-described subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0029] Example 1

[0030] See Figure 1 Real-time fusion methods for multi-source heterogeneous image information, data information, and geographic information, including:

[0031] Establish a sensor information classification database, and detect and classify multi-source heterogeneous sensor information based on the sensor information classification database to distinguish between image information and data information;

[0032] The image information and data information are preprocessed separately to obtain preprocessed image information and preprocessed data information;

[0033] After aligning the preprocessed data and geographic information in time, they are then aligned in space. The preprocessed data and geographic information with the same time and space are then fused together, and the fused preprocessed data and geographic information are used to create the first image information.

[0034] The preprocessed image information and the first image information are processed in a unified color space, then time-aligned, and then spatially aligned.

[0035] Target detection is performed on preprocessed image information and first image information with the same time and space to obtain common features between the preprocessed image information and first image information with the same time and space.

[0036] Based on the common features obtained, the preprocessed image information and the first image information with the same time and space are registered. Then, the preprocessed image information and the first image information with the same time and space are fused and enhanced. Finally, the fused and enhanced image is output to the display device for display.

[0037] In this embodiment, firstly, multi-source heterogeneous sensor information is detected and classified to distinguish between image information and data information, which are then preprocessed separately. Secondly, the data information and geographic information are aligned temporally and spatially and fused. Then, a 2D / 3D graphics rendering engine is used to render the fused information as image information. At this point, all multi-source data information and geographic information have been converted into image information, and all subsequent operations are based on the image information. Next, image fusion is achieved through operations such as unified color space, target detection, and image registration. Finally, to address issues such as low brightness and contrast in the fused image, image enhancement is performed, and the final fused image is output and displayed through display control logic. This embodiment first aligns the data information temporally and spatially before fusing the data to obtain the first image information. Then, the first image information and the preprocessed image information are aligned and fused a second time after color space unification to obtain the final fused image. The first image information simultaneously contains real-time data information and geographic information. The data fusion result can compensate for the shortcomings of pure image fusion, such as large errors, inability to detect targets of interest behind occlusions, easy misjudgment, long algorithm processing delay, and low reliability, thereby improving the fusion accuracy and precision. This ensures that the system has high reliability and robustness, enhances the credibility and accuracy of the data, expands the temporal and spatial coverage of the system, and increases the real-time performance and visualization effect of the system.

[0038] In this embodiment, the image information in the multi-source heterogeneous sensor information includes, but is not limited to, information acquired by CMOS cameras, infrared thermal imagers, and laser imaging radars; the data information includes, but is not limited to, information acquired by synthetic aperture radars and millimeter-wave radars.

[0039] Example 2

[0040] See Figure 1 This embodiment takes a certain visual navigation system as an example to describe in detail the process of the real-time fusion method of multi-source heterogeneous image information, data information and geographic information of the present invention. The specific implementation steps are as follows:

[0041] Step 1: Establish a sensor information classification database. Based on the sensor information classification database, detect and classify information from multi-source heterogeneous sensors, and distinguish between image information and data information.

[0042] In this embodiment, the visual navigation system is equipped with sensors such as a CMOS camera, an infrared thermal imager, a laser imaging radar, and a millimeter-wave radar, and is equipped with a large-capacity storage unit. The storage unit is used to store geographic information such as vector data, raster data, satellite orthophotos, digital elevation models, and obstacle data.

[0043] Step 2: Preprocess the image information and data information separately to obtain preprocessed image information and preprocessed data information;

[0044] In this embodiment, the method for preprocessing image information includes filtering the image information to obtain preprocessed image information; the method for preprocessing data information includes format conversion of the data information to obtain preprocessed data information containing the latitude, longitude, and altitude information of the target.

[0045] Step 3: After aligning the preprocessed data and geographic information in time, align the preprocessed data and geographic information in space, fuse the preprocessed data and geographic information with the same time and space, and draw the fused preprocessed data and geographic information into the first image information.

[0046] Step 4: Perform unified color space processing on the preprocessed image information and the first image information, then perform time alignment, and then perform spatial alignment on the preprocessed image information and the first image information.

[0047] In steps three and four of this embodiment, the time alignment method involves unifying the time parameters of the information to be aligned, including the frame rate. The spatial alignment method involves aligning the spatial parameters of the information to be aligned, including the viewpoint and resolution.

[0048] Step 5: Perform target detection on the preprocessed image information and the first image information in the same time and space to obtain the common features between the preprocessed image information and the first image information in the same time and space.

[0049] In this embodiment, an algorithm is used for target detection to detect common features between preprocessed image information and first image information at the same time and space. The algorithm includes neural network algorithm, YOLO algorithm, SSD algorithm or RetinaNet algorithm.

[0050] Step 6: Based on the obtained common features, register the preprocessed image information and the first image information with the same time and space. Then, perform image fusion and image enhancement on the preprocessed image information and the first image information with the same time and space and the registration. Finally, output the fused and enhanced image to the display device for display.

[0051] In this embodiment, an image fusion algorithm is used to fuse the registered pre-processed image information and the first image information that are in the same time and space. One or more of non-local mean filtering, brightness adjustment or contrast adjustment are used to enhance the fused image information.

[0052] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time fusion of multi-source heterogeneous image information, data information, and geographic information, characterized in that, include: Establish a sensor information classification database, and detect and classify multi-source heterogeneous sensor information based on the sensor information classification database to distinguish between image information and data information; The image information and data information are preprocessed separately to obtain preprocessed image information and preprocessed data information; After aligning the preprocessed data and geographic information in time, they are then aligned in space. The preprocessed data and geographic information with the same time and space are then fused together, and the fused preprocessed data and geographic information are used to create the first image information. The preprocessed image information and the first image information are processed in a unified color space, then time-aligned, and then spatially aligned. Target detection is performed on preprocessed image information and first image information with the same time and space to obtain common features between the preprocessed image information and first image information with the same time and space. Based on the common features obtained, the preprocessed image information and the first image information with the same time and space are registered. Then, the preprocessed image information and the first image information with the same time and space are fused and enhanced. Finally, the fused and enhanced image is output to the display device for display.

2. The real-time fusion method of multi-source heterogeneous image information, data information, and geographic information according to claim 1, characterized in that, In the multi-source heterogeneous sensor information, the image information includes, but is not limited to, information acquired by CMOS cameras, infrared thermal imagers, and laser imaging radars; the data information includes, but is not limited to, information acquired by synthetic aperture radars and millimeter-wave radars.

3. The real-time fusion method for multi-source heterogeneous image information, data information, and geographic information according to claim 1, characterized in that, Methods for preprocessing image information include filtering the image information to obtain preprocessed image information; methods for preprocessing data information include format conversion of the data information to obtain preprocessed data information containing the latitude, longitude, and altitude information of the target.

4. The real-time fusion method for multi-source heterogeneous image information, data information, and geographic information according to claim 1, characterized in that, The time alignment method includes: unifying the time parameters of the information to be aligned, the time parameters including frame rate; the information to be aligned includes alignment of preprocessed data information and geographic information, and alignment of preprocessed image information and first image information; the geographic information includes vector data, raster data, satellite orthophotos, digital elevation models, and obstacle data.

5. The real-time fusion method for multi-source heterogeneous image information, data information, and geographic information according to claim 4, characterized in that, The spatial alignment method includes aligning the spatial parameters of the information to be aligned, the spatial parameters including viewpoint and resolution.

6. The real-time fusion method for multi-source heterogeneous image information, data information, and geographic information according to claim 1, characterized in that, Methods for performing target detection on preprocessed image information and first image information at the same time and space, and obtaining common features between the preprocessed image information and the first image information at the same time and space, include: An algorithm is used for target detection to detect common features between preprocessed image information and first image information at the same time and space. The algorithm includes neural network algorithm, YOLO algorithm, SSD algorithm or RetinaNet algorithm.

7. The real-time fusion method for multi-source heterogeneous image information, data information, and geographic information according to claim 1, characterized in that, Methods for image fusion and image enhancement using preprocessed image information and first image information that are registered and have a unified time and space include: An image fusion algorithm is used to fuse the registered preprocessed image information and the first image information in a unified time and space. One or more of nonlocal mean filtering, brightness adjustment or contrast adjustment are used to enhance the fused image information.

Images