A distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration

Abstract

The invention discloses a distributed multi-source image interactive fusion method based on multi-level time-space calibration, which relates to the technical field of signal processing and comprises the following steps: step 1, time synchronization calibration; inputting an optical image and an infrared image, and synchronizing the acquisition time of the optical image and the infrared image through time synchronization calibration; step 2, time interpolation compensation calibration, correcting the time offset of the optical image and the infrared image; step 3, posture calibration, aligning the optical image and the infrared image to a unified spatial coordinate system, eliminating the deviation caused by different viewing angles; step 4, geometric distortion calibration, correcting the geometric distortion of the image; step 5, interactive image fusion, fusing the optical image and the infrared image through a deep learning image processing method; the method provided by the invention effectively solves the data deviation caused by time, space and posture differences between different sensors and imaging systems.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and in particular to a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration. Background Art

[0002] With the development of sensing technology, various types of sensors can widely cover monitoring and detection tasks in different fields. For example, optical sensors, infrared sensors, radars and other sensors can obtain different information about targets or scenes at the same time. By fusing data from multiple sensors, the advantages of different sensors can be fully utilized to perform target detection, tracking and recognition tasks. Distributed multi-source image fusion technology is based on this background. Distributed multi-source image fusion refers to the fusion of data from sensors at different spatial locations to obtain richer and more comprehensive information. Specifically, a single source image can often only provide limited information: optical images are known for their high-resolution color information and can clearly show the appearance characteristics of the target; while infrared images can capture the thermal radiation information of the target and reflect the temperature distribution of the target. When these two or more types of images are fused, they can complement each other to form a more complete target feature description, thereby significantly improving the accuracy and robustness of target recognition.

[0003] However, distributed multi-source image fusion is not an easy task, and it faces many challenges. First, different sensors may adopt different imaging principles and technical parameters, resulting in differences in image resolution, color space, lighting conditions, etc. Secondly, since sensors are distributed in different spatial locations, the problem of spatiotemporal synchronization between images also needs to be properly solved. In addition, how to effectively retain and integrate the key information of various images in the fusion process to avoid information loss or redundancy is also a major problem. In response to the above challenges, researchers have proposed many solutions. For example, in the application document with the publication number CN118279768A and the name "MSFR-based UAV aerial multi-source ship image fusion and recognition method and equipment", the problem of pixel deviation in multi-source images collected by sensors is solved by designing an end-to-end network MSFR, using spatial relationships to jointly learn modulation parameters and dynamic convolution kernels, so that images of different modalities are aligned in the high-level feature space, and accurate registration and fusion of visible light and infrared images are achieved, which improves the fusion effect and recognition performance. For example, in the application document with publication number CN118780997B and titled "UAV visible light-infrared image target-level fusion method and system", a target-level fusion strategy is adopted. It first extracts the target slices in the visible light and infrared images, performs registration, uses unsupervised methods to perform semantic segmentation, and injects the temperature information in the infrared image into the visible light region segmentation map, and finally obtains the fusion result through guided filtering. This method effectively solves the problems of precise alignment and resolution difference between visible light and infrared images.

[0004] Although the technical means proposed in these studies have achieved the effective fusion of optical and infrared images of drones, most of them focus on the fusion of sensor images on the same platform and the same payload, and do not consider the spatiotemporal differences between images on heterogeneous platforms. To address this limitation, the present invention proposes a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration to address the shortcomings of existing methods. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration to solve the problem of data deviation caused by time, space and posture differences between different sensors and imaging systems in the prior art.

[0006] To achieve the above object, the present invention provides a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration, comprising the following steps:

[0007] Step 1, time synchronization calibration: input the optical image and the infrared image, and synchronize the acquisition time of the optical image and the infrared image through time synchronization calibration;

[0008] Step 2: time interpolation compensation calibration to correct the time offset of the optical image and the infrared image;

[0009] Step 3: Attitude calibration: align the optical image and infrared image to a unified spatial coordinate system to eliminate the deviation caused by different viewing angles.

[0010] Step 4: Geometric distortion calibration, correcting the geometric distortion of the image;

[0011] Step 5: Interactive image fusion: fuse the optical image and infrared image through deep learning image processing method.

[0012] Preferably, the process of time synchronization calibration in step 1 is as follows:

[0013] S11. Obtain the timestamp of each sensor using the global positioning system (GPS) or a satellite-based high-precision time synchronization module; assuming that the time stamps of each sensor are and These timestamps can be synchronized via GPS or aligned to a reference clock;

[0014] S12. Set a unified time target The images collected by each sensor are time-corrected by multi-dimensional interpolation method. The calculation expression is as follows:

[0015] ;

[0016] In the formula, is the actual acquisition time of the sensor;

[0017] S13. Use Lagrange interpolation method to correct the time point. The expression is as follows:

[0018] ;

[0019] In the formula, is the Lagrangian basis function, is the timestamp of the known sampling point, Represents the time point corrected by the Lagrange interpolation method.

[0020] Preferably, the process of time interpolation compensation calibration in step 2 is as follows:

[0021] S21, timestamp of the sensor and actual acquisition time of the sensor , establish the time interval , and perform interpolation compensation based on the spline interpolation algorithm. The expression is as follows:

[0022] ;

[0023] In the formula, For time The corresponding image pixel value, is a spline-based interpolation function, is the interpolation coefficient;

[0024] S22, interpolate each image point in S21 to obtain a compensated image. Through this method, the images of all sensors can be accurately aligned to the target time point to avoid errors caused by time differences.

[0025] Preferably, the process of posture calibration in step 3 is as follows:

[0026] S31, assuming that the attitude information (pitch angle, yaw angle and roll angle) of each sensor is provided by the inertial measurement unit (IMU), using the rotation matrix Correct the image, the expression is as follows:

[0027] ;

[0028] In the formula, is the pitch angle, is the yaw angle, is the roll angle;

[0029] S32, use the rotation matrix to perform spatial transformation on the image, set is the point in the coordinate system of the optical and infrared images, and the point in the unified coordinate system is mapped through the attitude matrix. The expression is as follows:

[0030] ;

[0031] In the formula, Indicates the points in the unified coordinate system after mapping.

[0032] Preferably, the process of geometric distortion calibration in step 4 is as follows:

[0033] S41. Use the standard Zhang Zhengyou camera calibration method and the checkerboard calibration pattern to obtain distortion parameters and build a distortion model , the expression is as follows:

[0034] ;

[0035] In the formula, , , is the radial distortion coefficient, is the distance from the point to the center of the image, represents the image before geometric distortion calibration;

[0036] S42, perform distortion correction on each image to restore the true geometric structure of the image. Correct each pixel point one by one, calculate the corrected pixel position, and obtain the final geometrically corrected image.

[0037] Preferably, the process of interactive image fusion in step 5 is as follows:

[0038] S51. Construct an interactive fusion network to extract the deep features of optical images from images through the interactive fusion network and depth features of infrared images , and fuse them through the convolutional neural network fusion module to obtain deep feature fusion , the expression is as follows:

[0039] ;

[0040] in, It is the convolutional neural network fusion module;

[0041] S52. Training convolutional neural network model, loss function of the training process The expression is as follows:

[0042] ;

[0043] In the formula, For content loss, is the style loss, is the edge loss, To smooth the loss, 、 、 、 is a hyperparameter.

[0044] Therefore, the present invention adopts the above-mentioned distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration, which has the following beneficial effects:

[0045] (1) Time synchronization calibration ensures that images are collected at the same time point, spatial calibration corrects geometric distortion, and posture calibration adjusts device angle differences, thereby improving the matching accuracy of images from different sources and providing a high-precision data foundation for subsequent feature extraction and fusion;

[0046] (2) Image features are gradually extracted through multiple convolutional layers and pooling layers, and feature interaction is performed at each layer. In the process of multi-source image fusion, the interactive fusion network can capture the complementary information in various images, thereby improving the quality of the fused image.

[0047] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is an overall flow chart of a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration of the present invention;

[0049] Figure 2 It is a fusion structure diagram of the interactive fusion network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1-Figure 2 , a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration, comprising the following steps:

[0052] Step 1, time synchronization calibration; input optical images and infrared images, and synchronize the acquisition time of optical images and infrared images through time synchronization calibration; sensors are installed on different hardware platforms, and there are time synchronization errors. In order to ensure that the data collected by different sensors can be aligned at the same time, time synchronization calibration must be performed. This step eliminates the time difference between different sensors through precise clock synchronization technology to ensure consistency in subsequent processing. The specific process is as follows:

[0053] S11. Obtain the timestamp of each sensor using the global positioning system (GPS) or a satellite-based high-precision time synchronization module; assuming that the time stamps of each sensor are and These timestamps can be synchronized via GPS or aligned to a reference clock;

[0054] S12. Set a unified time target The images collected by each sensor are time-corrected by multi-dimensional interpolation method. The calculation expression is as follows:

[0055] ;

[0056] In the formula, is the actual acquisition time of the sensor;

[0057] S13. Use Lagrange interpolation method to correct the time point. The expression is as follows:

[0058] ;

[0059] In the formula, is the Lagrangian basis function, is the timestamp of the known sampling point, Represents the time point corrected by the Lagrange interpolation method.

[0060] Through this process, it can be ensured that all images are aligned at the same time, providing a synchronization basis for subsequent image processing.

[0061] Step 2: Time interpolation compensation calibration to correct the time offset of optical and infrared images. Even after the initial time synchronization, there may still be time deviations between sensors, especially in high-dynamic environments. Time interpolation compensation calibration further corrects the time offset of the image through the interpolation algorithm to ensure time accuracy in dynamic scenes. The specific process is as follows:

[0062] S21, timestamp of the sensor and actual acquisition time of the sensor , establish the time interval , and perform interpolation compensation based on the spline interpolation algorithm. The expression is as follows:

[0063] ;

[0064] In the formula, For time The corresponding image pixel value, is a spline-based interpolation function, is the interpolation coefficient;

[0065] S22, interpolate each image point in S21 to obtain a compensated image. Through this method, the images of all sensors can be accurately aligned to the target time point to avoid errors caused by time differences.

[0066] Step 3: Attitude calibration, align the optical image and infrared image to a unified spatial coordinate system to eliminate the deviation caused by different viewing angles; the purpose of attitude calibration is to solve the difference in viewing angles caused by factors such as sensor installation position, pitch angle, roll angle, etc. Through attitude calibration, multi-source images can be aligned to a unified spatial coordinate system to eliminate the deviation caused by different viewing angles; the specific process is as follows:

[0067] S31, assuming that the attitude information (pitch angle, yaw angle and roll angle) of each sensor is provided by the inertial measurement unit (IMU), using the rotation matrix Correct the image, the expression is as follows:

[0068] ;

[0069] In the formula, is the pitch angle, is the yaw angle, is the roll angle;

[0070] S32, use the rotation matrix to perform spatial transformation on the image, set is the point in the coordinate system of the optical and infrared images, and the point in the unified coordinate system is mapped through the attitude matrix. The expression is as follows:

[0071] ;

[0072] In the formula, Indicates the points in the unified coordinate system after mapping.

[0073] Through the attitude calibration process, all images are aligned in a unified coordinate system to eliminate geometric errors caused by sensor attitude differences.

[0074] Step 4: Geometric distortion calibration, correcting the geometric distortion of the image; Geometric distortion may occur during the sensor imaging process, mainly due to the lens effect and the imaging error of the sensor; Geometric distortion calibration corrects the geometric distortion of the image by solving the distortion parameters, making the fused image more realistic and accurate; the specific process is as follows:

[0075] S41. Use the standard Zhang Zhengyou camera calibration method and the checkerboard calibration pattern to obtain distortion parameters and build a distortion model , the expression is as follows:

[0076] ;

[0077] In the formula, , , is the radial distortion coefficient, is the distance from the point to the center of the image, represents the image before geometric distortion calibration;

[0078] S42, perform distortion correction on each image to restore the true geometric structure of the image. Correct each pixel point one by one, calculate the corrected pixel position, and obtain the final geometrically corrected image.

[0079] Step 5: Interactive image fusion: The optical image and infrared image are fused by deep learning image processing method. The specific process is as follows:

[0080] S51. Construct an interactive fusion network to extract the deep features of optical images from images through the interactive fusion network and depth features of infrared images , and fuse them through the convolutional neural network fusion module to obtain deep feature fusion , the expression is as follows:

[0081] ;

[0082] in, It is the convolutional neural network fusion module;

[0083] S52. Training convolutional neural network model, loss function of the training process The expression is as follows:

[0084] ;

[0085] In the formula, For content loss, is the style loss, is the edge loss, To smooth the loss, 、 、 、 is a hyperparameter.

[0086] The final fused image will have both the high spatial resolution of optical images and the strong temperature sensing ability of infrared images, and is suitable for high-precision target detection in dynamic scenes.

[0087] Therefore, the present invention adopts the above-mentioned distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration, and realizes the fusion of optical images and infrared images through five processes of time synchronization calibration, time interpolation compensation calibration, attitude calibration, geometric distortion calibration, and interactive image fusion, thereby effectively solving the data deviation caused by time, space and attitude differences between different sensors and imaging systems.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration, characterized in that: The following steps are involved: Step 1: Time synchronization calibration; Input an optical image and an infrared image, and synchronize the acquisition time of the optical image and the infrared image through time synchronization calibration; S11, obtaining the timestamp of each sensor using a global positioning system or a satellite-based high-precision time synchronization module; S12. Set a target time The images collected by each sensor are time-corrected by multi-dimensional interpolation method. The calculation expression is as follows: ； In the formula, is the actual acquisition time of the sensor; S13. Use Lagrange interpolation method to correct the time point. The expression is as follows: ； In the formula, is the Lagrangian basis function, is the timestamp of the known sampling point, represents the time point corrected by Lagrange interpolation method; Step 2: time interpolation compensation calibration to correct the time offset of the optical image and the infrared image; Step 3: Attitude calibration: align the optical image and infrared image to a unified spatial coordinate system to eliminate the deviation caused by different viewing angles. S31, assuming that the attitude information of each sensor is provided by an inertial measurement unit, using the rotation matrix Correct the image, the expression is as follows: ； In the formula, is the pitch angle, is the yaw angle, is the roll angle; S32, use the rotation matrix to perform spatial transformation on the image, set is the point in the coordinate system of the optical and infrared images, and the point in the unified coordinate system is mapped through the attitude matrix. The expression is as follows: ； In the formula, Indicates the points in the unified coordinate system after mapping; Step 4: Geometric distortion calibration, correcting the geometric distortion of the image; Step 5: Interactive image fusion: fuse the optical image and infrared image through deep learning image processing method.

2. According to claim 1, a distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration is characterized in that: The process of time interpolation compensation calibration in step 2 is as follows: S21, timestamp of the sensor and actual acquisition time of the sensor , establish the time interval , and perform interpolation compensation based on the spline interpolation algorithm. The expression is as follows: ； In the formula, For time The corresponding image pixel value, is a spline-based interpolation function, is the interpolation coefficient; S22, performing interpolation of S21 on each image point to obtain a compensated image.

3. The distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration according to claim 1 is characterized in that: The process of geometric distortion calibration in step 4 is as follows: S41. Use the standard Zhang Zhengyou camera calibration method and the checkerboard calibration pattern to obtain distortion parameters and build a distortion model , the expression is as follows: ； In the formula, , , is the radial distortion coefficient, is the distance from the point to the center of the image, represents the image before geometric distortion calibration; S42, performing distortion correction on each image to restore the real geometric structure of the image.

4. The distributed multi-source image interactive fusion method based on multi-level spatiotemporal calibration according to claim 3 is characterized in that: The process of interactive image fusion in step 5 is as follows: S51. Construct an interactive fusion network to extract the deep features of optical images from images through the interactive fusion network and depth features of infrared images , and fuse them through the convolutional neural network fusion module to obtain deep feature fusion , the expression is as follows: ； in, It is the convolutional neural network fusion module; S52. Training convolutional neural network model, loss function of the training process The expression is as follows: ； In the formula, For content loss, is the style loss, is the edge loss, To smooth the loss, 、 、 、 is a hyperparameter.

Images