Sea surface target tracking method and system based on infrared and visible light image fusion

Abstract

The invention provides a sea surface target tracking method and system based on infrared and visible light image fusion, and relates to the technical field of ocean monitoring, and the method comprises the steps: collecting visible light and infrared image data of a sea surface target; performing feature extraction and fusion through wavelet transform fusion to generate a comprehensive feature map, and performing target recognition on the comprehensive feature map by using a deep learning target detection model; after target recognition, the system calculates the position of a target based on image data and radar data, performs multi-target matching and association through a Hungary algorithm combined with multi-modal features, predicts the position of the target and updates trajectory information in combination with a Kalman filtering or particle filtering algorithm. The infrared camera and the visible light camera carry out dynamic angle adjustment according to the position and the movement track of the target; whether light information correction is carried out or not is judged based on the light correction threshold value, when light information correction is carried out, light information correction features are constructed based on the visible light compensation model and the infrared compensation model through the image data, and information errors caused by the illumination angle are eliminated.

Description

Technical Field

[0001] This application relates to the field of marine monitoring technologies, and more specifically, to a sea surface target tracking method and system based on infrared and visible light image fusion. Background Art

[0002] Most traditional sea surface target monitoring devices are based on a single sensor, such as a radar or a visible light camera. However, these devices have obvious limitations in some special environments. The performance of a visible light camera is limited under low light or night conditions, while radar and infrared cameras perform poorly in high-precision target recognition. To solve this problem, multi-modal data fusion technology has gradually become a research hotspot in the field of maritime target monitoring. By fusing infrared images with visible light images, the stability of infrared under low light conditions and the high resolution advantage of visible light can be combined, so as to achieve all-weather and high-precision monitoring of sea surface targets. The existing methods for fusing infrared images with visible light images include: (1) The Chinese invention patent with the publication number CN104200452A discloses an infrared and visible light image fusion method and device based on spectral graph wavelet transform, including performing spectral graph wavelet decomposition on the infrared image and the visible light image according to a four-layer decomposition scale to obtain low-frequency subband coefficients and high-frequency subband coefficients, fusing the obtained low-frequency subband coefficients and high-frequency subband coefficients at different decomposition scales according to a fusion rule, and performing spectral graph wavelet inverse transform on the fused low-frequency subband coefficients and high-frequency subband coefficients to obtain a fused image.

[0003] (2) The Chinese invention patent with the publication number CN119579436A discloses a fusion method and system for infrared and visible light images based on CRITIC weights, including decomposing the input visible light image and infrared image into a feature map and an original map based on a Gaussian filter; calculating corresponding variable weight coefficients based on the visible light original map and the visible light feature map respectively; fusing the visible light original map and the infrared original map based on the variable weight coefficients and obtaining a visible light fused map; fusing the visible feature map and the infrared feature map based on the variable weight coefficients and obtaining an infrared fused map; fusing the visible light fused map and the infrared fused map to obtain a final fused map; (3) The Chinese invention patent with the publication number CN119206419A discloses an infrared and visible light image fusion method and system based on high-low frequency separation enhancement, including obtaining an infrared image and a visible light image to be fused; performing image fusion on the infrared image and the visible light image to be fused by using an image fusion network to obtain a fused image; wherein, the image fusion step includes: extracting shallow features of the infrared image and the visible light image, processing the shallow features by using a feature separation algorithm to obtain high-frequency features and low-frequency features, performing feature enhancement on the high-frequency features by using a feature enhancement algorithm, and fusing the enhanced high-frequency features with the low-frequency features to obtain a fused image.

[0004] It can be seen that the research and development direction of the existing technology focuses on processing the visible light and infrared image information to be fused to improve the fusion effect. However, the existing solutions do not consider the characteristic changes of the target to be recognized and its interaction state with light in the actual usage scenario, so there is a problem of tracking loss. Summary of the Invention

[0005] To solve the above problems, the technical solution adopted in this application is a sea target tracking method based on infrared and visible light image fusion, including: Information acquisition: Synchronously collect the image data of the sea target through an infrared camera and a visible light camera; Image data processing: Denoise, enhance, and register the image data; the processed image data is subjected to feature extraction and fusion through wavelet transform fusion to generate a comprehensive feature map, and a deep learning object detection model is used to perform object recognition on the comprehensive feature map; Target tracking: After completing object recognition, the system enters the target tracking mode, calculates the target position based on the image data and radar data, performs multi-target matching and association through the Hungarian algorithm that combines multi-modal features, and combines the Kalman filter or particle filter algorithm to predict the next position of the target, update the target trajectory information, and the infrared camera and visible light camera perform dynamic angle adjustment according to the position and movement trajectory of the target to ensure that the target is always within the field of view of the camera; Light information correction: Judge whether to perform light information correction based on the light correction threshold. When performing light information correction, based on the image data, construct light information correction features based on the visible light compensation model and the infrared compensation model, and combine with the Hungarian algorithm during target tracking to construct a cost matrix for multi-target matching and association.

[0006] Optionally, calculating the target position based on the image data and radar data includes taking the center of the horizontal positions of the radar, visible light, and infrared camera modules as the origin to establish a coordinate system, and the three-dimensional coordinates of the target are obtained based on the following formula: ; ; ; In the formula, , , respectively represent the three-dimensional coordinates of the target, R is the distance between the target and the radar, is the pitch angle of the target relative to the camera, is the azimuth angle of the target relative to the radar.

[0007] Optionally, feature extraction and fusion through wavelet transform fusion includes performing wavelet transform processing on the image according to the following steps: ; ; In the formula, and respectively represent the wavelet coefficients of the visible light and infrared images, is the wavelet basis function, represents the spatial coordinates of the image, m represents the index variable, represents the scale of the wavelet transform, represents the translation parameter in the wavelet transform, and respectively represent the visible light and infrared images; Perform weighted fusion on the wavelet coefficients of the visible light and infrared images: ; In the formula, and are respectively the fusion weights of the wavelet coefficients of the visible light and infrared images; represents the fused wavelet coefficient; Reconstruct the fused image through inverse wavelet transform: ; In the formula, is the fused image, represents the scaling function.

[0008] Optionally, target recognition includes extracting high-frequency information based on high-frequency units through discrete cosine transform, and the formula is as follows: ; In the formula, , represents the high-frequency spectrum, and are respectively the height and width of the image, and respectively represent the position indices of the frequency components, x i,j represents the pixel value of the image at the position ( i, j ), represents the value of the orthogonal basis function at the position ( i, j ); The low-frequency unit uses a multi-scale convolution kernel to capture the low-frequency structure and background information of the sea surface target. The dilation rate and the convolution kernel size of the low-frequency convolution kernel satisfy the following constraints: ; In the formula, is the receptive field size.

[0009] Optionally, the target recognition outputs the category, bounding box position, and confidence of the target, and adopts a CIoU-based loss function to optimize the fitting of the bounding box. The formula is as follows: ; In the formula, represents the square of the Euclidean distance between the center points of the predicted box and the ground truth box, represents the predicted box, represents the center box, is the length of the diagonal of the minimum bounding rectangle of the two, is the aspect ratio consistency term, α is the balance parameter.

[0010] Optionally, the matching and association of multiple targets through the Hungarian algorithm includes constructing a cost matrix that combines the Euclidean distance between comprehensive target positions and the multimodal feature similarity. The formula is: ; In the formula, and respectively represent the multimodal feature vectors of the trajectory and the target , and are the environment adaptive weights; The Hungarian algorithm takes the cost matrix as input and determines the optimal matching scheme between the trajectory and the target by minimizing the total cost. The optimization objective is: ; In the formula, is the binary matching matrix. If the target matches the trajectory , then , otherwise it is 0.

[0011] Optionally, the light correction threshold includes a visible light compensation enable threshold and an infrared compensation enable threshold. The visible light compensation enable threshold is calculated based on the following formula: ; In the formula, represents the angle between the sunlight and the normal of the side of the object facing the camera, I a represents the ambient light intensity, I s represents the direct sunlight intensity. When the above formula is satisfied, visible light compensation is enabled; The infrared compensation enable threshold is calculated based on the following formula: ; In the formula, represents the theoretical temperature rise, I represents the solar radiation intensity received by the ground surface, represents the time interval for obtaining infrared images, represents the material density, d represents the material thickness, represents the specific heat capacity of the material. When , infrared compensation is enabled; represents the noise standard deviation of the infrared temperature sensor.

[0012] Optionally, the visible light compensation model includes: Calculating the solar altitude angle and azimuth angle, calculating the angle between the sunlight and the normal of the side of the object facing the camera based on the target position and the normal vector of the side of the object facing the camera, calculating the target theoretical illumination intensity according to the sunlight intensity calibrated by the photometer, and taking the target theoretical illumination intensity as the visible light supplementary feature; The infrared compensation model includes: Taking the theoretical temperature rise as the infrared supplementary feature; In the light information correction step, a cost matrix is constructed in combination with the Hungarian algorithm for multi-target matching and association. The constructed cost matrix is: ; In the formula, and respectively represent the visible light supplementary features of the trajectory and the target , and respectively represent the infrared supplementary features of the trajectory and the target .

[0013] Optionally, the visible light compensation model and the infrared compensation model include calculating visible light image correction and infrared temperature correction, calculating the visible light credibility factor and the infrared credibility factor based on the visible light attenuation and the infrared attenuation, and further calculating the visible light weight and the infrared weight as the fusion weights of the visible light and infrared image wavelet coefficients; In the light information correction step, constructing a cost matrix in combination with the Hungarian algorithm for multi-target matching and association means taking the visible light weight and the infrared weight as the fusion weights of the visible light and infrared image wavelet coefficients.

[0014] This application also provides a sea surface target tracking system based on the fusion of infrared and visible light images, which is carried out by using any one of the foregoing sea surface target tracking methods based on the fusion of infrared and visible light images, and includes an infrared camera, a visible light camera, a fixed bracket, a connecting bracket, an edge computing module, a power module, a communication module, a data center, and a radar; An infrared camera and a visible light camera are installed on a connecting bracket for real-time acquisition of image data of sea surface targets. The relative angle is adjusted by an electric adjustment mechanism. The horizontal angle adjustment range is 0° - 360°, the vertical angle adjustment range is ±45°, and the accuracy error is 0.1° - 0.5°. It is fixed on the upper deck of the ship or the top of the observation tower. An electric drive system is provided between the connecting bracket and the fixed bracket, which can adjust the angle of the camera according to remote instructions; The fixed bracket is provided with a vibration suppression structure; The edge computing module includes a processor, a memory, and input / output interfaces. It is set in the ship control center or the equipment cabin for preprocessing image data, realizing target recognition, tracking, and multi-modal fusion by combining deep learning algorithms, storing relevant information, and interacting with the communication and data center; The communication module is used for two-way communication between the system and the external network in the ship monitoring room or the remote command center and the data center; the data center is responsible for remote operation, monitoring, and storage management.

[0015] The beneficial effects of the sea surface target tracking method and system based on infrared and visible light image fusion provided by this application are as follows: (1) Collect multi-modal sea surface target information through infrared cameras and visible light cameras to achieve all-weather monitoring of sea surface targets. After target recognition, the Hungarian algorithm combining multi-modal features is used for multi-target matching and association, avoiding the tracking loss problem of the Hungarian algorithm of a single radar information source in the case of dense targets, and ensuring the stability of multi-target tracking and the continuity of trajectories in complex marine environments; (2) There is a problem of feature change caused by relative angle change in the continuous tracking of multi-information sources. Based on the visible light compensation model and the infrared compensation model, optical information correction features are constructed. During the target tracking process, a cost matrix is constructed in combination with the Hungarian algorithm for multi-target matching and association to eliminate the information error caused by the illumination angle during the continuous tracking target recognition process; (3) Before optical information correction, it is judged whether to perform optical information correction based on the optical correction threshold, so as to increase the judgment process, improve the accuracy of target recognition at the cost of slightly increasing the system complexity and power consumption, and reduce the power consumption when optical information correction is not applicable. Description of the Drawings

[0016] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art.

[0017] Figure 1 It is a schematic flowchart of the sea surface target tracking method based on infrared and visible light image fusion provided by the embodiments of this application; Figure 2It is a schematic structural diagram of a sea surface target tracking system based on infrared and visible light image fusion provided by an embodiment of the present application; Figure 3 It is a schematic diagram of a data center provided by an embodiment of the present application; Explanation of reference numerals: 1 - infrared camera; 2 - visible light camera; 3 - fixed bracket; 4 - connecting bracket; 5 - edge computing module; 6 - power module; 7 - communication module; 8 - data center; 81 - data processing module; 82 - display module. Specific implementation manners

[0018] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] Embodiment 1 As Figure 1 shown, the present application provides a sea surface target tracking method based on infrared and visible light image fusion, including: Information acquisition: Image data of sea surface targets is synchronously acquired through an infrared camera and a visible light camera. When the system starts, the infrared camera and the visible light camera enter the standby state. When the device detects the ambient light conditions or task requirements, the system dynamically adjusts the working modes and acquisition frequencies of the infrared and visible light cameras according to preset parameters. The cameras are initialized in angle through the connecting bracket and the electric drive system to ensure that the field of view covers the set monitoring range.

[0020] Image data processing: After the image data is transmitted from the camera to the edge computing terminal, the terminal performs denoising, enhancement and registration processing on it to eliminate the influence of sea surface reflection, noise and modal differences on target detection. The processed image data is subjected to multi-scale feature extraction through wavelet transform, and a comprehensive feature map is generated in combination with a fusion algorithm. The system uses a deep learning target detection model to perform target recognition on the fused feature map. The target recognition includes performing target positioning and outputting the category, position and confidence information of the target.

[0021] Target tracking: After the target is detected, the system enters the target tracking mode, calculates the target position based on the image data and radar data, realizes the matching and association of multiple targets through the Hungarian algorithm combining multi-modal features, and predicts the next position of the target in combination with the Kalman filter or particle filter algorithm to update the target trajectory information. The infrared camera and the visible light camera perform dynamic angle adjustment according to the position and movement trajectory of the target to ensure that the target is always within the field of view of the camera. At the same time, the edge computing terminal records the distance, speed and trajectory changes of the target in real time and continuously monitors the target state.

[0022] Optical information correction: Determine whether to perform optical information correction based on the optical correction threshold. When performing optical information correction, using the image data, construct optical information correction features based on the visible light compensation model and the infrared compensation model. During the target tracking process, combine with the Hungarian algorithm to construct a cost matrix for multi-target matching and association.

[0023] During the target tracking process, the system presents the detection and tracking results on the display screen in real time, including information such as the type, movement trajectory, position, and speed of the target. The remote control platform synchronously receives the data, and the user can adjust the device parameters or intervene in the tracking process through the wireless network. When the target leaves the monitoring range or the environmental light conditions improve, the system enters the standby state, the infrared camera stops working, and only the visible light camera remains to continuously monitor the sea surface environment.

[0024] Collect multi-modal sea surface target information through the infrared camera and the visible light camera to achieve all-weather monitoring of the sea surface targets. After target recognition, perform multi-target matching and association through the Hungarian algorithm that combines multi-modal features, avoiding the tracking loss problem of the Hungarian algorithm of a single radar information source in the case of dense targets, and ensuring the stability of multi-target tracking and the continuity of trajectories in complex marine environments; There is a problem of feature change due to the relative angle change in the continuous tracking of multi-information sources. Construct optical information correction features based on the visible light compensation model and the infrared compensation model. During the target tracking process, combine with the Hungarian algorithm to construct a cost matrix for multi-target matching and association, eliminating the information error caused by the illumination angle during the continuous tracking target recognition process; Before optical information correction, determine whether to perform optical information correction based on the optical correction threshold, at the cost of slightly increasing the system complexity and power consumption to increase the judgment process, improve the accuracy of target recognition, and reduce the power consumption when optical information correction is not applicable.

[0025] Calculate the target position based on the image data and radar data, including taking the center of the horizontal positions of the radar, visible light, and infrared camera modules as the origin to establish an x-y-z coordinate system. Since the distances of the radar, visible light, and infrared camera modules are very close compared to the distance between the target and the system structure, it is approximately considered that the radar, visible light, and infrared camera modules are at the same origin. The three-dimensional coordinates of the target can be calculated from the current scanning direction of the turntable, the radial distance detected by the radar, and the pitch angle, and then the height information of the target, that is, the z-axis coordinate, can be obtained. The calculation formula is as follows: ; Among them, R is the distance between the target and the origin (when approximately considering that the radar, visible light, and infrared camera modules are at the same origin, the distance between the target and the radar), is the pitch angle of the target relative to the camera; The x-axis and y-axis coordinates of the target are calculated from the distance, azimuth, and elevation angle of the target relative to the radar. The calculation formulas are as follows: ; ; In the formula, , , respectively represent the three-dimensional coordinates of the target, is the azimuth angle of the target relative to the radar.

[0026] The wavelet transform fusion method extracts features and fuses infrared images and visible light images to enhance the quality and accuracy of the fused image. Specifically, the edge computing terminal processes the image through the following steps for wavelet transform: ; ; Among them, and respectively represent the wavelet coefficients of the visible light and infrared images, is the wavelet basis function, represents the spatial coordinates of the image, that is, the horizontal and vertical positions in the image, m represents the index variable, represents the scale of the wavelet transform, that is, the number of layers of wavelet decomposition, represents the translation parameter in the wavelet transform, and respectively represent the visible light and infrared images.

[0027] Perform weighted fusion on the wavelet coefficients of the visible light and infrared images: ; In the formula, and are the fusion weights of the wavelet coefficients of the visible light and infrared images respectively; represents the fused wavelet coefficient.

[0028] Reconstruct the fused image through inverse wavelet transform: ; Among them, is the fused image, represents the scaling function.

[0029] Target recognition includes extracting high-frequency information based on high-frequency units through discrete cosine transform. The formula is as follows: ; In the formula, , represents the high - frequency spectrum, and are the height and width of the image respectively, and represent the position indices of the frequency components, \(x\) i,j represents the pixel value of the image at the position (\( i, j ); represents the value of the orthogonal basis function at the position (\( i, j ); The low - frequency unit uses a multi - scale convolutional kernel to capture the low - frequency structure and background information of the sea - surface target. The dilation rate and the size of the convolutional kernel satisfy the following constraints: ; In the formula, is the receptive field size.

[0030] The target recognition outputs the category, bounding - box position, and confidence of the target. It uses a CIoU - based loss function to optimize the fitting of the bounding box, and its formula is as follows: ; In the formula, represents the square of the Euclidean distance between the center points of the predicted box and the ground - truth box, represents the predicted box, represents the center box, is the length of the diagonal of the minimum enclosing rectangle of the two, is the aspect - ratio consistency term, α is the balance parameter.

[0031] Optionally, the matching and association of multiple targets through the Hungarian algorithm include constructing a cost matrix that combines the Euclidean distance between the comprehensive target positions and the multi - modal feature similarity. Its formula is: ; In the formula, and represent the multi - modal feature vectors of the trajectory and the target respectively, and are the environment - adaptive weights; The Hungarian algorithm takes the cost matrix as input and determines the optimal matching scheme between the trajectory and the target by minimizing the total cost. Its optimization objective is: ; In the formula, is the binary matching matrix. If the target matches the trajectory , then , otherwise it is 0.

[0032] The Hungarian algorithm is used to solve the multi-target trajectory management problem. This algorithm optimizes the matching process between the object detection results and the historical trajectories to ensure the dynamic tracking of objects and the continuity of trajectories.

[0033] Specifically, the Hungarian algorithm constructs a cost matrix , and each element of the matrix represents the matching cost between the object detected in the current frame and the tracking trajectory in the previous frame . The matching cost can be calculated by the Euclidean distance between the object positions, and the formula is: ; In the formula, and respectively represent the center coordinates of the trajectory in the previous frame and the object detected in the current frame .

[0034] In practical applications, the matching results are used to update the status information of each object, including position, speed, etc.; for the un-matched objects, the system initializes new trajectories; for the trajectories that have not been matched for a long time, the system determines them as lost and removes them from the tracking list. Through the Hungarian algorithm, this method can effectively solve problems such as occlusion and drift in multi-object tracking in a complex sea surface environment, ensure the continuity and accuracy of object trajectories, and provide strong support for object management in a dynamic environment.

[0035] Optionally, the light correction threshold includes a visible light compensation enabling threshold and an infrared compensation enabling threshold. The visible light compensation enabling threshold is calculated based on the following formula: ; In the formula, represents the angle between the sunlight and the normal of the side of the object facing the camera, I a represents the ambient light intensity, I s represents the direct sunlight intensity. When the above formula is satisfied, visible light compensation is enabled; The ambient light intensity can be directly measured using an ambient light sensor, and the direct sunlight intensity can be directly measured using a photometer.

[0036] When identifying multiple objects, as a simplified solution, the influence of the sun on a single object can also be not considered, and the reading of the photometer can be directly used as the visible light compensation enabling threshold to determine whether to apply light information correction.

[0037] As another feasible solution, since the distances between the sun and the camera, and the sun and the target are extremely far compared to the distance between the camera and the target, with the sun angle and the camera angle fixed, the sides of multiple targets facing the camera can be simplified as a single object for recognition.

[0038] The infrared compensation enabling threshold is calculated based on the following formula: ; In the formula, represents the theoretical temperature rise, I represents the solar radiation intensity received by the ground surface, represents the infrared image acquisition time interval, represents the material density, d represents the material thickness, represents the specific heat capacity of the material. When , infrared compensation is enabled; represents the noise standard deviation of the infrared temperature sensor.

[0039] When identifying sea surface targets, the hull parameters including the material density and the material thickness can be obtained through the known ship type information. In this application, only the above formula is proposed as a typical and feasible calculation method, without delving into the in-depth research on the hull projection area, reflection material, and hull material during the image recognition process.

[0040] Optionally, the visible light compensation model includes: Calculate the solar altitude angle and azimuth angle based on the following formula: ; In the formula, represents the solar altitude angle, represents the local latitude, represents the solar declination angle, and H represents the solar hour angle.

[0041] ; In the formula, represents the solar azimuth angle; Convert the solar azimuth angle and the solar altitude angle into a three-dimensional vector based on the following formula: ; In the formula, is the solar direction vector in the world coordinate system.

[0042] Convert the solar direction vector in the world coordinate system to the coordinate system of this system through the external parameters of the camera: ; In the formula, R is the rotation matrix, representing the rotation relationship from the world coordinate system to the coordinate system of this system, Tis the translation vector, representing the translation relationship from the world coordinate system to the coordinate system of this system; Taking P o represents the three-dimensional coordinates of the target, N o represents the normal vector of the side of the object facing the camera. The angle between the sunlight and the normal of the side of the object facing the camera is calculated according to the following formula: ; In the formula, represents the angle between the sunlight and the normal of the side of the object facing the camera; Based on the target position and the normal vector of the side of the object facing the camera, calculate the angle between the sunlight and the normal of the side of the object facing the camera, calculate the target theoretical illumination intensity according to the sunlight intensity calibrated by the photometer, and use the target theoretical illumination intensity as the visible light supplementary feature; The target theoretical illumination intensity is calculated according to the following formula: ; In the formula, represents the target theoretical illumination intensity; The infrared compensation model includes: Taking the theoretical temperature rise as the infrared supplementary feature, the calculation method of the theoretical temperature rise has been described above; In the light information correction step, combined with the Hungarian algorithm to construct a cost matrix for multi-target matching and association. The constructed cost matrix is: ; In the formula, and respectively represent the visible light supplementary features of the trajectory and the target , and respectively represent the infrared supplementary features of the trajectory and the target .

[0043] Example 2 Compared with Example 1, the difference in this example is that the visible light compensation model and the infrared compensation model include calculating visible light image correction and infrared temperature correction. The visible light image correction coefficient is calculated based on the following formula: ; The infrared temperature correction coefficient is calculated based on the following formula: ; In the formula, represents the ambient temperature.

[0044] Calculate the visible light credibility factor and the infrared credibility factor based on the visible light attenuation and the infrared attenuation. The visible light credibility factor C v is calculated based on the following formula: ; In the formula, represents the visible light attenuation coefficient; the visible light attenuation coefficient is calculated based on the following formula: ; In the formula, represents the Rayleigh scattering received by the visible light, represents the Mie scattering received by the visible light, represents the gas absorption coefficient of the visible light.

[0045] The infrared credibility factor is calculated based on the following formula: ; In the formula, represents the infrared attenuation coefficient; the infrared attenuation coefficient is calculated based on the following formula: ; In the formula, represents the Mie scattering received by the infrared, represents the gas absorption coefficient of the infrared light.

[0046] Furthermore, calculate the visible light weight and the infrared weight as the fusion weights of the wavelet coefficients of the visible light and the infrared images; The weight calculation includes: ； ； In the optical information correction step, combined with the Hungarian algorithm, construct a cost matrix for multi-object matching and association, which means using the visible light weight and the infrared weight as the fusion weights of the wavelet coefficients of the visible light and the infrared images.

[0047] Embodiment 3 As a feasible implementation, in this embodiment, when the optical information is not corrected, the fusion weights of the wavelet coefficients of the visible light and the infrared images are controlled by directly obtaining the illumination intensity with a photometer.

[0048] At night and on cloudy days with low illumination, is assigned a lower weight, and is assigned a higher weight, as shown in Table 1: Table 1 Comparison table of image fusion coefficients and photometer readings

[0049] Environment Adaptive Weight and The calculation method is: ; In the formula, It represents the ratio of radar target signal strength to background noise strength. represents the attenuation coefficient of the radar signal, Indicates the distance between the target and the radar.

[0050] ; represents the standard deviation of brightness in the local area around pixel k, represents the average brightness of the local area around pixel k, To prevent the distribution from being a constant of zero, N is the number of pixels in the visible light image, represents the target temperature gradient, Indicates the maximum value of the temperature gradient in the infrared image.

[0051] Example 4 like Figure 2 - Figure 3 As shown, the present application also provides a sea surface target tracking system based on infrared and visible light image fusion, including an infrared camera 1, a visible light camera 2, a fixing bracket 3, a connecting bracket 4, an edge computing module 5, a power module 6, a communication module 7 and a data center 8; The infrared camera 1 and the visible light camera 2 are used to collect image data of sea surface targets in real time. The infrared camera 1 can fully capture the target thermal radiation characteristics at night, low light or bad weather, and the visible light camera 2 can collect high-definition detail characteristics during the day or under high light conditions. The two are installed on the connecting bracket 4, and the relative angle can be accurately adjusted by the electric adjustment mechanism. The horizontal direction can be adjusted from 0° to 360°, and the vertical direction can be adjusted within ±45°, with an accuracy error within 0.1° to 0.5°. The viewing angle is dynamically adapted according to the target, and is usually fixed on the upper deck of the ship or the top of the observation tower. With the help of the fixed bracket 3 made of aluminum alloy, it has the characteristics of high strength, light weight, corrosion resistance and vibration suppression, and can work stably in harsh marine environments.

[0052] The fixed bracket 3 is made of aluminum alloy material, which has the characteristics of high strength, light weight and corrosion resistance, and is suitable for the wet and high salt fog environment at sea. The fixed bracket 3 adopts a vibration suppression design, which can effectively reduce the interference of equipment shaking caused by waves or wind on the camera image collection, ensuring the stability of target capture and the accuracy of data collection.

[0053] The edge computing module 5 includes high-performance processing, storage, and input / output interfaces. It is placed in the ship control center or equipment cabin, preprocesses image data, realizes target recognition, tracking, and multi-modal fusion by combining deep learning algorithms, stores relevant information, and interacts with the communication and data center 8.

[0054] The power supply module 6 consists of a high-capacity lithium battery pack and a solar panel. The conversion efficiency of the solar panel reaches 20%, and the charge and discharge cycle times of the lithium battery pack are not less than 3000 times. It is installed near the ship power area or deck and has functions of power monitoring and energy-saving mode switching.

[0055] The communication module 7 supports multiple protocols to realize two-way communication between the system and the external network in the ship monitoring room or remote command center and the data center 8. The data center 8 is responsible for remote operation, monitoring, and storage management.

[0056] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A sea surface target tracking method based on infrared and visible light image fusion, characterized in that Including: Information acquisition: Synchronously acquire image data of sea surface targets through an infrared camera and a visible light camera; Image data processing: Denoise, enhance, and register the image data; The processed image data undergoes feature extraction and fusion through wavelet transform fusion to generate a comprehensive feature map, and a deep learning object detection model is used to perform object recognition on the comprehensive feature map; Object tracking: After object recognition is completed, the system enters the object tracking mode, calculates the target position based on the image data and radar data, performs multi-object matching and association through the Hungarian algorithm that combines multi-modal features, and combines the Kalman filter or particle filter algorithm to predict the next position of the target, update the target trajectory information, and the infrared camera and visible light camera perform dynamic angle adjustment according to the target's position and movement trajectory to ensure that the target is always within the field of view of the camera; Optical information correction: Judge whether to perform optical information correction based on the optical correction threshold. When performing optical information correction, based on the image data, construct optical information correction features based on the visible light compensation model and the infrared compensation model, and combine with the Hungarian algorithm during object tracking to construct a cost matrix for multi-object matching and association.

2. The method for tracking sea surface targets based on infrared and visible light image fusion according to claim 1, characterized in that: The calculation of the target position based on the image data and radar data includes taking the center of the horizontal positions of the radar, visible light, and infrared camera modules as the origin to establish a coordinate system, and the three-dimensional coordinates of the target are obtained based on the following formula: ； ； ； In the formula, , , respectively represent the three-dimensional coordinates of the target, R is the distance between the target and the radar, is the pitch angle of the target relative to the camera, is the azimuth angle of the target relative to the radar.

3. The sea surface target tracking method based on infrared and visible light image fusion according to claim 1, characterized in that: The feature extraction and fusion through wavelet transform fusion includes performing wavelet transform processing on the image according to the following steps: ； ； wherein, and represent the wavelet coefficients of visible light and infrared images respectively, is the wavelet basis function, represents the spatial coordinates of the image, m represents the index variable, represents the scale of the wavelet transform, represents the translation parameter in the wavelet transform, and represent visible light and infrared images respectively; Perform weighted fusion on the wavelet coefficients of the visible light and infrared images: ； In the formula, and are the fusion weights of the visible light and infrared image wavelet coefficients, respectively; represents the fused wavelet coefficients; Reconstruct the fused image through inverse wavelet transform: ； In the formula, is the fused image, represents the scaling function.

4. The method for tracking sea surface targets based on infrared and visible light image fusion according to claim 1, characterized in that: The object recognition includes extracting high-frequency information through discrete cosine transform based on high-frequency units, and the formula is as follows: ； In the formula, , represents the high-frequency spectrum, and are the height and width of the image respectively, and represent the position indices of the frequency components respectively, and x i,j represents the pixel value of the image at the position ( i,j ), represents the value of the orthogonal basis function at the position ( i,j ); The low-frequency unit uses multi-scale convolutional kernels to capture the low-frequency structure and background information of sea targets, and the dilation rate and the size of the convolutional kernel satisfy the following constraints: ； In the formula, is the receptive field size.

5. The method for tracking sea surface targets based on infrared and visible light image fusion according to claim 1, wherein: The object recognition outputs the category, bounding box position, and confidence of the target, and adopts a CIoU-based loss function to optimize the fitting of the bounding box, and its formula is as follows: ； In the formula, represents the square of the Euclidean distance between the center points of the predicted box and the ground truth box, represents the predicted box, represents the center box, is the length of the diagonal of the minimum bounding rectangle of the two, is the aspect ratio consistency term, α is the balance parameter.

6. The method for tracking sea surface targets based on infrared and visible light image fusion according to claim 1, wherein: The multi-object matching and association through the Hungarian algorithm includes constructing a cost matrix that combines the Euclidean distance between comprehensive target positions and the similarity of multi-modal features, and its formula is: ； In the formula, and respectively represent the multi-modal feature vectors of the trajectory and the target ; and are environment adaptive weights; The Hungarian algorithm takes the cost matrix as input and determines the optimal matching scheme between the trajectory and the target by minimizing the total cost, and its optimization objective is: ； In the formula, is a binary matching matrix. If the target matches the trajectory , then , otherwise it is 0.

7. The method for tracking sea surface targets based on infrared and visible light image fusion according to claim 1, wherein: The optical correction threshold includes a visible light compensation enabling threshold and an infrared compensation enabling threshold, and the visible light compensation enabling threshold is calculated based on the following formula: ； In the formula, represents the angle between the sunlight and the normal of the side of the object facing the camera, and I a represents the ambient light intensity, and I s represents the direct sunlight intensity. When the above formula is satisfied, visible light compensation is enabled; The infrared compensation enabling threshold is calculated based on the following formula: ； In the formula, represents the theoretical temperature rise, I represents the solar radiation intensity received by the ground surface, represents the infrared image acquisition time interval, represents the material density, d represents the material thickness, represents the specific heat capacity of the material. When is satisfied, infrared compensation is enabled; represents the noise standard deviation of the infrared temperature sensor.

8. The method for tracking sea surface targets based on infrared and visible image fusion according to claim 7, wherein: The visible light compensation model includes: Calculate the solar altitude angle and azimuth angle, calculate the angle between the sunlight and the normal of the side of the object facing the camera based on the target position and the normal vector of the side of the object facing the camera, calculate the theoretical illumination intensity of the target according to the sunlight intensity calibrated by the photometer, and use the theoretical illumination intensity of the target as the visible light supplementary feature; The infrared compensation model includes: Take the theoretical temperature rise as the infrared supplementary feature; In the optical information correction step, combine with the Hungarian algorithm to construct a cost matrix for multi-object matching and association, and the constructed cost matrix is: ； In the formula, and respectively represent the visible light supplementary features of the trajectory and the target . and respectively represent the infrared supplementary features of the trajectory and the target .

9. The method for tracking sea surface targets based on infrared and visible image fusion according to claim 7, wherein: The visible light compensation model and the infrared compensation model include calculating visible light image correction and infrared temperature correction, calculating a visible light credibility factor and an infrared credibility factor based on visible light attenuation and infrared attenuation, and further calculating a visible light weight and an infrared weight, which are used as the fusion weights of the wavelet coefficients of the visible light and infrared images; In the light information correction step, a cost matrix is constructed in combination with the Hungarian algorithm for multi-target matching and association, which means using the visible light weight and the infrared weight as the fusion weights of the wavelet coefficients of the visible light and infrared images.

10. A sea surface target tracking system based on infrared and visible light image fusion, characterized in that: It is carried out by using any one of the sea surface target tracking methods based on infrared and visible light image fusion as claimed in claims 1-9, including an infrared camera, a visible light camera, a fixed bracket, a connecting bracket, an edge computing module, a power supply module, a communication module, a data center and a radar; The infrared camera and the visible light camera are installed on the connecting bracket, and are used to collect the image data of the sea surface target in real time. The relative angle is adjusted by an electric adjustment mechanism. The horizontal angle adjustment range is 0°-360°, and the vertical angle adjustment range is ±45°. The precision error is 0.1°-0.5°. They are fixed on the upper deck of the ship or the top of the observation tower. An electric drive system is provided between the connecting bracket and the fixed bracket, which can adjust the angle of the camera according to a remote command; The fixed bracket is provided with a vibration suppression structure; The edge computing module includes a processor, a memory and an input / output interface, and is set in the ship control center or the equipment cabin, and is used for preprocessing the image data, realizing target recognition, tracking and multi-modal fusion in combination with a deep learning algorithm, storing relevant information and interacting with the communication and data center; The communication module is used for two-way communication between the system and the external network in the ship monitoring room or the remote command center and the data center; the data center is responsible for remote operation, monitoring and storage management.

Images