Ship track joint calibration method based on radar and vision technology

Abstract

The present application relates to the field of radar and vision joint calibration technology, and specifically to a ship track joint calibration method based on radar and vision technology. The method includes: acquiring radar point cloud data and image data; performing three-dimensional synchronous acquisition of multiple groups of point cloud image data pairs at each moment; performing cross-correlation analysis on the radar point cloud data and image data of the dynamic target, and combining the multiple modal function values ​​of the sequence formed after the fusion of the two data to determine the track fusion confidence of the dynamic target at each acquisition moment; determining the sea condition stability index of the dynamic target at each acquisition moment by combining the smoothed mean value of the track fusion confidence of the dynamic target with the local periodic fluctuation index; selecting a calibration method based on the sea condition stability index to complete the calibration of the dynamic target. The present application avoids the delay problem of the traditional fusion model, circumvents the pollution of the main sensor degradation data, and ensures the continuity and anti-interference ability of the track calibration under complex sea conditions.

Description

Technical Field

[0001] The present application relates to the field of radar and vision joint calibration technology, and specifically to a ship track joint calibration method based on radar and vision technology. Background Art

[0002] With the surge in global maritime traffic density and the escalating demand for navigation safety, radar, while capable of all-weather detection in complex weather conditions, has low resolution and is susceptible to multipath interference. Visual technology, while capable of capturing rich texture and semantic information, is severely affected by environmental factors such as lighting and haze. With breakthroughs in deep learning and multi-source data fusion, joint calibration has become an inevitable choice for improving perception accuracy. In the joint calibration of ship tracks using radar and vision technologies, severe weather and complex sea conditions not only degrade the performance of individual sensors but also disrupt the spatiotemporal consistency and complementarity of radar and vision data, making reliable fusion difficult for the joint calibration system. For example, in strong wave conditions, radar generates a large number of false alarm point clouds due to surface clutter, while vision loses target texture features due to rain and fog interference. Differences in noise distribution can lead to erroneous tracks when the two data are aligned in time and space. Existing technologies mainly address the above problems through cross-modal consistency verification. However, existing adaptive algorithms are mostly offline training models, which have response delays when facing sudden environmental changes (such as instantaneous strong light or surges) and cannot meet the timing synchronization requirements of real-time joint calibration. There is an urgent need to develop environmental perception online learning and multi-source redundant fusion mechanisms. Summary of the Invention

[0003] In order to solve the technical problem that time synchronization is difficult to complete due to sudden environmental changes, this application provides a joint ship track calibration method based on radar and vision technology. The technical solution adopted is as follows:

[0004] This application proposes a joint ship track calibration method based on radar and vision technology, which includes the following steps:

[0005] Radar point cloud data and image data are acquired at each moment using radar and vision technology. Three-dimensional synchronization of radar point cloud data and image data is performed using a high-precision GPS module and atomic clock to obtain multiple sets of point cloud image data pairs at each moment. Target tracking algorithms are used to ensure one-to-one correspondence between point cloud image data pairs of dynamic targets at different moments.

[0006] For the dynamic target, the cross-correlation analysis of the point cloud data and image data at the preset time before each moment is performed to obtain the cross-correlation function peak value and time delay; the fusion sequence of the dynamic target is constructed based on the different point cloud image data of the dynamic target at the preset time before each moment; the empirical mode decomposition method is used to decompose the fusion sequence into several modal subsequences, and the track fusion confidence of the dynamic target at each acquisition moment is calculated based on the modal function value, cross-correlation function peak value and time delay of the modal subsequences;

[0007] The sea state stability index of the dynamic target at each acquisition moment is determined by combining the smoothed mean of the track fusion confidence of the dynamic target at the preset moment before each moment with the volatility index in the preset sliding window;

[0008] The calibration method is selected based on the comparison between the sea state stability index and the preset error to complete the calibration of the dynamic target.

[0009] In the above scheme, this application first uses cross-correlation analysis to quantify the spatiotemporal matching degree of the two types of sensor data, combines empirical mode decomposition to distinguish transient interference from stable features, and improves the consistency of radar and visual data under complex sea conditions through spatiotemporal alignment and interference separation, thereby reducing the target false detection rate; then, to address the interference problem of the mirror results caused by surge mutations and instantaneous strong light noise, an exponentially weighted moving average filter is used, combined with a sliding window standard deviation analysis to comprehensively analyze long-term stability and local mismatch risks, eliminate the impact of transient interference such as rain, fog, and surges on the confidence level, and identify the track mismatch risk caused by sensor desynchronization or environmental mutations; then, the sea condition stability index is used to dynamically quantify the system reliability, quickly identify extreme environmental interference (such as strong light and surges), avoid the delay problem of the traditional fusion model, directly switch to the inertial navigation system, avoid the pollution of the main sensor degradation data, and ensure the continuity and anti-interference capability of the track calibration under complex sea conditions.

[0010] In one embodiment, the steps of performing three-dimensional synchronization of radar point cloud data and image data based on a high-precision GPS module and an atomic clock to obtain multiple sets of point cloud image data pairs at each moment are as follows:

[0011] The radar point cloud data is clustered to obtain its centroid and three-dimensional coordinates;

[0012] Use image data as neural network input, obtain the minimum bounding rectangle of each dynamic target, and obtain the three-dimensional coordinates of its center point;

[0013] Based on the high-precision GPS module and atomic clock, the three-dimensional coordinates of the radar point cloud data and image data are mapped to obtain the synchronized radar and visual three-dimensional coordinate data, and the one-to-one matching of the radar point cloud data and image data is completed. The matched pair of radar point cloud data and image data is recorded as a point cloud image data pair.

[0014] In one embodiment, the step of performing cross-correlation analysis on the point cloud data and the image data of the dynamic target at a preset time before each time to obtain the cross-correlation function peak value and time delay is:

[0015] For each moment, a preset number of moments before it are taken as the target time period. The radar point cloud data of each group of point cloud image data points in the target time period and the image data in the target time period are cross-correlated and analyzed to obtain the cross-correlation function peak and time delay. The cross-correlation analysis presets the time window and the maximum lag compensation.

[0016] In one embodiment, the step of forming a fusion sequence of the dynamic target based on different data of the dynamic target at a preset time before each time is:

[0017] The mean of the corresponding data points of the radar and vision in each quadrant of the coordinate system in the point cloud image data at each moment is taken as the fused data point; for each moment, the fused data points of different dynamic targets in the target time period are sorted in time sequence, where the three-dimensional data of each fused data point are respectively used as the three values ​​of the sequence to form a fusion sequence of the dynamic target.

[0018] In one embodiment, the modal function value is the cumulative sum of the absolute values ​​of all sequence values ​​in each modal subsequence.

[0019] In one embodiment, the track fusion confidence is positively correlated with the negative exponential mapping of the modal function and the peak value of the cross-correlation function, and negatively correlated with the delay.

[0020] In one embodiment, the sea state stability index is positively correlated with the mean of the track fusion confidence, and negatively correlated with the mean of the volatility index of all sliding windows in the target time period.

[0021] In one embodiment, the step of obtaining the mean value of the track fusion confidence is:

[0022] The track fusion confidence of all dynamic targets in the target time period before each moment is taken as input, and an exponentially weighted moving average filter is used to output the smoothed track fusion confidence and calculate its mean.

[0023] In one embodiment, the steps of obtaining the volatility index are:

[0024] Calculate the standard deviation of all track fusion confidences in the sliding window as the volatility indicator of the sliding window;

[0025] Each sliding window contains a preset number of time windows.

[0026] In one embodiment, the step of selecting a calibration method based on the comparison between the sea state stability index and the preset error is:

[0027] For dynamic targets, the sea state stability index calculated at all acquisition moments is normalized. When the normalized sea state stability index is continuously lower than the error threshold within the preset time, the use of radar and visual data is immediately stopped, and the inertial navigation system is switched to for route calibration; if it is not continuously lower than the error threshold within the preset time, joint calibration through radar and vision is continued.

[0028] The beneficial effects of this application are:

[0029] This application first uses cross-correlation analysis to quantify the spatiotemporal matching of two types of sensor data, combines empirical mode decomposition to distinguish transient interference from stable features, and improves the consistency of radar and visual data under complex sea conditions through spatiotemporal alignment and interference separation, thereby reducing the target false detection rate. Then, to address the interference problem of surface mirror results caused by noise from sudden surges and instantaneous strong light, an exponentially weighted moving average filter is used, combined with sliding window standard deviation analysis to comprehensively analyze long-term stability and local mismatch risks, eliminate the impact of transient interference such as rain, fog, and surges on confidence, and identify the risk of track mismatch caused by sensor desynchronization or sudden environmental changes. Then, the sea condition stability index is used to dynamically quantify system reliability, quickly identify extreme environmental interference (such as strong light and surges), avoid the delay problem of traditional fusion models, directly switch to the inertial navigation system, avoid pollution of degraded data of the main sensor, and ensure the continuity and anti-interference capability of track calibration under complex sea conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 This is a flow chart of a method for joint calibration of ship tracks based on radar and vision technology provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] To further illustrate the technical means and effectiveness of this application's implementation of the intended invention, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effectiveness of the combined ship track calibration method based on radar and vision technology proposed in this application. In the following description, references to different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0033] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0034] Example of ship track joint calibration method based on radar and vision technology:

[0035] The specific scheme of the ship track joint calibration method based on radar and vision technology provided by this application is described in detail below with reference to the accompanying drawings.

[0036] See also Figure 1 , which shows a flow chart of a ship track joint calibration method based on radar and vision technology provided by an embodiment of the present application, the method comprising the following steps:

[0037] Step S001: Acquire radar point cloud data and image data; perform three-dimensional synchronization on them to acquire multiple sets of point cloud image data pairs at each moment.

[0038] Millimeter-wave radars and high-resolution multispectral cameras are installed on the top of the ship's mast and on the top of the bulwarks on both sides of the hull to synchronously collect radar point cloud data and RGB-D depth image data. The millimeter-wave radar continuously acquires the distance of moving targets from the ship, the speed of dynamic targets, and the direction of the dynamic targets by transmitting frequency modulation. In addition, for dynamic targets, the millimeter-wave radar obtains their reflection intensity and treats each dynamic target as a point cloud data, thereby generating a dense point cloud to represent the dynamic environment around the ship. The camera uses wide dynamic imaging technology to capture high-resolution visual data on both sides of the ship, enhancing the target's three-dimensional spatial perception capabilities. In this embodiment, the acquisition frequency of the millimeter-wave radar and the high-resolution multispectral camera is 10Hz.

[0039] At each moment, each point cloud data point and its corresponding reflection intensity is used as the input of the clustering algorithm, and the output is a point cloud cluster. In this embodiment, the DBSCAN clustering algorithm is used, with a neighborhood radius of 0.5 and a minimum number of point clouds in a point cloud cluster of 10. Based on the point cloud cluster, its centroid and its 3D coordinates are determined.

[0040] Then, the RGB-D image is used as input, and the neural network model is used to output the three-dimensional coordinates of the center point of the dynamic target bounding box in the hull coordinate system; in this embodiment, the YOLO model is used, and the SGD optimizer is used in the YOLOv5 model training, and the momentum is set. The initial learning rate is 0.01; a cross-entropy loss function is used. The dynamic object bounding box is the minimum bounding rectangle of the dynamic object. The hull coordinate system is a coordinate system with the ship as the origin, describing the position of the ship.

[0041] To achieve temporal and spatial consistency, a high-precision GPS module and an atomic clock are integrated. Through timestamp alignment and geographic coordinate mapping, the time delay and posture deviation between sensors are eliminated. The elimination of time delay and posture deviation by high-precision GPS modules and atomic clocks is a technology well known to those in the field and will not be elaborated here. The final output is the three-dimensional coordinate data of the radar and vision that are synchronized in time and space.

[0042] It's important to note that the 3D coordinate data output by radar and vision systems, after spatiotemporal alignment and geographic coordinate mapping, have the same meaning. Both represent the absolute spatial positions of dynamic targets around the ship (such as other vessels or obstacles) in a unified geographic coordinate system (e.g., latitude, longitude, and altitude). This is because the system uses high-precision GPS and atomic clocks to convert both sensor data into the same global coordinate system. Point clouds and bounding box center points are matched one-to-one, achieving a consistent representation of target positions in the real world. Each point cloud and its corresponding bounding box center point are considered a point cloud image data pair, and a target tracking algorithm is used to establish a one-to-one correspondence between multiple sets of point cloud image data pairs at different times. This provides a reliable data foundation for a joint radar and vision-based ship track calibration method.

[0043] At this point, multiple sets of point cloud image data pairs are obtained at each moment.

[0044] Step S002 , performing cross-correlation analysis on the radar point cloud data and image data of the dynamic target, and determining the track fusion confidence of the dynamic target at each acquisition moment by combining multiple modal function values ​​of the sequence formed after the two data are fused.

[0045] Because radar and visual sensors are affected by multipath effects, rain and fog interference, and sea surface clutter in severe weather and complex sea conditions, the temporal and spatial registration deviation between radar point clouds and visual images increases significantly, the false alarm rate in track calibration increases, and the target tracking accuracy decreases. Especially in scenes with sudden surges or instantaneous strong light, traditional offline fusion models are difficult to ensure data consistency.

[0046] At each moment, there are multiple sets of point cloud image data pairs, and each point cloud image data pair also has different positions at different moments. A preset time period before each moment is used as the target time period. For dynamic targets, all point cloud image data pairs corresponding to the dynamic targets in the target time period are subjected to cross-correlation analysis to obtain the cross-correlation function peak and its corresponding delay. The peak value indicates the matching degree of the two types of sensor data in the spatiotemporal dimension, and the delay reflects the system clock synchronization error. In this embodiment, the target time period is 5 minutes, and the time window length and maximum lag compensation for the cross-correlation analysis are set to 0.5 seconds and 10 steps, respectively.

[0047] Then, for each moment, the mean of the radar and visual corresponding data points in each quadrant of the coordinate system in the point cloud image data is used as the fused data point. The dimension value on the x-axis of the fused data point is the mean of the dimension values ​​on the x-axis of the radar corresponding data point and the visual corresponding data point, and the same applies to the y-axis and z-axis.

[0048] For each moment, the fused data points in the target time period at that moment are sorted in time series, where the three-dimensional data of each fused data point are respectively used as three values ​​of the sequence, thereby forming a fusion sequence of each dynamic target; for example, the dimension value on the x-axis of the fused data point at the first moment is the first value of the fused sequence, the dimension value on the y-axis is the second value of the fused sequence, and the dimension value on the z-axis is the third value of the fused sequence.

[0049] The fused sequence is divided into several modal subsequences using the empirical mode decomposition method, and the sum of the absolute values ​​of all sequence values ​​in each modal subsequence is used as the modal function value. In this embodiment, the number of modal subsequences is 6.

[0050] After the fusion sequence is decomposed by the empirical mode decomposition method, the larger the modal function value, the higher its frequency, and the smaller the modal function value, the lower its frequency. After the modal function value is negatively mapped, it reflects the stability of the track signal.

[0051] The track fusion confidence at each acquisition moment is calculated based on the negative exponential mapping of the modal function, the peak of the cross-correlation function and its corresponding time delay.

[0052] The track fusion confidence is positively correlated with the negative exponential mapping of the modal function and the peak of the cross-correlation function, and negatively correlated with the delay.

[0053] It should be noted that positive correlation means that when one variable increases, the other variable also increases, and the two variables change in the same direction. When one variable changes from large to small or from small to large, the other variable also changes from large to small or from small to large; the specific relationship is determined by actual application and this application does not impose any special restrictions.

[0054] It should be noted that negative correlation means that when one variable increases, the other variable decreases accordingly, and the two variables change in opposite directions. When one variable changes from large to small or from small to large, the other variable also changes from small to large or from large to small. The specific relationship is determined by actual application and this application does not impose any special restrictions.

[0055] Preferably, in this embodiment, the expression of track fusion confidence is:

[0056] , represents the modal function value of the i-th modal subsequence, represents an exponential function with a natural constant as the base, represents the peak value of the cross-correlation function, Indicates delay, Represents the track fusion confidence at each moment.

[0057] The peak value of the cross-correlation function reflects the degree of temporal alignment between the radar and visual sensor data. Cross-correlation analysis shows that the peak value indicates the degree of temporal alignment of the ship's track observations from the two sensors at a specific time delay. A larger peak value indicates a higher consistency in the spatiotemporal characterization of the ship's track between the radar point cloud and the visual image. The time delay quantifies the time synchronization error between the two sensors. A smaller delay indicates higher system clock synchronization accuracy and better spatial consistency of target position in track calibration. The negative reflection of the modal function value reflects the strength of stable track features in the signal; a larger value indicates a more stable track signal.

[0058] The track fusion confidence is used to quantify the fusion confidence of radar and visual data in track calibration. The larger the value, the better the spatiotemporal alignment of radar and visual data and the more stable the track characteristics, and the higher the accuracy of the fusion track output by the joint calibration method.

[0059] At this point, the fused track confidence of the dynamic target at each moment is obtained.

[0060] Step S003: The sea state stability index of the dynamic target at each acquisition moment is determined by combining the smoothed mean value of the dynamic target's track fusion confidence and the local periodic fluctuation index.

[0061] Due to the increased temporal and spatial misalignment between radar point clouds and visual images in adverse weather and complex sea conditions, the time series data of the track fusion confidence A is subject to high-frequency noise interference and short-term fluctuations. This leads to increased false alarm rates and target tracking inaccuracies in the joint calibration system under sudden surges or transient strong light conditions. This application constructs a stability index based on the fluctuations of the track fusion confidence.

[0062] For each moment, all track fusion confidences of dynamic targets in the previous target time period are used as input and filtered using exponential weighted moving average (EWMA) to avoid transient noise interference, and the mean of the smoothed track fusion confidence is output. This value represents the long-term stability trend of the track fusion confidence. By suppressing the impact of transient interference such as rain, fog, and surge on the track fusion confidence, the system's ability to perceive the stable motion characteristics of the ship is enhanced. In this embodiment, the smoothing factor is 0.2.

[0063] A sliding window is then set up, and the standard deviation of all track fusion confidence scores within the sliding window is calculated as a volatility indicator for the sliding window. This quantifies the intensity of local fluctuations in track fusion confidence, thereby reflecting the risk of track feature mismatch between radar and visual data in the spatiotemporal dimensions due to environmental interference (such as sea clutter, rain and fog scattering) or sensor desynchronization. In this example, the sliding window size is 2 seconds, covering four time windows.

[0064] The sea state stability index is obtained based on the mean value of track fusion confidence and all volatility indicators in the target time period.

[0065] The sea state stability index is positively correlated with the mean value of the track fusion confidence, and negatively correlated with the mean value of all volatility indicators in the target time period.

[0066] Preferably, in this embodiment, the expression of the sea state stability index is:

[0067] , is the mean of the smoothed track fusion confidence, represents the mean of the volatility indicators of all sliding windows, Indicates adjustment parameters, which is used to prevent the denominator from being 0. Indicates the sea state stability index at each moment.

[0068] The mean of the smoothed track fusion confidence retains the long-term trend characteristics of the track fusion confidence (such as the stable navigation mode of the ship), while filtering out high-frequency jitter caused by transient environmental interference or sensor clock micro-jumps. The larger the value, the stronger the spatiotemporal alignment consistency of the radar and visual data in the long term, and the higher the credibility of the track calibration. The mean of the sliding window volatility index is the discrete degree of the track fusion confidence, which can identify the risk of track feature mismatch caused by sea surface clutter, sensor desynchronization or instantaneous strong light. The larger the value, the more dramatic the fluctuation of the matching between the radar and visual data in the spatiotemporal dimension, and the more significant the impact of multipath effects, rain and fog scattering or surge mutations on the system, reflecting the decreased stability of the track calibration due to environmental interference or sensor anomalies.

[0069] The sea state stability index dynamically characterizes the reliability of the radar and vision joint calibration system by fusing smoothing confidence and volatility. A higher value indicates that the track characteristics are well aligned in time and space, the environmental interference is controllable, and the calibration results are highly reliable. A lower value indicates that sensor weight adjustment or redundant data fusion mechanism needs to be triggered to deal with the problem of perception degradation in complex sea conditions.

[0070] At this point, the sea state stability index is obtained.

[0071] Step S004: Select a calibration method based on the sea stability index to complete the calibration of the dynamic target.

[0072] In adverse weather and complex sea conditions, radar and visual sensors are affected by multipath, rain and fog, and sea clutter, leading to significant increases in the temporal and spatial alignment errors between radar point clouds and visual images. This increases false alarm rates during track calibration and reduces target tracking accuracy. Traditional offline fusion models, lacking real-time response capabilities, struggle to ensure data temporal and spatial consistency, resulting in response delays and data mismatches in the joint calibration system, severely limiting the reliability of ship track perception.

[0073] Therefore, first, the radar and visual data points are matched one by one for calibration in step S001. When the sea stability index does not meet the conditions, the route calibration is performed through the inertial navigation system (INS).

[0074] In order to filter out occasional noise, for dynamic targets, the sea state stability index calculated at all acquisition moments is normalized. When the normalized sea state stability index is continuously lower than the error threshold within the preset time, the data of the main sensor (radar or vision) is immediately stopped, and the inertial navigation system is switched to perform route calibration. The inertial navigation system calculates the position through the gyroscope and accelerometer, and provides positioning when the satellite signal is lost, ensuring the continuity and robustness of the track calibration under complex sea conditions, thereby completing the calibration of dynamic targets. The inertial navigation system is a technology well known to people in this field and will not be elaborated on here. In this embodiment, the error threshold is 0.4 and the preset time is 5s.

[0075] By dynamically monitoring the sea stability index and implementing hard switching rather than complex fusion, the system avoids the impact of degraded primary sensor data on the system, ensuring the continuity and robustness of track calibration in extreme environments. This significantly improves the temporal and spatial registration accuracy and anti-interference capabilities of the ship track joint calibration system, effectively resolving the data mismatch and response delay issues caused by sudden environmental changes in existing technologies.

[0076] It should be noted that the above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

[0077] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A ship track joint calibration method based on radar and vision technology is characterized by: The method comprises the following steps: Radar point cloud data and image data are acquired at each moment using radar and vision technology. Three-dimensional synchronization of radar point cloud data and image data is performed using a high-precision GPS module and atomic clock to obtain multiple sets of point cloud image data pairs at each moment. Target tracking algorithms are used to ensure one-to-one correspondence between point cloud image data pairs of dynamic targets at different moments. For the dynamic target, the cross-correlation analysis of the point cloud data and image data at the preset time before each moment is performed to obtain the cross-correlation function peak value and time delay; the fusion sequence of the dynamic target is constructed based on the different point cloud image data of the dynamic target at the preset time before each moment; the empirical mode decomposition method is used to decompose the fusion sequence into several modal subsequences, and the track fusion confidence of the dynamic target at each acquisition moment is calculated based on the modal function value, cross-correlation function peak value and time delay of the modal subsequences; The sea state stability index of the dynamic target at each acquisition moment is determined by combining the smoothed mean of the track fusion confidence of the dynamic target at the preset moment before each moment with the volatility index in the preset sliding window; The calibration method is selected based on the comparison between the sea state stability index and the preset error to complete the calibration of the dynamic target.

2. The ship track joint calibration method based on radar and vision technology according to claim 1 is characterized in that: The steps of performing three-dimensional synchronization of radar point cloud data and image data based on a high-precision GPS module and an atomic clock to obtain multiple sets of point cloud image data pairs at each moment are as follows: The radar point cloud data is clustered to obtain its centroid and three-dimensional coordinates; Use image data as neural network input, obtain the minimum bounding rectangle of each dynamic target, and obtain the three-dimensional coordinates of its center point; Based on the high-precision GPS module and atomic clock, the three-dimensional coordinates of the radar point cloud data and image data are mapped to obtain the synchronized radar and visual three-dimensional coordinate data, and the one-to-one matching of the radar point cloud data and image data is completed. The matched pair of radar point cloud data and image data is recorded as a point cloud image data pair.

3. The ship track joint calibration method based on radar and vision technology according to claim 1 is characterized in that: The step of performing cross-correlation analysis on the point cloud data and image data of the dynamic target at a preset time before each time to obtain the cross-correlation function peak value and time delay is as follows: For each moment, a preset number of moments before it are taken as the target time period. The radar point cloud data of each group of point cloud image data points in the target time period and the image data in the target time period are cross-correlated and analyzed to obtain the cross-correlation function peak and time delay. The cross-correlation analysis presets the time window and the maximum lag compensation.

4. The ship track joint calibration method based on radar and vision technology according to claim 3 is characterized in that: The step of forming a fusion sequence of the dynamic target based on different data of the dynamic target at a preset time before each time is as follows: The mean of the corresponding data points of the radar and vision in each quadrant of the coordinate system in the point cloud image data at each moment is taken as the fused data point; for each moment, the fused data points of different dynamic targets in the target time period are sorted in time sequence, where the three-dimensional data of each fused data point are respectively used as the three values ​​of the sequence to form a fusion sequence of the dynamic target.

5. The ship track joint calibration method based on radar and vision technology according to claim 1 is characterized in that: The modal function value is the cumulative sum of the absolute values ​​of all sequence values ​​in each modal subsequence.

6. The ship track joint calibration method based on radar and vision technology according to claim 1 is characterized in that: The track fusion confidence is positively correlated with the negative exponential mapping of the modal function and the peak value of the cross-correlation function, and negatively correlated with the time delay.

7. The ship track joint calibration method based on radar and vision technology according to claim 3 is characterized in that: The sea state stability index is positively correlated with the mean value of the track fusion confidence, and negatively correlated with the mean value of the volatility index of all sliding windows in the target time period.

8. The ship track joint calibration method based on radar and vision technology according to claim 7 is characterized in that: The steps for obtaining the mean value of the track fusion confidence are as follows: The track fusion confidence of all dynamic targets in the target time period before each moment is taken as input, and an exponentially weighted moving average filter is used to output the smoothed track fusion confidence and calculate its mean.

9. The ship track joint calibration method based on radar and vision technology according to claim 7, characterized in that: The steps for obtaining the volatility index are as follows: Calculate the standard deviation of all track fusion confidences in the sliding window as the volatility indicator of the sliding window; Each sliding window contains a preset number of time windows.

10. The ship track joint calibration method based on radar and vision technology according to claim 1, characterized in that: The steps of selecting the calibration method based on the comparison between the sea state stability index and the preset error are as follows: For dynamic targets, the sea state stability index calculated at all acquisition moments is normalized. When the normalized sea state stability index is continuously lower than the error threshold within the preset time, the use of radar and visual data is immediately stopped, and the inertial navigation system is switched to for route calibration; if it is not continuously lower than the error threshold within the preset time, joint calibration through radar and vision is continued.

Images