A sea surface target tracking method based on visual guidance radar in high sea state

Abstract

The application discloses a sea surface target tracking method based on visual guidance radar under high sea conditions, and the method comprises the following steps: collecting millimeter wave radar point cloud, image, positioning, direction and speed data synchronously when the unmanned ship sails; using a YOLOv8 model optimized by slideloss weighting training and model distillation to detect the image target and obtain a target detection frame; preprocessing the radar point cloud; performing point cloud proposal and classification based on the image detection frame, respectively generating a clustering cluster through DBSCAN clustering, and fusing visual angle information of the proposed cluster; converting the clustering cluster to a world coordinate system, realizing double matching with a historical target through a Hungarian algorithm, initializing an unmatched target, and updating the state of a matched target and predicting the future position of the matched target. The application fills the short-distance perception blind area, significantly improves the precision and stability of sea surface target tracking under high sea conditions, and provides reliable protection for safe sailing of the unmanned ship.

Description

Technical Field

[0001] This invention relates to the field of sea surface target tracking technology, and specifically to a sea surface target tracking method based on visual guidance radar under high sea states. Background Technology

[0002] In recent years, the rapid development of artificial intelligence and autonomous driving technologies has driven the widespread application of unmanned vessels in the marine field. They are present in numerous key scenarios, including marine resource exploration, maritime search and rescue, border patrol, nearshore logistics transportation, and marine environmental monitoring, becoming core equipment for ensuring marine economic development and maintaining marine security. Surface target tracking, as a core supporting technology for the autonomous navigation of unmanned vessels, directly determines the reliability of collision avoidance and mission completion; its performance is crucial to navigation safety and operational efficiency.

[0003] Currently, the industry commonly employs a multi-sensor collaborative solution combining navigation radar, cameras, and AIS (Automatic Identification System) to achieve long-range perception and tracking of sea surface targets. However, this traditional solution has significant shortcomings: AIS relies on the target vessel actively transmitting signals, rendering it completely ineffective against close-range targets such as small boats, buoys, and underwater obstacles without AIS equipment; while navigation radar has all-weather capabilities, in high sea states, wave clutter severely interferes with the signal, leading to decreased target resolution and difficulty in accurately distinguishing real targets from environmental noise, and insufficient detection accuracy for small close-range targets; cameras are greatly affected by lighting conditions, fog and rain, and ship turbulence, resulting in blurred and distorted images, further reducing the stability of target detection.

[0004] More importantly, traditional solutions suffer from large-scale perception blind spots in the vicinity of unmanned vessels, posing a significant risk in scenarios such as ports, narrow waterways, and near-shore operations, and easily leading to collisions with nearby targets. Furthermore, high sea states and wave turbulence cause drastic changes in sensor attitude, significantly increasing data noise and causing violent fluctuations in target position and velocity information. Traditional tracking algorithms lack effective anti-interference mechanisms, making it difficult to achieve continuous and stable target tracking and failing to provide reliable decision-making support for unmanned vessels.

[0005] Although existing technologies have attempted to optimize multi-sensor data fusion strategies, most solutions have failed to fully leverage the complementarity of vision and radar, resulting in issues such as data synchronization lag and simplistic fusion logic. These solutions have failed to fundamentally address the core pain points of blind spots in close-range perception and insufficient tracking accuracy under high sea states, necessitating a more efficient and robust method for tracking sea surface targets. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a method for tracking sea surface targets under high sea states based on visual guidance radar. By constructing a tracking system using millimeter-wave radar and a camera, a set of visual guidance radar-based sea surface target tracking algorithms is proposed, which solves the problem of perception blind spots at close range and further improves the accuracy and stability of sea surface target tracking under high sea states.

[0007] To achieve the above objectives, this invention discloses a method for tracking sea surface targets based on visual guidance radar under high sea states, comprising the following steps:

[0008] Step S1: The unmanned vessel collects sensor data during navigation, including millimeter-wave radar point cloud data, image data, positioning data, and orientation data;

[0009] Step S2: Use the trained YOLOv8 model to perform object detection on the image data to obtain image object detection boxes. ;

[0010] Step S3: Preprocess the millimeter-wave radar point cloud, including multi-frame fusion, pass-through filtering, density filtering, and point cloud motion / static semantic distinction, to obtain the preprocessed millimeter-wave radar point cloud;

[0011] Step S4: Utilize image target detection bounding boxes Candidate point cloud proposals are made from the preprocessed millimeter-wave radar point cloud to obtain proposed point clouds and unproposed point clouds;

[0012] Step S5: Perform target clustering on the proposed point cloud and the unproposed point cloud respectively to obtain the proposed point cloud cluster and the unproposed point cloud cluster. Merge the visual target angle information into the proposed point cloud cluster to obtain the proposed visual point cloud cluster.

[0013] Step S6: Combine the proposed visual point cloud clusters with the unproposed point cloud clusters using historical information to perform target tracking;

[0014] Step S7: Output the target information of the current frame and save the relevant status for sea surface target tracking in the next frame.

[0015] Furthermore, in step S1, the unmanned surface vessel collects sensor data during navigation, specifically by using a data acquisition card to transfer millimeter-wave radar point cloud data during the vessel's operation. Image data Location data Directional data Speed ​​data After time synchronization, the data is output to the unmanned vessel's main control platform, including point cloud data. middle For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upward; Radial velocity of the radar point cloud position relative to the radar; positioning data middle The positive direction of the axis is due east. The positive direction of the axis is due north. The origin of the coordinate system can be randomly selected as the origin, and the direction data... The heading angle is the angle in which the unmanned surface vessel (USV) is facing, with true north being 0 degrees and clockwise being the positive direction. After collecting the current data, the USV's main control platform places it on a surface of length [missing information]. Data queue In the middle, it is used to track the platform's processing of current and historical data; N is the millimeter-wave radar point cloud data. The number of point clouds.

[0016] Furthermore, step S2 utilizes the trained YOLOv8 model to perform object detection on the image data, obtaining image object detection boxes, specifically as follows:

[0017] Step S21: Process image data Perform the same standardization process as the training set data, including contrast normalization, chroma normalization, and luminance normalization, to obtain the standardized image data. ;

[0018] Step S22: Convert the standardized image data Input the data into the trained YOLOv8 model to obtain the model's detection results. ,in Here are the coordinates of the top-left vertex of the image target bounding box. Here are the coordinates of the bottom right vertex of the image target bounding box. Define target categories and specify appropriate target heights for each target category. ;

[0019] Step S23: Analyze the model detection results. Perform NMS suppression to obtain the object detection box in the image. , .

[0020] Furthermore, the training process of the trained YOLOv8 model specifically includes the following steps:

[0021] Step S221: Normalize the contrast, brightness, and chromaticity of the public dataset to retain only the scene differences between different data and retain the relevant normalization parameters for inference.

[0022] Step S222: Input the processed public dataset into the YOLOv8s model for training. The class loss function used for training is slide loss. Slide loss is weighted by the cross-entropy used for training the target class. The weight function is as follows:

[0023]

[0024] in, The average IoU calculated for all matched boxes and truth boxes. The IoU value calculated for a given box and the true box;

[0025] Step S223: Use mask generation distillation to perform model distillation on yolov8s to obtain the distilled yolov8 model;

[0026] Step S224: Convert the distilled yologv8 model into ONNX format, which will be used as the yolov8 model for the final algorithm.

[0027] Furthermore, step S3 specifically includes:

[0028] Step S31: Process millimeter-wave radar point clouds Perform a pass-through filter to remove fixed-point noise caused by power supply noise. The fixed-point region is... The radar point cloud after direct filtering is obtained. :

[0029] ;

[0030] Step S32: Filter the radar point cloud after pass-through filtering Density filtering is performed to remove water noise; the radius of the density region is... The density-filtered radar point cloud is obtained. :

[0031] ;

[0032] Step S33: Filter the density-filtered radar point cloud Based on point cloud velocity and unmanned surface vessel operating speed Perform point cloud velocity correction to obtain velocity-corrected point cloud. ;

[0033] Step S34: Correct the velocity point cloud obtained from k consecutive frames. Multi-frame accumulation is performed to further increase the point cloud density, resulting in a pre-processed millimeter-wave radar point cloud. .

[0034] Furthermore, step S33 specifically includes:

[0035] Step S331: Filter the density-filtered radar point cloud Doppler velocity in Compensated to the horizontal plane coordinate system, the velocity is obtained. :

[0036] ;

[0037] Step S332: Utilize the unmanned vessel's speed and operating speed in the current frame. Calculation in Projected velocity in the direction of each point :

[0038] ;

[0039] Step S333: Calculate the compensation speed at each point :

[0040] ;

[0041] Step S334: Correct the point cloud using the compensated velocity. , where N is the number of point clouds in the current packet;

[0042] in, For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upwards.

[0043] Furthermore, step S4 specifically includes:

[0044] Step S41: Detect the target bounding box in the image. in, To detect the x and y coordinates of the top-left corner vertex of the bounding box, The coordinates of the bottom right corner of the detection frame are given. Camera intrinsics are used. Convert to target angle in camera coordinate system Distance to target The intrinsic parameters include focal length. , light center Use MATLAB to calibrate intrinsic parameters for the target angle. Distance to target The specific formula is as follows:

[0045] ;

[0046] ;

[0047] in, Determine the camera mounting height.

[0048] Step S42: Set the target angle in the camera coordinate system Based on the external parameters of the camera and radar Target angle converted to radar coordinate system , Euler angles in the heading, pitch, and roll directions, respectively:

[0049] ;

[0050] Step S43: In the radar coordinate system, using the target angle of each target in the radar coordinate system and target distance Drawing radius is circle The point cloud is located in any circle Internally, this represents the target point cloud proposed in the first instance.

[0051] Step S44: In the first proposed target point cloud, select targets whose absolute velocity value is greater than the first threshold. The point cloud as the proposed point cloud Other point clouds are unproposed point clouds. .

[0052] Furthermore, step S5 specifically includes:

[0053] Step S51: Process the proposed point cloud Perform DBSCAN clustering to obtain the clustered proposal point cloud. The radius used for clustering. The minimum number of point clouds required to classify a cluster of point clouds is the proposed point cloud after clustering. The expression is:

[0054] ;

[0055] Step S52: Calculate the spatial coordinates and mean velocity of the clustered proposed point clouds according to their categories, and use the PCA algorithm to calculate the target orientation to obtain the proposed point cloud clusters. ,in To determine the number of categories for the proposed point cloud, A unit vector , representing the target orientation, the specific parameters related to the proposed point cloud clustering are:

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] in, The x-coordinate, y-coordinate, and radial velocity of the point cloud belonging to target k. Let k be the number of point clouds for target k. The x and y coordinates of all point clouds with target category c;

[0061] Step S53: Cluster the proposed point cloud clusters By merging visual target angle information and using the PCA algorithm to calculate target orientation, the proposed visual point cloud clusters are obtained. ,in,( (x) represents the target's x-coordinate, y-coordinate, and radial velocity in the radar coordinate system. The target angle in the radar coordinate system. , The target orientation estimated using millimeter-wave radar point clouds;

[0062] ;

[0063] Step S54: For unproposed point clouds Perform DBSCAN clustering according to steps S51 and S52, and calculate the spatial coordinates and average velocity to obtain unproposed point cloud clusters. , This indicates the number of categories of point clouds that were not proposed.

[0064] Furthermore, step S6 specifically includes:

[0065] Step S61: Cluster the proposed visual point cloud into clusters Compared with unproposed point cloud clusters Rotate to the world coordinate system using position and orientation information;

[0066] Step S62: Cluster the proposed world coordinate point cloud With the tracked target The matching process yields proposed point cloud clusters that match the tracking target. Unmatched proposal point cloud clusters The first matched tracking target The first unmatched tracking target ;

[0067] Step S63: Initialize the target point cloud cluster that was not matched in the first time for subsequent tracking of the target. The initialization includes initializing the target's absolute position, initial velocity, and Kalman related information, which includes the observation error matrix, state error matrix, and observation matrix.

[0068] Step S64: From the first unmatched tracking target The selection process involves screening for absolute radial velocities less than the second threshold. tracking target The target matching algorithm described in step S62 is used to obtain unproposed point cloud clusters that match the tracking target. The second matching tracking target The second unmatched tracking target ; matching proposed point cloud clusters with the tracking target and unproposed point cloud clusters matching the tracking target Merge into matching point cloud clusters The first matched tracking target will be... The second matching target Merge into matched tracking targets ;

[0069] Step S65: Utilize matching point cloud clusters Perform corresponding matching tracking targets The extended Kalman filter is used to update the state, and the target information after the state update is obtained.

[0070] Step S66: Use the updated target information to predict the state covariance and future state. Predictions, including:

[0071] ;

[0072] ;

[0073] in, Let H be the predicted future state of the target in the (j+1)th frame, and let H be the target observation matrix. This represents the target state estimate after state update via extended Kalman filtering in the j-th frame. Let E be the target state covariance matrix updated in frame j, where E is the identity matrix and K is the Kalman gain. Let J be the final observation matrix for the j-th frame. Let be the predicted state covariance matrix of the j-th frame;

[0074] Step S67: Repeat the number of unmatched tracking targets for the second time. Cumulative number of unmatched times Reaching the third threshold Then, delete the relevant information about the target.

[0075] Furthermore, step S61 specifically includes:

[0076] Step S611: Utilize direction information Construct rotation matrix :

[0077] ;

[0078] Step S612: Using the rotation matrix With location data Cluster the proposed visual point cloud Location information Transform to the world coordinate system to obtain coordinates in the world coordinate system. Construct the proposed world coordinate point cloud cluster. ;

[0079] Core formula for coordinate transformation: ;

[0080] in, This is the location information vector of the proposed visual point cloud clusters. This represents the x-axis coordinate of a single point cloud in the world coordinate system after transformation. R represents the y-coordinate of a single point cloud in the world coordinate system after transformation, and R is the rotation matrix. Location data for unmanned vessels; The mean radial velocity of a single point cloud. The target angle in the radar coordinate system. The target orientation estimated from the radar point cloud. The number of categories for the proposed point cloud;

[0081] Step S613: Using the rotation matrix With location data Cluster the unproposed point clouds The location information is converted to the world coordinate system according to step S612 to obtain the unproposed world coordinate point cloud cluster. .

[0082] Furthermore, step S62 clusters the proposed world coordinate point cloud into clusters. With the tracked target To perform the matching, specifically:

[0083] Step S621: Obtain the proposed world coordinate point cloud clusters respectively. coordinates With the tracked target coordinates ;

[0084] Step S622: Construct the proposed world coordinate point cloud cluster coordinates With the tracked target coordinates The relative distance matrix between them is used as the cost matrix;

[0085] Step S623: Calculate the world coordinate point cloud clusters using the Hungarian algorithm with the cost matrix. With the tracked target The matching relationships are used to identify point cloud clusters that meet the matching conditions, which are then called proposed point cloud clusters that match the tracking target. The target is called the first matched tracking target. Those that do not meet the conditions are respectively called unmatched proposal point cloud clusters. Tracked target that did not match the first time .

[0086] Furthermore, step S63, which initializes the target point cloud cluster that was not matched in the first instance, specifically involves:

[0087] Initialize target state information Target state covariance matrix Target observation noise covariance matrix Target observation matrix Process transfer noise covariance matrix Q, target transfer matrix The specific parameters are as follows:

[0088] ;

[0089] in, These represent the x and y coordinates in the world coordinate system, respectively. These represent the x-axis velocity and the y-axis velocity, respectively. These represent the x-axis acceleration and the y-axis acceleration, respectively.

[0090] Target transition matrix for:

[0091] ;

[0092] Where dt is the time from the previous frame of data to the current frame of data, in seconds;

[0093] Process noise covariance matrix for:

[0094] ;

[0095] Measurement noise covariance matrix for:

[0096] ；

[0097] Target observation matrix :

[0098] ;

[0099] Target state covariance matrix : , Represents a 3rd order identity matrix;

[0100] Furthermore, step S65 specifically includes:

[0101] Step S651: Utilize matching point cloud clusters Construct the observation matrix of the target position and direction Observation matrix of target orientation ;

[0102] ;

[0103] ;

[0104] Step S652: Merge the observation matrix with the target state observation matrix to form the final observation matrix. and perform Kalman gain Calculation:

[0105] ;

[0106] ;

[0107] ;

[0108] in, F is the predicted state covariance matrix; F is the target transition matrix. Let be the updated state covariance matrix of the (j-1)th frame. Let F be the transpose of the transition matrix, and Q be the process transfer noise covariance matrix. Let be the observation noise covariance matrix of the j-th frame;

[0109] Step S653: The observations of the target are as follows At this point, calculate the target azimuth angle state quantity. Target towards state quantity And calculate the observation compensation amount. ;

[0110] ;

[0111] ;

[0112] ;

[0113] in, The target azimuth angle state quantity. For the target orientation state quantity, These represent the target's y and x positions in the world coordinate system, respectively. These are the target's velocities along the y and x axes in the world coordinate system, respectively. These represent the x and y coordinates of the target transformed into the world coordinate system, respectively. To observe the compensation amount, For target observations;

[0114] Step S654: Calculate the target state information update:

[0115] ;

[0116] in, Let j be the updated target state. Let j be the observation input vector of the j-th frame. Let be the basic observation matrix for the j-th frame.

[0117] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0118] This invention effectively addresses the blind spot problem in close-range perception using millimeter-wave radar and cameras, significantly improving the accuracy and stability of surface target tracking in high sea states. It employs a YOLOv8 model combined with slideloss and model distillation to optimize image target detection and enhance target recognition capabilities in complex sea conditions. Multi-frame fusion, dual filtering, and velocity correction preprocessing of radar point clouds reduce clutter interference and improve data quality. Visually guided point cloud proposal, DBSCAN clustering, and extended Kalman filtering enable precise target matching and state updates, ensuring tracking continuity. This method provides reliable close-range target perception support for the safe navigation of unmanned vessels and is suitable for target tracking tasks in complex marine environments. Attached Figure Description

[0119] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0120] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0121] Combination Figure 1 This invention proposes a method for tracking sea surface targets based on visual guidance radar under high sea states, comprising the following steps:

[0122] Step S1: The unmanned vessel collects sensor data during navigation, including millimeter-wave radar point cloud data, image data, positioning data, and orientation data;

[0123] Step S2: Use the trained YOLOv8 model to perform object detection on the image data to obtain the image object detection box;

[0124] Step S3: Preprocess the millimeter-wave radar point cloud, including multi-frame fusion, pass-through filtering, density filtering, and point cloud motion-static semantic distinction, to obtain the preprocessed millimeter-wave radar point cloud;

[0125] Step S4: Use the image target detection bounding box to propose candidate point clouds in the preprocessed millimeter-wave radar point cloud, and obtain the proposed point cloud and the unproposed point cloud;

[0126] Step S5: Perform target clustering on the proposed point cloud and the unproposed point cloud respectively to obtain the proposed point cloud cluster and the unproposed point cloud cluster. Merge the visual target angle information into the proposed point cloud cluster to obtain the proposed visual point cloud cluster.

[0127] Step S6: Combine the proposed visual point cloud clusters with the unproposed point cloud clusters using historical information to perform target tracking;

[0128] Step S7: Output the target information of the current frame and save the relevant status for sea surface target tracking in the next frame.

[0129] In this embodiment, the unmanned surface vessel (USV) collects sensor data during its navigation in step S1, specifically by using a data acquisition card to transfer millimeter-wave radar point cloud data during the USV's operation. Image data Location data Directional data Speed ​​data After time synchronization, the data is output to the unmanned vessel's main control platform, including point cloud data. middle For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upward; Radial velocity of the radar point cloud position relative to the radar; positioning data middle The positive direction of the axis is due east. The positive direction of the axis is due north. The origin of the coordinate system can be randomly selected as the origin, and the direction data... The heading angle is the angle in which the unmanned surface vessel (USV) is facing, with true north being 0 degrees and clockwise being the positive direction. After collecting the current data, the USV's main control platform places it on a surface of length [missing information]. Data queue In the middle, it is used to track the platform's processing of current and historical data; N is the millimeter-wave radar point cloud data. The number of point clouds.

[0130] In this embodiment, step S2 uses the trained YOLOv8 model to perform object detection on the image data to obtain image object detection boxes, specifically as follows:

[0131] Step S21: Process image data Perform the same standardization process as the training set data, including contrast normalization, chroma normalization, and luminance normalization, to obtain the standardized image data. ;

[0132] Step S22: Convert the standardized image data Input the data into the trained YOLOv8 model to obtain the model's detection results. ,in Here are the coordinates of the top-left vertex of the image target bounding box. Here are the coordinates of the bottom right vertex of the image target bounding box. Define target categories and specify appropriate target heights for each target category. ;

[0133] Step S23: Analyze the model detection results. Perform NMS suppression to obtain the object detection box in the image. , .

[0134] In this embodiment, the training process of the trained YOLOv8 model specifically includes the following steps:

[0135] Step S221: Normalize the contrast, brightness, and chromaticity of the public dataset to retain only the scene differences between different data and retain the relevant normalization parameters for inference.

[0136] Step S222: Input the processed public dataset into the YOLOv8s model for training. The class loss function used for training is slide loss. Slide loss is weighted by the cross-entropy used for training the target class. The weight function is as follows:

[0137]

[0138] in, The average IoU calculated for all matched boxes and truth boxes. The IoU value calculated for a given box and the true box;

[0139] Step S223: Use mask generation distillation to perform model distillation on yolov8s to obtain the distilled yolov8 model;

[0140] Step S224: Convert the distilled yologv8 model into ONNX format, which will be used as the yolov8 model for the final algorithm.

[0141] In this embodiment, step S3 specifically includes:

[0142] Step S31: Process millimeter-wave radar point clouds Perform a pass-through filter to remove fixed-point noise caused by power supply noise. The fixed-point region is... The radar point cloud after direct filtering is obtained. :

[0143] ;

[0144] Step S32: Filter the radar point cloud after pass-through filtering Density filtering is performed to remove water noise; the radius of the density region is... The density-filtered radar point cloud is obtained. :

[0145] ;

[0146] Step S33: Filter the density-filtered radar point cloud Based on point cloud velocity and unmanned surface vessel operating speed Perform point cloud velocity correction to obtain velocity-corrected point cloud. ;

[0147] Step S34: Correct the velocity point cloud obtained from k consecutive frames. Multi-frame accumulation is performed to further increase the point cloud density, resulting in a pre-processed millimeter-wave radar point cloud. .

[0148] In this embodiment, step S33 specifically includes:

[0149] Step S331: Filter the density-filtered radar point cloud Doppler velocity in Compensated to the horizontal plane coordinate system, the velocity is obtained. :

[0150] ;

[0151] Step S332: Utilize the unmanned vessel's speed and operating speed in the current frame. Calculation in Projected velocity in the direction of each point :

[0152] ;

[0153] Step S333: Calculate the compensation speed at each point :

[0154] ;

[0155] Step S334: Correct the point cloud using the compensated velocity. , where N is the number of point clouds in the current packet;

[0156] in, For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upwards.

[0157] In this embodiment, step S4 specifically includes:

[0158] Step S41: Detect the target bounding box in the image. in, To detect the x and y coordinates of the top-left corner vertex of the bounding box, The coordinates of the bottom right corner of the detection frame are given. Camera intrinsics are used. Convert to target angle in camera coordinate system Distance to target The intrinsic parameters include focal length. , light center Use MATLAB to calibrate intrinsic parameters for the target angle. Distance to target The specific formula is as follows:

[0159] ;

[0160] ;

[0161] in, Determine the camera mounting height.

[0162] Step S42: Set the target angle in the camera coordinate system Based on the external parameters of the camera and radar Target angle converted to radar coordinate system , Euler angles in the heading, pitch, and roll directions, respectively:

[0163] ;

[0164] Step S43: In the radar coordinate system, using the target angle of each target in the radar coordinate system and target distance Drawing radius is circle The point cloud is located in any circle Internally, this represents the target point cloud proposed in the first instance.

[0165] Step S44: In the first proposed target point cloud, select targets whose absolute velocity value is greater than the first threshold. The point cloud as the proposed point cloud Other point clouds are unproposed point clouds. .

[0166] In this embodiment, step S5 specifically includes:

[0167] Step S51: Process the proposed point cloud Perform DBSCAN clustering to obtain the clustered proposal point cloud. The radius used for clustering. The minimum number of point clouds required to classify a cluster of point clouds is the proposed point cloud after clustering. The expression is:

[0168] ;

[0169] Step S52: Calculate the spatial coordinates and mean velocity of the clustered proposed point clouds according to their categories, and use the PCA algorithm to calculate the target orientation to obtain the proposed point cloud clusters. ,in To determine the number of categories for the proposed point cloud, A unit vector , representing the target orientation, the specific parameters related to the proposed point cloud clustering are:

[0170] ;

[0171] ;

[0172] ;

[0173] ;

[0174] in, The x-coordinate, y-coordinate, and radial velocity of the point cloud belonging to target k. Let k be the number of point clouds for target k. The x and y coordinates of all point clouds with target category c;

[0175] Step S53: Cluster the proposed point cloud clusters By merging visual target angle information and using the PCA algorithm to calculate target orientation, the proposed visual point cloud clusters are obtained. ,in, The target angle in the radar coordinate system. , The target orientation estimated using millimeter-wave radar point clouds;

[0176] ;

[0177] Step S54: For unproposed point clouds Perform DBSCAN clustering according to steps S51 and S52, and calculate the spatial coordinates and average velocity to obtain unproposed point cloud clusters. , This indicates the number of categories of point clouds that were not proposed.

[0178] In this embodiment, step S6 specifically includes:

[0179] Step S61: Cluster the proposed visual point cloud into clusters Compared with unproposed point cloud clusters Rotate to the world coordinate system using position and orientation information;

[0180] Step S62: Cluster the proposed world coordinate point cloud With the tracked target The matching process yields proposed point cloud clusters that match the tracking target. Unmatched proposal point cloud clusters The first matched tracking target The first unmatched tracking target ;

[0181] Step S63: Initialize the target point cloud cluster that was not matched in the first time for subsequent tracking of the target. The initialization includes initializing the target's absolute position, initial velocity, and Kalman related information. The target Kalman related information includes the observation error matrix, state error matrix, and observation matrix.

[0182] Step S64: From the first unmatched tracking target The selection process involves screening for absolute radial velocities less than the second threshold. tracking target The target matching algorithm in step S62 yields unproposed point cloud clusters that match the tracked target. The second matching tracking target The second unmatched tracking target ; matching proposed point cloud clusters with the tracking target and unproposed point cloud clusters matching the tracking target Merge into matching point cloud clusters The first matched tracking target will be... The second matching target Merge into matched tracking targets ;

[0183] Step S65: Utilize matching point cloud clusters Perform corresponding matching tracking targets The extended Kalman filter is used to update the state, and the target information after the state update is obtained.

[0184] Step S66: Use the updated target information to predict the state covariance and future state. Predictions, including:

[0185] ;

[0186] ;

[0187] in, Let H be the predicted future state of the target in the (j+1)th frame, and let H be the target observation matrix. This represents the target state estimate after state update via extended Kalman filtering in the j-th frame. Let E be the target state covariance matrix updated in frame j, where E is the identity matrix and K is the Kalman gain. Let J be the final observation matrix for the j-th frame. Let be the predicted state covariance matrix of the j-th frame;

[0188] Step S67: Repeat the number of unmatched tracking targets for the second time. Cumulative number of unmatched times Reaching the third threshold Then, delete the relevant information about the target.

[0189] In this embodiment, step S61 specifically includes:

[0190] Step S611: Utilize direction information Construct rotation matrix :

[0191] ;

[0192] Step S612: Using the rotation matrix With location data Cluster the proposed visual point cloud Location information Transform to the world coordinate system to obtain coordinates in the world coordinate system. Construct the proposed world coordinate point cloud cluster. ;

[0193] Core formula for coordinate transformation: ;

[0194] in, This is the location information vector of the proposed visual point cloud clusters. This represents the x-axis coordinate of a single point cloud in the world coordinate system after transformation. R represents the y-coordinate of a single point cloud in the world coordinate system after transformation, and R is the rotation matrix. Location data for unmanned vessels; The mean radial velocity of a single point cloud. The target angle in the radar coordinate system. The target orientation estimated from the radar point cloud. The number of categories for the proposed point cloud;

[0195] Step S613: Using the rotation matrix With location data Cluster the unproposed point clouds The location information is converted to the world coordinate system according to step S612 to obtain the unproposed world coordinate point cloud cluster. .

[0196] In this embodiment, step S62 clusters the proposed world coordinate point cloud. With the tracked target To perform the matching, specifically:

[0197] Step S621: Obtain the proposed world coordinate point cloud clusters respectively. coordinates With the tracked target coordinates ;

[0198] Step S622: Construct the proposed world coordinate point cloud cluster coordinates With the tracked target coordinates The relative distance matrix between them is used as the cost matrix;

[0199] Step S623: Calculate the world coordinate point cloud clusters using the Hungarian algorithm with the cost matrix. With the tracked target The matching relationships are used to identify point cloud clusters that meet the matching conditions, which are then called proposed point cloud clusters that match the tracking target. The target is called the first matched tracking target. Those that do not meet the conditions are respectively called unmatched proposal point cloud clusters. Tracked target that did not match the first time .

[0200] In this embodiment, step S63, which initializes the target point cloud cluster that was not matched in the first instance, specifically involves:

[0201] Initialize target state information Target state covariance matrix Target observation noise covariance matrix Target observation matrix Process transfer noise covariance matrix Q, target transfer matrix The specific parameters are as follows:

[0202] ;

[0203] in, These represent the x and y coordinates in the world coordinate system, respectively. These represent the x-axis velocity and the y-axis velocity, respectively. These represent the x-axis acceleration and the y-axis acceleration, respectively.

[0204] Target transition matrix for:

[0205] ;

[0206] Where dt is the time from the previous frame of data to the current frame of data, in seconds.

[0207] Process noise covariance matrix for:

[0208] ;

[0209] Measurement noise covariance matrix for:

[0210] ；

[0211] Target observation matrix :

[0212] ;

[0213] Target state covariance matrix : .

[0214] In this embodiment, step S65 specifically includes:

[0215] Step S651: Utilize matching point cloud clusters Construct the observation matrix of the target position and direction Observation matrix of target orientation ;

[0216] ;

[0217] ;

[0218] Step S652: Merge the observation matrix with the target state observation matrix to form the final observation matrix. and perform Kalman gain Calculation:

[0219] ;

[0220] ;

[0221] ;

[0222] in, F is the predicted state covariance matrix; F is the target transition matrix. Let be the updated state covariance matrix of the (j-1)th frame. Let F be the transpose of the transition matrix, and Q be the process transfer noise covariance matrix. Let be the observation noise covariance matrix of the j-th frame;

[0223] Step S653: The observations of the target are as follows At this point, calculate the target azimuth angle state quantity. Target towards state quantity And calculate the observation compensation amount. ;

[0224] ;

[0225] ;

[0226] ;

[0227] in, The target azimuth angle state quantity. For the target orientation state quantity, These represent the target's y and x positions in the world coordinate system, respectively. These are the target's velocities along the y and x axes in the world coordinate system, respectively. These represent the x and y coordinates of the target transformed into the world coordinate system, respectively. To observe the compensation amount, For target observations;

[0228] Step S654: Calculate the target state information update:

[0229] ;

[0230] in, Let j be the updated target state. Let j be the observation input vector of the j-th frame. Let be the basic observation matrix for the j-th frame.

[0231] In summary, this invention effectively addresses the blind spot problem in close-range perception using millimeter-wave radar and camera collaboration, significantly improving the accuracy and stability of surface target tracking in high sea states. The YOLOv8 model, combined with slide loss and model distillation, optimizes image target detection and enhances target recognition capabilities in complex sea states. Multi-frame fusion, dual filtering, and velocity correction preprocessing of radar point clouds reduce clutter interference and improve data quality. Visually guided point cloud proposal, DBSCAN clustering, and extended Kalman filtering enable accurate target matching and state updates, ensuring tracking continuity. This method provides reliable close-range target perception support for the safe navigation of unmanned vessels and is suitable for target tracking tasks in complex marine environments.

[0232] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for tracking sea surface targets based on visual guidance radar under high sea states, characterized in that, Includes the following steps: Step S1: The unmanned vessel collects sensor data during navigation, including millimeter-wave radar point cloud data, image data, positioning data, and orientation data; Step S2: Use the trained YOLOv8 model to perform object detection on the image data to obtain image object detection boxes; Step S3: Preprocess the millimeter-wave radar point cloud, including multi-frame fusion, pass-through filtering, density filtering, and point cloud motion / static semantic distinction, to obtain the preprocessed millimeter-wave radar point cloud; Step S4: Use the image target detection bounding box to propose candidate point clouds in the preprocessed millimeter-wave radar point cloud, and obtain the proposed point cloud and the unproposed point cloud; Step S5: Perform target clustering on the proposed point cloud and the unproposed point cloud respectively to obtain the proposed point cloud cluster and the unproposed point cloud cluster. Merge the visual target angle information into the proposed point cloud cluster to obtain the proposed visual point cloud cluster. Step S6: Combine the proposed visual point cloud clusters with the unproposed point cloud clusters using historical information to perform target tracking; Step S7: Output the target information of the current frame and save the relevant status for sea surface target tracking in the next frame.

2. The method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 1, characterized in that, In step S1, the unmanned surface vessel (USV) collects sensor data during its navigation, specifically by using a data acquisition card to transfer millimeter-wave radar point cloud data. Image data Location data Directional data Speed ​​data After time synchronization, the data is output to the unmanned vessel's main control platform, including point cloud data. middle For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upward; Radial velocity of the radar point cloud position relative to the radar; positioning data middle The positive direction of the axis is due east. The positive direction of the axis is due north. The origin of the coordinate system can be randomly selected as the origin, and the direction data... The heading angle is the angle in which the unmanned surface vessel (USV) is facing, with true north being 0 degrees and clockwise being the positive direction. After collecting the current data, the USV's main control platform places it on a surface of length [missing information]. Data queue In this context, N represents the number of millimeter-wave radar point cloud data points used by the tracking platform to process current and historical data.

3. The method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 1, characterized in that, Step S2 uses the trained YOLOv8 model to perform object detection on the image data to obtain image object detection boxes, specifically as follows: Step S21: Process image data Perform the same standardization process as the training set data, including contrast normalization, chroma normalization, and luminance normalization, to obtain the standardized image data. ; Step S22: Convert the standardized image data Input the data into the trained YOLOv8 model to obtain the model's detection results. ,in Here are the coordinates of the top-left vertex of the image target bounding box. Here are the coordinates of the bottom right vertex of the image target bounding box. Define target categories and specify appropriate target heights for each target category. ; Step S23: Analyze the model detection results. Perform NMS suppression to obtain the object detection box in the image. , .

4. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 3, characterized in that, The training process of the trained YOLOv8 model specifically includes the following steps: Step S221: Normalize the contrast, brightness, and chromaticity of the public dataset to retain only the scene differences between different data and retain the relevant normalization parameters for inference. Step S222: Input the processed public dataset into the YOLOv8s model for training. The class loss function used for training is slide loss. Slide loss is weighted by the cross-entropy used for training the target class. The weight function is as follows: in, The average IoU calculated for all matched boxes and truth boxes. The IoU value calculated for a given box and the true box; Step S223: Use mask generation distillation to perform model distillation on yolov8s to obtain the distilled yolov8 model; Step S224: Convert the distilled yolov8 model into ONNX format, which will be used as the yolov8 model for the final algorithm.

5. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 1, characterized in that, Step S3 specifically involves: Step S31: Process millimeter-wave radar point clouds Perform a pass-through filter to remove fixed-point noise caused by power supply noise. The fixed-point region is... The radar point cloud after direct filtering is obtained. : ； Step S32: Filter the radar point cloud after pass-through filtering Density filtering is performed to remove water noise; the radius of the density region is... The density-filtered radar point cloud is obtained. : ； Step S33: Filter the density-filtered radar point cloud Based on point cloud velocity and unmanned surface vessel operating speed Perform point cloud velocity correction to obtain velocity-corrected point cloud. ; Step S34: Correct the velocity point cloud obtained from k consecutive frames. Multi-frame accumulation is performed to further increase the point cloud density, resulting in a pre-processed millimeter-wave radar point cloud. .

6. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 5, characterized in that, Step S33 specifically involves: Step S331: Filter the density-filtered radar point cloud Doppler velocity in Compensated to the horizontal plane coordinate system, the velocity is obtained. : ； Step S332: Utilize the unmanned vessel's speed and operating speed in the current frame. Calculation in Projected velocity in the direction of each point : ； Step S333: Calculate the compensation speed at each point : ； Step S334: Correct the point cloud using the compensated velocity. , where N is the number of point clouds in the current packet; in, For radar coordinate system Coordinates, positive direction to the right; Radar coordinate system Coordinates, positive direction forward; For radar coordinate system Coordinates, positive direction upwards.

7. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 1, characterized in that, Step S4 specifically involves: Step S41: Detect the target bounding box in the image. in, To detect the x and y coordinates of the top-left corner vertex of the bounding box, To detect the horizontal and vertical coordinates of the lower right corner of the bounding box, camera intrinsics are used. Convert to target angle in camera coordinate system Distance to target The intrinsic parameters include focal length. , light center Use MATLAB to calibrate intrinsic parameters for the target angle. Distance to target The specific formula is as follows: ； ； in, Determine the camera installation height; Step S42: Set the target angle in the camera coordinate system Based on the external parameters of the camera and radar Target angle converted to radar coordinate system , Euler angles in the heading, pitch, and roll directions, respectively: ； Step S43: In the radar coordinate system, using the target angle of each target in the radar coordinate system and target distance Drawing radius is circle The point cloud lies in any circle Internally, this represents the target point cloud proposed in the first instance. Step S44: In the first proposed target point cloud, select targets whose absolute velocity value is greater than the first threshold. The point cloud as the proposed point cloud Other point clouds are unproposed point clouds. .

8. A method for tracking sea surface targets based on visual guidance radar in high sea states according to claim 7, characterized in that, Step S5 specifically involves: Step S51: Process the proposed point cloud Perform DBSCAN clustering to obtain the clustered proposal point cloud. The radius used for clustering. The minimum number of point clouds required to classify a cluster of point clouds is the proposed point cloud after clustering. The expression is: ； Step S52: Calculate the spatial coordinates and mean velocity of the clustered proposed point clouds according to their categories, and use the PCA algorithm to calculate the target orientation to obtain the proposed point cloud clusters. ,in To determine the number of categories for the proposed point cloud, A unit vector , representing the target orientation, the specific parameters related to the proposed point cloud clustering are: ； ; ; ; in, The x-coordinate, y-coordinate, and radial velocity of the point cloud belonging to target k. Let k be the number of point clouds for target k. The x and y coordinates of all point clouds with target category c; Step S53: Cluster the proposed point cloud clusters By merging visual target angle information and using the PCA algorithm to calculate target orientation, the proposed visual point cloud clusters are obtained. ,in, The target angle in the radar coordinate system. , The target orientation is estimated using millimeter-wave radar point clouds; where: ； Step S54: For unproposed point clouds Perform DBSCAN clustering according to steps S51 and S52, and calculate the spatial coordinates and average velocity to obtain unproposed point cloud clusters. , This indicates the number of categories of point clouds that were not proposed.

9. A method for tracking sea surface targets based on visual guidance radar in high sea states according to claim 8, characterized in that, Step S6 specifically involves: Step S61: Cluster the proposed visual point cloud into clusters Compared with unproposed point cloud clusters Rotate to the world coordinate system using position and orientation information; Step S62: Cluster the proposed world coordinate point cloud With the tracked target The matching process yields proposed point cloud clusters that match the tracking target. Unmatched proposal point cloud clusters The first matched tracking target The first unmatched tracking target ; Step S63: Initialize the target point cloud cluster that was not matched in the first time for subsequent tracking of the target. The initialization includes initializing the target's absolute position, initial velocity, and Kalman related information, which includes the observation error matrix, state error matrix, and observation matrix. Step S64: From the first unmatched tracking target The selection process involves screening for absolute radial velocities less than the second threshold. tracking target The target matching algorithm described in step S62 is used to obtain unproposed point cloud clusters that match the tracking target. The second matching tracking target The second unmatched tracking target ; Proposed point cloud clusters that match the tracking target and unproposed point cloud clusters matching the tracking target Merge into matching point cloud clusters The first matched tracking target will be... Tracking target matched for the second time Merge into matched tracking targets ; Step S65: Utilize matching point cloud clusters Perform corresponding matching tracking targets The extended Kalman filter is used to update the state, and the target information after the state update is obtained. Step S66: Use the updated target information to predict the state covariance and future state. Predictions, including: ； ； in, Let H be the predicted future state of the target in the (j+1)th frame, and let H be the target observation matrix. This represents the target state estimate after state update via extended Kalman filtering in the j-th frame. Let E be the target state covariance matrix updated in frame j, where E is the identity matrix and K is the Kalman gain. Let J be the final observation matrix for the j-th frame. Let be the predicted state covariance matrix of the j-th frame; Step S67: Repeat the number of unmatched tracking targets for the second time. Cumulative number of unmatched times Reaching the third threshold Then, delete the relevant information about the target.

10. A method for tracking sea surface targets under high sea states based on visual guidance radar according to claim 9, characterized in that, Step S61 specifically involves: Step S611: Utilize direction information Construct rotation matrix : ； Step S612: Using the rotation matrix With location data Cluster the proposed visual point cloud Location information Transform to the world coordinate system to obtain coordinates in the world coordinate system. Construct the proposed world coordinate point cloud cluster. ; Core formula for coordinate transformation: ; in, This is the location information vector of the proposed visual point cloud clusters. This represents the x-axis coordinate of a single point cloud in the world coordinate system after transformation. R represents the y-coordinate of a single point cloud in the world coordinate system after transformation, and R is the rotation matrix. Location data for unmanned vessels; The mean radial velocity of a single point cloud. The target angle in the radar coordinate system. The target orientation estimated from the radar point cloud. The number of categories for the proposed point cloud; Step S613: Using the rotation matrix With location data Cluster the unproposed point clouds The location information is converted to the world coordinate system according to step S612 to obtain the unproposed world coordinate point cloud cluster. .

11. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 9, characterized in that, Step S62 clusters the proposed world coordinate point cloud. With the tracked target To perform the matching, specifically: Step S621: Obtain the proposed world coordinate point cloud clusters respectively. coordinates With the tracked target coordinates ; Step S622: Construct the proposed world coordinate point cloud cluster coordinates With the tracked target coordinates The relative distance matrix between them is used as the cost matrix; Step S623: Calculate the world coordinate point cloud clusters using the Hungarian algorithm with the cost matrix. With the tracked target The matching relationships are used to identify point cloud clusters that meet the matching conditions, which are then called proposed point cloud clusters that match the tracking target. The target is called the first matched tracking target. Those that do not meet the conditions are respectively called unmatched proposal point cloud clusters. Tracked target that did not match the first time .

12. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 9, characterized in that, The specific steps of setting up the target initialization for the first unmatched target point cloud cluster are as follows: Initialize target state information Target state covariance matrix Target observation noise covariance matrix Target observation matrix Process transfer noise covariance matrix Q, target transfer matrix The specific parameters are as follows: ； in, These represent the x and y coordinates in the world coordinate system, respectively. These represent the x-axis velocity and the y-axis velocity, respectively. These represent the x-axis acceleration and the y-axis acceleration, respectively. Target transition matrix for: ； Where dt is the time from the previous frame of data to the current frame of data, in seconds; Process noise covariance matrix for: ； Measurement noise covariance matrix for: ； Target observation matrix : ； Target state covariance matrix : , This represents a 3rd order identity matrix.

13. A method for tracking sea surface targets based on visual guidance radar under high sea states according to claim 9, characterized in that, Step S65 specifically involves: Step S651: Utilize matching point cloud clusters Construct the observation matrix of the target position and direction Observation matrix of target orientation ; ； ； Step S652: Merge the observation matrix with the target state observation matrix to form the final observation matrix. and perform Kalman gain Calculation: ； ； ； in, F is the predicted state covariance matrix; F is the target transition matrix. Let be the updated state covariance matrix of the (j-1)th frame. Let F be the transpose of the transition matrix, and Q be the process transfer noise covariance matrix. Let be the observation noise covariance matrix of the j-th frame; Step S653: The observations of the target are as follows At this point, calculate the target azimuth angle state variable. Target towards state quantity And calculate the observation compensation amount. ; ； ； ; in, The target azimuth angle state quantity. For the target orientation state quantity, These represent the target's y and x positions in the world coordinate system, respectively. These are the target's velocities along the y and x axes in the world coordinate system, respectively. These represent the x and y coordinates of the target transformed into the world coordinate system, respectively. To observe the compensation amount, For target observations; Step S654: Calculate the target state information update: ； in, Let j be the updated target state. Let j be the observation input vector of the j-th frame. Let be the basic observation matrix for the j-th frame.

Images