A marine unmanned surface vehicle radar-camera topology bidirectional fusion detection method

Abstract

The offshore unmanned vehicle radar camera topological bidirectional fusion detection algorithm belongs to the technical field of radar camera fusion in multi-sensor fusion. Firstly, time registration of the sensor system is carried out, and optical images and radar detection points at the same time are obtained. After detecting the camera image by adopting YOLOV7-tiny, the center coordinates of the detection frame and the target category are output. According to the maximum detection distance of the camera, the radar points in the range are projected to the optical image through matrix conversion, so that the detection results of the sensor system are located in the pixel coordinate system, space registration of the system is realized, the projected radar points are subjected to PROSAC straight line fitting, and the radar fitting line is deflected according to the ship body attitude angle. The system error of the sensor and the position error loss of the YOLO algorithm are used to set a rough correlation wave gate to screen the radar points and reduce the calculation amount. Then, the topological fusion index is set to carry out radar and camera topological fusion. Finally, the unassociated radar points and the detection frame are subjected to bidirectional fusion.

Description

Technical Field

[0001] This invention relates to a two-way fusion detection method for radar and camera topology of unmanned surface vessels at sea, belonging to the field of radar and camera fusion technology in multi-sensor fusion. Background Technology

[0002] Radar-camera fusion technology can effectively improve the detection capabilities of unmanned surface vessels (USVs) in the surrounding sea areas. In the development of USV surface target detection technology, traditional methods typically use radar to guide the camera system to capture targets. This method relies on adjusting the camera system's angle according to the radar's azimuth to achieve target acquisition. Because it heavily relies on manual interpretation and lacks real-time performance, it is practical in environments with few targets and no obstructions, but cannot accurately detect multiple targets simultaneously. In recent years, IoU fusion methods between LiDAR and vehicle-mounted cameras have become increasingly common in the field of autonomous driving. Multiple fixed cameras mounted on the vehicle perform 360° shooting, achieving all-around fusion with radar. Vehicle-mounted LiDAR has high accuracy and can form radar point cloud images, but its detection range is only a few hundred meters. In maritime scenarios, USVs often have longer detection ranges and larger errors. Coupled with the influence of sea waves and the swaying of USVs, algorithm improvements are needed to achieve radar-camera fusion detection for USVs.

[0003] Therefore, it is urgent to design a two-way fusion detection algorithm for radar and camera topology of unmanned surface vessels (USVs) to solve the problem of effective fusion detection of radar and camera carried by USVs and to obtain information such as images, distance, category, and orientation of maritime targets. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the existing technology and provide a two-way fusion detection method for radar and camera topology of unmanned surface vessels (USVs). This method can effectively detect distant targets at sea under conditions of USV swaying, multiple target obstruction, and errors.

[0005] The present invention discloses a two-way fusion detection algorithm for radar and camera topology of unmanned surface vessels, characterized by the following steps:

[0006] Step 1) Time registration between radar and camera;

[0007] Step 2) Spatial registration of radar and camera: Perform external and internal calibration, obtain the geometric parameters of radar and camera, establish the spatial transformation relationship between them, find the correspondence between radar data and camera data, and realize spatial registration;

[0008] Step 3) PROSAC algorithm line fitting: The algorithm is used to construct a straight line model of the radar data projection points, and the obtained straight line model is adjusted by the attitude information of the unmanned surface vessel acquired synchronously.

[0009] Step 4) Coarse correlation gate establishment: Based on the range and orientation error of the sensor system and the loss error of the bounding box center position of the YOLO algorithm, a coarse correlation gate threshold is established to coarsely screen the radar points;

[0010] Step 5) Establishment of Topological Fusion Similarity Index: Establish a comprehensive fusion similarity index using side length similarity, polygon angle similarity, and line similarity between polygon center points;

[0011] Step 6) Matrix establishment and radar / camera matching: Establish a fusion matrix and match the radar projection points and camera detection box center points according to the threshold.

[0012] Step 7) Two-way fusion of unfused radar and camera data: Image clipping near unfused radar points is detected and fused; for unfused optical detection frames, the nearest neighbor radar point on the azimuth line is selected for fusion to achieve two-way fusion.

[0013] Step 1 uses the "interpolation / extrapolation method" for time registration, and specifically includes the following sub-steps:

[0014] Step 1.1: If the time difference between the interpolation point and the previous radar sampling is less than the sampling interval, extrapolate using the previous measurement data;

[0015] Step 1.2: If the time difference between the interpolation point and the radar sampling point used in the previous determination is greater than the sampling interval, extrapolate using the previous measurement data that is closest to the interpolation point;

[0016] Step 1.3: If the time difference between the interpolation point and the radar sampling time of the last judgment is equal to the sampling interval, keep the radar data at that moment unchanged;

[0017] Step 1.4: Process the radar and camera data according to the above steps to align them at the same time point and achieve time registration.

[0018] Step 2 specifically includes the following sub-steps:

[0019] Step 2.1: Perform external and internal calibration to obtain the geometric parameters of the radar and camera, establish the spatial transformation relationship between them, and then find the correspondence between radar data and camera data through feature point or feature region extraction and matching to achieve spatial registration.

[0020]

[0021] Formula (27) is the coordinate transformation matrix, (X cam ,Y cam Z cam (X) represents the coordinates in the photoelectric coordinate system. world ,Y world Z worldLet ) be the coordinates in the world coordinate system, R be the rotation matrix, T be the translation matrix, and f be the translation matrix. cam Let (x,y) be the photoelectric focal length, (x,y) be the coordinates in the image coordinate system, and (u,v) be the pixel coordinates. Set the radar coordinate system to the world coordinate system.

[0022] Step 3 specifically includes the following sub-steps:

[0023] Step 3.1: The set of N radar projection points is First, calculate the quality score of each point in the point set. The calculation method is as follows: traverse each point, calculate the square of its distance from all other points, and then calculate the inverse sum of these squared distances as the quality score of that point.

[0024] Projection point The mass fraction is defined as:

[0025]

[0026] Step 3.2: Assess the set based on the mass fraction Sort in descending order as follows: To ensure that high-quality points are selected as candidate points, the candidate point set is defined as follows: n = min(t, N), the initial number of candidate points is n = 1, and it increases with the number of iterations t until it reaches the value of the number of iterations t or The number of points N, in the t-th iteration, will use the first n sorted points as candidate points to generate the model and find the interior points, and select the first two points from them to calculate the equation of the line;

[0027] Step 3.3: Calculate the distance from all points to the current line, and determine which points are interior points based on the threshold υ. If the number of interior points found exceeds the current best result, update the best line and the set of interior points. Assuming the number of interior points is η, and P is the prior probability, then the number of iterations t is defined as:

[0028]

[0029] After straight-line fitting, straight-line deflection is performed based on the ship's attitude angle.

[0030] Step 4 specifically includes the following sub-steps:

[0031] Step 4.1: Center position loss threshold:

[0032] Based on the bounding box center position loss of the YOLO algorithm, a loss threshold λ is generated. loss :

[0033]

[0034] Where, λ coordThis represents the corresponding weight term, and λ is set. coord =1, s 2 All grid cells representing s×s need to be calculated, and B represents the number of box classes. This indicates whether the j-th box in the i-th grid should predict the target. If it should, then... otherwise λ loss As a center position loss threshold, it is used to form a coarse correlation gate to filter radar points;

[0035] Step 4.2: System Error Threshold

[0036] At time k, the track point spacing R between radar α and camera β for the same target (target i detected by radar, target j detected by camera) is... ij (k) can be derived as follows:

[0037]

[0038] Δr α ,Δr β For the positional error of the radar and camera, Δθ α ,Δθ β For the angle error of the radar and camera, the remaining values ​​are fixed values. According to equation (31), if R ij (k) Take the maximum value R max , Δθ α ,Δθ β ,Δr α ,Δr β The value should be:

[0039] Δθα=Δθ αmax Δθ β =Δθ βmax Δr α =Δr αmax Δr β =-Δr βmax (19)

[0040] Use the maximum value R of the trackpoint spacing max As a systematic error threshold, the above calculation is performed on all targets at time k, and the maximum value is taken as the systematic error threshold at that time.

[0041] Step 4.3: Generation of coarse correlation gates

[0042] The system error threshold is located in the radar coordinate system (X). world ,Y world Z world Under the given conditions, the center position loss threshold is located in the pixel coordinate system (u,v), therefore the system error threshold R is set to...max Transform to the pixel coordinate system using formula (27), and define the systematic error threshold in the pixel coordinate system as: R uv max ;

[0043] The coarse correlation gate size is defined as:

[0044] L max =s*(R) uv max +λ loss (20)

[0045] s is a scaling factor that can be adjusted according to actual conditions. If the coarse correlation gate is too large, it will increase unnecessary calculations; if the coarse correlation gate is too small, it will filter out the correct radar points. Set s = 2.

[0046] With coarse correlation gate L max Using the radius as the center point of each YOLO detection frame, a coarse correlation gate is generated. For each YOLO detection frame, only radar points within the gate are fused, while radar points outside the gate are excluded.

[0047] Step 5 specifically includes the following sub-steps:

[0048] Step 5.1: Side Length Similarity Index

[0049] An evaluation index is constructed based on the similarity of the lengths of the three sides of a triangle, defined as follows:

[0050]

[0051] The similarity of the side lengths at time t is represented by . Let i represent the i-th side of radar triangle α. Represents the i-th side of camera triangle β;

[0052] Step 5.2: Polygon Angular Similarity Index

[0053] The similarity of polygon angles is defined as:

[0054]

[0055] The polygon angular similarity at time t is represented by... This represents the i-th interior angle of the radar polygon α. Let n represent the i-th interior angle of the camera polygon β, and n1 be the number of interior angles of the polygon.

[0056] Step 5.3: Similarity index of center point connection lines

[0057] Connecting the center point of radar polygon α to the i-th vertex yields line L, which is the line connecting the center points of radar polygon α. α Connect the center point of the camera polygon β and the i-th vertex to obtain the line L connecting the center points of the camera polygon β. β The similarity of the lines connecting the center points is defined as:

[0058]

[0059] The similarity of the lines connecting the center points at time t. and For the i-th L α The included angle and length, and For the i-th L β The included angle and length;

[0060] Step 5.4: Comprehensive Similarity Index

[0061] By applying a logarithmic transformation to the three indices, a comprehensive topological similarity model ω is established at time t. t :

[0062]

[0063] Step 6 specifically includes the following sub-steps:

[0064] Step 6.1: Traverse all topologies. Assuming there are m radar targets and n camera targets, generate the radar-camera fusion matrix F. In F, ζ nm ζ represents the similarity between radar projection point m and camera detection box center point n. If radar projection point m and camera detection box center point n lack correlation, their interaction is not considered. nm =0; If there are multiple fusion evaluations for radar projection point m and camera detection box center point n, the topology with the highest similarity among these fusion evaluations is selected to ensure data accuracy and system effectiveness, with similarity ζ. nm =max(ζ nm1 ,ζ nm2 ,…);

[0065]

[0066] Set the topological similarity threshold to ω. t =0.5, for the obtained fusion matrix F, find the maximum value of all elements in F, let's say it is ζ. ijmax Then, set the remaining elements in the i-th row and j-th column of the matrix to 0, and repeat the above process until the maximum value ζ in F is reached. ijmax ≤0.5, complete the matching of radar projection points and camera detection frame center points.

[0067] Step 7 specifically includes the following sub-steps:

[0068] Step 7.1: Unfused YOLO detection boxes

[0069] To address the issue of unfused target bounding boxes in camera detection, a method using radar data for assisted fusion is proposed. The method obtains the corresponding radar azimuth line based on the azimuth information of the unfused camera detection bounding boxes. Then, a search operation is performed on the azimuth line to identify the radar detection point closest to the azimuth line. This method can find the corresponding radar target point for the target bounding boxes that were not correctly fused in the camera system detection, thereby achieving effective fusion of the two types of sensor data.

[0070] φ r φ is the azimuth angle of the radar image. g φ is the camera azimuth angle. s Let u be the camera's field of view, and u be the x-coordinate of the center point of the unfused camera's bounding box. z Given the horizontal resolution of the camera image, the formula for the azimuth angle of the radar image is:

[0071]

[0072] Based on the radar chart azimuth information determined by formula (39), the corresponding azimuth line is drawn on the radar chart, and the nearest radar detection point is located through this line. In this process, the identified radar point is fused with a specific detection box to achieve more accurate target identification and positioning.

[0073] Step 7.2: Unfused radar points

[0074] For unfused radar target points, an image cropping technique centered on the radar detection point is adopted to further analyze the surrounding image information and solve the problem of data non-fusion;

[0075] For the target points identified by the radar, it is first determined whether there is a detection box within the coarse correlation gate. If not, a local region centered on these points is extracted from the original image. These local region images are usually small. When this small image is enlarged to 640×640 pixels to meet the input requirements of the YOLO network, the proportion of pixels occupied by the target in the image increases, which can highlight the target features and make it easier for the algorithm to perform effective detection.

[0076] The present invention provides a method for detecting the fusion of a radar and a camera on an unmanned boat. The radar and the camera are respectively used to detect a target. A coarse association gate is set to reduce the calculation amount. The straight-line fitting deflection method is used to reduce the influence of hull swaying. A topological fusion index is set for the fusion of the radar and the camera. Finally, bidirectional association is performed on the unassociated data. This method realizes the detection of sea surface targets by the radar and the camera carried on the unmanned boat, can simultaneously obtain information such as the target distance, category, image, and azimuth, and improves the comprehensive situation awareness ability of the unmanned boat. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a flowchart of the radar-camera fusion in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0078] A topological bidirectional fusion detection algorithm for a marine unmanned boat radar and camera proposed by the present invention refers to Figure 1 , and includes the following steps:

[0079] Step 1: Time registration of the radar and the camera. Time registration is to align the data acquired by the radar and the camera in time to ensure that the moments of their data acquisition are the same. In this paper, the "interpolation / extrapolation method" is used for time registration.

[0080] To achieve the time registration of the radar and camera data, assuming that the initial measurement times are the same, the subsequent steps are as follows:

[0081] Step 1.1: If the time difference between the interpolation point and the last radar sampling time is less than the sampling interval, use the data of the last measurement for extrapolation.

[0082] Step 1.2: If the time difference between the interpolation point and the last radar sampling time participating in the determination is greater than the sampling interval, use the data of the last measurement closest to the interpolation point for extrapolation.

[0083] Step 1.3: If the time difference between the interpolation point and the last radar sampling time participating in the determination is equal to the sampling interval, keep the data of the radar at this moment unchanged.

[0084] Step 1.4: Process the radar and camera data according to the above steps to align them at the same time point and achieve time registration.

[0085] Step 2: Spatial registration of the radar and the camera. Perform external and internal calibration to obtain the geometric parameters of the radar and the camera, and establish the spatial transformation relationship between them. Then, through the extraction and matching of feature points or feature regions, find the corresponding relationship between the radar data and the camera data, and finally achieve spatial registration.

[0086]

[0087] Equation (27) is the coordinate transformation matrix, (X cam , Ycam Z cam (X) represents the coordinates in the photoelectric coordinate system. world ,Y world Z world ) represents the coordinates in the world coordinate system, R is the rotation matrix, and T is the translation matrix. f cam Let (x, y) be the photoelectric focal length, (x, y) be the coordinates in the image coordinate system, and (u, v) be the pixel coordinates. The radar coordinate system is set to the world coordinate system.

[0088] Step 3: Linear fitting using the PROSAC algorithm.

[0089] Unmanned surface vessels (USVs) equipped with 2D radar and cameras detect targets on the sea surface. However, due to the hull's movement caused by waves, the projection error of radar detection points is relatively large. Targets in the camera images are all located near the waterline, and the radar detection points projected onto the camera image appear to be distributed around the waterline. To address this issue, a radar point straight-line fitting technique based on the PROSAC algorithm is introduced. This algorithm is used to construct a straight-line model of the radar data projection points. Furthermore, the obtained straight-line model is adjusted using synchronously acquired USV attitude information to reduce radar target positioning errors caused by USV hull movement, thereby optimizing the detection results.

[0090] The PROSAC algorithm's linear fitting process is as follows:

[0091] Step 3.1: The set of N radar projection points is First, calculate the quality score of each point in the point set. The calculation method is as follows: traverse each point, calculate the squared distance of it to all other points, and then calculate the inverse sum of these squared distances as the quality score of that point.

[0092] Projection point The mass fraction is defined as:

[0093]

[0094] Step 3.2: Assess the set based on the mass fraction Sort in descending order as follows: Ensure that high-quality points are prioritized as candidate points. The candidate point set is defined as follows: The initial number of candidate points is n=1, which increases with the number of iterations t until the value of iteration t is reached or The number of points is N. This means that in the t-th iteration, the first n sorted points will be used as candidate points to generate the model and find the inliers, and the first two points will be selected to calculate the equation of the line.

[0095] Step 3.3: Calculate the distance from all points to the current line, and determine which points are interior points based on a threshold v. If the number of interior points found exceeds the current best result, update the best line and the set of interior points. Assuming the number of interior points is η, and P is the prior probability, then the number of iterations t is defined as:

[0096]

[0097] After straight-line fitting, straight-line deflection is performed based on the ship's attitude angle.

[0098] Step 4: Coarse Correlation Gate Generation. A coarse correlation gate threshold is established based on the range and azimuth error of the sensor system and the bounding box center position loss error of the YOLO algorithm to coarsely filter radar points.

[0099] Step 4.1: Center position loss threshold.

[0100] Based on the bounding box center position loss of the YOLO algorithm, a loss threshold λ is generated. loss :

[0101]

[0102] λ coord λ represents the corresponding weight term. Since this loss is only used to generate the loss threshold in this paper, λ is set accordingly. coord =1. s 2 All grids representing s×s need to be calculated, and B represents the number of box classes. This indicates whether the j-th box in the i-th grid should predict the target. If it should, then... otherwise λ loss As a center position loss threshold, it is used to form a coarse correlation gate to filter radar points.

[0103] Step 4.2: System Error Threshold

[0104] At time k, the track point spacing R between radar α and camera β for the same target (target i detected by radar, target j detected by camera) is... ij (k) can be derived as follows:

[0105]

[0106] Δr α ,Δr β For the positional error of the radar and camera, Δθ α ,Δθ β Let R be the angle error of the radar and camera, and the rest be fixed values. Therefore, according to equation (31), if R... ij (k) Take the maximum value R max , Δθα ,Δθ β ,Δr α ,Δr β The value should be:

[0107] Δθ α =Δθ αmax Δθ β =Δθ βmax Δr α =Δr αmax Δr β =-Δr βmax (32)

[0108] Use the maximum value R of the trackpoint spacing max This serves as the system error threshold. The above calculation is performed on all targets at time k, and the maximum value is taken as the system error threshold for that time.

[0109] Step 4.3: Generate coarse correlation gate.

[0110] The system error threshold is located in the radar coordinate system (X). world ,Y world Z world Under this condition, the center position loss threshold lies in the pixel coordinate system (u,v). Therefore, the system error threshold R is set to... max Transform to the pixel coordinate system using formula (27), and define the systematic error threshold in the pixel coordinate system as: R uv max

[0111] The coarse correlation gate size is defined as:

[0112] L max =s*(R) uv max +λ loss (33)

[0113] s is a scaling factor that can be adjusted according to actual conditions. If the coarse correlation gate is too large, it will increase unnecessary computation; if the coarse correlation gate is too small, it will filter out correct radar points. In this paper, s is set to 2.

[0114] With coarse correlation gate L max Using a radius of 1, draw a circle centered at the center point of each YOLO detection frame to generate a coarse correlation gate. For each YOLO detection frame, only radar points within the gate are fused, excluding radar points outside the gate.

[0115] Step 5: Establishment of Topological Fusion Similarity Index. Based on traditional side length similarity comparison, a comprehensive topological fusion index is developed. This index adds two new dimensions: polygon angle similarity and the similarity of lines connecting polygon center points.

[0116] Step 5.1: Side length similarity index.

[0117] An evaluation index is constructed based on the similarity of the lengths of the three sides of a triangle, defined as follows:

[0118]

[0119] The similarity of the side lengths at time t is represented by . Let i represent the i-th side of radar triangle α. Let represent the i-th side of the camera triangle β.

[0120] Step 5.2: Polygon angular similarity index.

[0121] The similarity of polygon angles is defined as:

[0122]

[0123] The polygon angular similarity at time t is represented by... This represents the i-th interior angle of the radar polygon α. Let n1 represent the i-th interior angle of the camera polygon β, and n1 be the number of interior angles of the polygon.

[0124] Step 5.3: Similarity index of connecting center points.

[0125] Connecting the center point of radar polygon α to the i-th vertex yields line L, which is the line connecting the center points of radar polygon α. α Connect the center point of the camera polygon β and the i-th vertex to obtain the line L connecting the center points of the camera polygon β. β The similarity of the lines connecting the center points is defined as:

[0126]

[0127] The similarity of the lines connecting the center points at time t. and For the i-th L α The included angle and length, and For the i-th L β The included angle and length.

[0128] Step 5.4: Comprehensive similarity index.

[0129] By applying a logarithmic transformation to the three indices, a comprehensive topological similarity model ω is established at time t. t :

[0130]

[0131] Step 6: Establish the fusion matrix and match radar and camera.

[0132] Traverse all topologies, assuming the number of radar targets is m and the number of camera targets is n, generate the radar-camera fusion matrix F, where ζ nm ζ represents the similarity between the radar projection point m and the camera detection box center point n. If the radar projection point m and the camera detection box center point n lack correlation, their interaction is not considered. nm =0; If there are multiple fusion evaluations for radar projection point m and camera detection box center point n, the topology with the highest similarity among these fusion evaluations is selected to ensure data accuracy and system effectiveness, with similarity ζ. nm =max(ζ nm1 ,ζ nm2 ,…).

[0133]

[0134] Set the topological similarity threshold to ω. t =0.5, for the obtained fusion matrix F, find the maximum value of all elements in F, let's say it is ζ. ijmax Then, set the remaining elements in the i-th row and j-th column of the matrix to 0. Repeat the above process until the maximum value ζ in F is reached. ijmax ≤0.5. Complete the matching of radar projection points and camera detection frame center points.

[0135] Step 7: Two-way fusion of unfused radar and camera data

[0136] Step 7.1: Unfused YOLO detection boxes

[0137] To address the issue of unfused target bounding boxes detected by cameras, a method using radar data for assisted fusion is proposed. The method involves obtaining the corresponding radar azimuth line from the azimuth information of the unfused camera bounding boxes. Then, a search operation is performed on this azimuth line to identify the radar detection point closest to it. This approach enables the identification of corresponding radar target points for target bounding boxes that were not correctly fused in the camera system detection, thus achieving effective fusion of the two types of sensor data.

[0138] φ r φ is the azimuth angle of the radar image. g φ is the camera azimuth angle. s Let u be the camera's field of view, and u be the x-coordinate of the center point of the unfused camera's bounding box. z Given the horizontal resolution of the camera image, the formula for the azimuth angle of the radar image is:

[0139]

[0140] Based on the radar azimuth information determined by formula (39), the corresponding azimuth line is drawn on the radar image, and the nearest radar detection point is located through this line. In this process, the identified radar points are fused with specific detection boxes to achieve more accurate target identification and positioning.

[0141] Step 7.2: Unfused radar points

[0142] For unfused radar target points, an image cropping technique centered on the radar detection point is adopted to further analyze the surrounding image information and solve the problem of data non-fusion.

[0143] For target points identified by radar, the algorithm first checks if a detection box exists within the coarse correlation gate. If not, a local region centered on these points is extracted from the original image. These local regions are typically small. When this small image is enlarged to 640×640 pixels to meet the input requirements of the YOLO network, the proportion of pixels occupied by the target in the image increases. This highlights the target features, making it easier for the algorithm to perform effective detection.

[0144] This invention proposes a two-way fusion detection algorithm for radar and camera topology of unmanned surface vessels (USVs), comprising the following technical measures: First, time registration of the sensor system is performed to solve the time synchronization problem, acquiring optical images and radar detection points at the same time. YOLOV7-tiny is used to detect the camera image, outputting the center coordinates of the detection box and the target category. Based on the camera's maximum detection range, radar points within that range are projected onto the optical image through matrix transformation, ensuring that the sensor system's detection results are all located in the pixel coordinate system, achieving system spatial registration. PROSAC straight-line fitting is performed on the projected radar points, and the fitted radar line is deflected according to the ship's attitude angle to reduce the influence of ship sway. A coarse correlation gate is set using the sensor's system error and the position error loss of the YOLO algorithm to filter radar points and reduce computational load. Then, a topology fusion index is set, and radar and camera topology fusion is performed. Finally, two-way fusion is performed on uncorrelated radar points and detection boxes.

Claims

1. A method for bidirectional fusion detection of a radar-camera topology of a marine unmanned surface vehicle, characterized in that, The method comprises the following steps: Step 1) Time registration of radar and camera; Step 2) Space registration of radar and camera: external and internal calibration are performed, geometric parameters of radar and camera are obtained, a space transformation relationship between them is established, a corresponding relationship between radar data and camera data is found, and space registration is realized; Step 3) PROSAC algorithm straight line fitting: the algorithm is used to construct a straight line model for radar data projection points, and the obtained straight line model is adjusted through the simultaneously obtained unmanned ship attitude information; Step 4) Establishment of coarse association wave gate: a coarse association wave gate threshold is established according to the distance and azimuth error of the sensor system and the center position loss error of the bounding box of the YOLO algorithm, and the radar points are coarsely screened; Step 5) Establishment of topological fusion similarity index: an integrated fusion similarity index is established by using edge length similarity, polygon angle similarity and line similarity between polygon center points; Step 6) Matrix establishment and radar and camera matching: a fusion matrix is established, and matching of radar projection points and camera detection frame center points is completed according to the threshold; Step 7) Two-way fusion of unfused radar and camera data: image cutting around the unfused radar points is detected and fused; for the unfused optical detection frame, the nearest neighbor radar point of the azimuth line is selected for fusion, so as to realize two-way fusion.

2. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 1 adopts an "interpolation / extrapolation method" for time registration, and specifically comprises the following sub-steps: Step 1.1: if the interpolation point and the last radar sampling time difference are less than the sampling interval, the last measurement data is used for extrapolation; Step 1.2: if the interpolation point and the last radar sampling time difference participating in judgment are greater than the sampling interval, the last measurement data closest to the interpolation point is used for extrapolation; Step 1.3: if the interpolation point and the last radar sampling time difference participating in judgment are equal to the sampling interval, the data of the radar at this time is kept unchanged; Step 1.4: the radar and camera data are processed according to the above steps to align them at the same time point, so as to realize time registration.

3. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 2 specifically comprises the following sub-steps: Step 2.1: external and internal calibration are performed, geometric parameters of radar and camera are obtained, a space transformation relationship between them is established, and then through extraction and matching of feature points or feature regions, a corresponding relationship between radar data and camera data is found, so as to realize space registration; （1） Equation (1) is a coordinate conversion matrix, is a rotation matrix, is a translation matrix, is a pixel coordinate, and the radar coordinate system is set as the world coordinate system.

4. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 3 specifically comprises the following sub-steps: Step 3.1: A set of radar projection points is First, the mass fraction of each point in the point set is calculated. The calculation method is as follows: traverse each point, calculate the square of the distance between the point and all other points, and then take the inverse sum of the square of the distance as the mass fraction of the point. projected point The mass fraction of the point of projection is defined as: （2） Step 3.2: Assess the set based on the mass fraction Sort in descending order as follows: To ensure that high-quality points are selected as candidate points, the candidate point set is defined as follows: The initial number of candidate points is As the number of iterations increases Increment until the number of iterations is reached. value or Points In the In the next iteration, the previous one will be used. The sorted points are used as candidate points to generate the model and find the interior points. The first two points are selected to calculate the equation of the line. Step 3.3: Calculate the distance of all points to the current line and according to the threshold Determine which points are inliers, if the number of inliers found exceeds the current best result, update the best line and inlier set, assuming the number of inliers is , The prior probability is, then the number of iterations is defined as: （3） After straight line fitting, the straight line is deflected according to the ship body attitude angle.

5. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 3, characterized in that The step 4 specifically comprises the following sub-steps: Step 4.1: center position loss threshold: According to the bounding box center position loss of the YOLO algorithm, a loss threshold is generated ; （4） in, Represents the corresponding weight item, set , represent All grids need to be calculated. The number of classes representing boxes, Indicates the first The first grid Does each box predict the target? If so, then... ,otherwise ,Will As a center position loss threshold, it is used to form a coarse correlation gate to filter radar points; Step 4.2: system error threshold; Radar detection as target Camera detects the target , Time Radar and camera Report the distance between track points for the same target After derivation, it can be expressed as: （5) , The position error for the radar and camera, , The angle error for the radar and camera, the rest values are determined values, according to equation (5), if The maximum value is taken , The value of should be: （6) Adopting the maximum value of the track point interval As the system error threshold, for All targets at the moment are calculated above, and the maximum value is taken as the system error threshold at the moment; Step 4.3: coarse association wave gate generation; The system error threshold is located in the radar coordinate system The center position loss threshold is located in the pixel coordinate system Therefore, the system error threshold By formula (1) into the pixel coordinate system, the definition of the pixel coordinate system under the system error threshold size is: ; The size of the coarse association wave gate is defined as: （7） For the scale factor, it can be adjusted according to the actual situation. If the coarse correlation wave gate is too large, it will increase unnecessary calculation; if the coarse correlation wave gate is too small, it will filter out the correct radar points. Set ; coarse correlation gate Using the radius as the center point of each YOLO detection frame, a coarse correlation gate is generated. For each YOLO detection frame, only radar points within the gate are fused, while radar points outside the gate are excluded.

6. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 5 specifically comprises the following sub-steps: Step 5.1: edge length similarity index An evaluation index is constructed according to the similarity of the lengths of the three sides of a triangle, and is defined as: （8） representative edge length similarity, representative radar triangle first side, representative camera triangle first side; Step 5.2: polygon angle similarity index The polygon angle similarity is defined as: （9） representative polygonal angle similarity at the time instant, representative radar polygon first interior angle, representative camera polygon first interior angle, is the number of interior angles of the polygon; Step 5.3: center point line similarity index; connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon connecting the center point of the radar polygon and the first vertex of the camera polygon （10) represent Similarity of the lines connecting the center points at different times. and For the first indivual The included angle and length, and For the first indivual The included angle and length; Step 5.4: comprehensive similarity index The three indexes are logarithmically transformed to establish a comprehensive topology similarity model comprehensive topology similarity model : （11）。 7. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 6 specifically comprises the following sub-steps: Step 6.1: Traverse all topologies, assuming the number of radar targets is... The number of camera targets is Generate radar and camera fusion matrix ,exist middle, Represents radar projection point and the center point of the camera detection frame The similarity, if the radar projection points and the center point of the camera detection frame If there is a lack of correlation, then the interaction between them is not considered. If the radar projection point and the center point of the camera detection frame There are multiple fusion assessments. The topology with the highest similarity among these assessments is selected to ensure data accuracy and system effectiveness. The similarity is [value missing]. ; （12） The topology similarity threshold is set as , the maximum value of all elements in the obtained fusion matrix is found , and is assumed to be , and then the remaining elements in the first row and the first column of the matrix are set to 0, and the above process is repeated until the maximum value in the matrix is found , and the matching of the radar projection points and the camera detection box center points is completed.

8. The radar-camera topology bidirectional fusion detection method for an unmanned marine vehicle according to claim 1, characterized in that The step 7 specifically comprises the following sub-steps: Step 7.1: unfused YOLO detection frame Aiming at the unfused target frame in camera detection, a method of auxiliary fusion using radar data is proposed. According to the orientation information of the unfused camera detection frame, the corresponding radar chart orientation line is obtained. Then, for the orientation line, a search operation is performed to identify the radar detection point closest to the orientation line. In this way, the radar target point corresponding to the target frame that was not correctly fused in the camera system detection can be found, thereby realizing effective fusion of the two kinds of sensing data. is the radar plot azimuth angle, is the camera azimuth angle, is the camera field of view angle, is the un-fused camera bounding box center point x-coordinate, is the camera image horizontal resolution, so the radar plot azimuth angle formula is: （13) According to the radar chart azimuth angle information determined by formula (13), the corresponding azimuth line is drawn on the radar chart, and the nearest radar detection point is located by this line. In this process, the identified radar point is fused with a specific detection frame to achieve more accurate target recognition and positioning. Step 7.2: Unfused radar points For unfused radar target points, the image cropping technique centered on the radar detection point is used to further analyze the surrounding image information and solve the data unfusion problem. For the radar-identified target points, first determine whether there are detection frames within the coarse correlation gate. If not, then the local area centered on these points is cropped from the original image. These local area images are usually small. When such small-sized images are enlarged to 640x640 pixels to meet the YOLO network input requirements, the pixel proportion of the target in the image increases, highlighting the target features and making it easier for the algorithm to perform effective detection.

Images