Bidirectional fusion detection method for radar camera topology of unmanned ship at sea

Abstract

The invention discloses a maritime unmanned ship radar camera topology bidirectional fusion detection algorithm, and belongs to the technical field of radar camera fusion in multi-sensor fusion. Firstly, time registration of a sensor system is carried out, an optical image and a radar detection point at the same moment are obtained, after a camera image is detected by adopting YOLOV7-tiny, a center coordinate of a detection frame and a target category are output, and the radar point in a range is projected to the optical image through matrix conversion according to the maximum detection distance of a camera; the method comprises the following steps: ensuring that detection results of a sensor system are all located in a pixel coordinate system, realizing system space registration, carrying out PROSAC straight line fitting on projection radar points, deflecting a radar fitting line according to a ship attitude angle, and setting a coarse correlation gate by using system errors of a sensor and position error loss of a YOLO algorithm to screen radar points, reduce the calculation amount and improve the detection accuracy. And then topology fusion indexes are set, radar and camera topology fusion is carried out, and finally bidirectional fusion is carried out on unassociated radar points and detection frames.

Description

Technical Field

[0001] The invention relates to a topological bidirectional fusion detection method for a radar camera of an unmanned marine boat, belonging to the technical field of radar camera fusion in multi-sensor fusion. Background Art

[0002] Radar and camera fusion technology can effectively improve the detection capability of unmanned boats in the surrounding sea areas. In the technical development of unmanned boats for sea surface target detection, the traditional method usually uses radar to guide the camera system to capture the target. This method relies on adjusting the angle of the camera system according to the radar position to achieve target capture. Because it relies heavily on manual interpretation and lacks real-time performance, it is practical in an environment with fewer targets and no obstructions, but it cannot achieve accurate detection of multiple targets at the same time. In recent years, there are many IoU fusion methods for lidar and on-board cameras in the field of autonomous driving. Multiple fixed cameras are carried on the car to implement 360° shooting and achieve all-round fusion with radar. The on-board lidar has high accuracy and can form a radar point cloud image, but the detection distance is only a few hundred meters. In the marine scene, in most cases, the detection distance of the unmanned boat is far and the error is large. In addition, the unmanned boat is shaken by the influence of sea waves. The algorithm needs to be improved to achieve radar camera fusion detection of the unmanned boat.

[0003] Therefore, it is urgent to design a topological bidirectional fusion detection algorithm for the radar and camera of unmanned boats at sea, aiming to solve the problem of effective fusion detection of radars and cameras carried by unmanned boats and obtain the image, distance, category, direction and other information of maritime targets. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the above-mentioned prior art and to provide a topological bidirectional fusion detection method for radar cameras of unmanned marine boats, which can effectively detect long-distance targets at sea when the unmanned marine boat is shaking, multiple targets are blocked, and errors occur.

[0005] The present invention discloses a topological bidirectional fusion detection algorithm for a marine unmanned boat radar camera, which is special in that it includes the following steps:

[0006] Step 1) Time alignment between radar and camera;

[0007] Step 2) Spatial registration of radar and camera: perform external and internal calibration, obtain the geometric parameters of radar and camera, establish the spatial transformation relationship between them, find the correspondence between radar data and camera data, and realize spatial registration;

[0008] Step 3) PROSAC algorithm straight line fitting: Use the algorithm to construct a straight line model for the radar data projection points, and adjust the obtained straight line model through the synchronously acquired unmanned boat attitude information;

[0009] Step 4) Establishing a coarse correlation gate: Establish a coarse correlation gate threshold based on the distance and azimuth error of the sensor system and the boundary box center position loss error of the YOLO algorithm to roughly screen the radar points;

[0010] Step 5) Establishing the topological fusion similarity index: Establishing a comprehensive fusion similarity index using the edge length similarity, polygon angle similarity, and line similarity between polygon center points;

[0011] Step 6) Matrix establishment and radar and camera matching: Establish a fusion matrix and complete the matching of the radar projection point and the center point of the camera detection frame according to the threshold;

[0012] Step 7) Bidirectional fusion of unfused radar and camera data: perform detection fusion on the image capture near the unfused radar point; for the unfused optical detection frame, select the nearest neighbor radar point of the azimuth line for fusion to achieve bidirectional fusion.

[0013] The step 1 adopts the “interpolation / extrapolation method” to perform time registration, which specifically includes the following sub-steps:

[0014] Step 1.1: If the time difference between the interpolation point and the last radar sampling is less than the sampling interval, use the last measurement data for extrapolation;

[0015] Step 1.2: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is greater than the sampling interval, use the last measurement data closest to the interpolation point for extrapolation;

[0016] Step 1.3: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is equal to the sampling interval, keep the radar data at that moment unchanged;

[0017] Step 1.4: Process the radar and camera data as described above to align them at the same time point to achieve temporal registration.

[0018] The step 2 specifically includes the following sub-steps:

[0019] Step 2.1: Perform external and internal calibration, obtain the geometric parameters of the radar and camera, establish the spatial transformation relationship between them, and then find the correspondence between the radar data and the camera data through the extraction and matching of feature points or feature areas to achieve spatial registration:

[0020]

[0022] Formula (27) is the coordinate transformation matrix, (X cam ,Y cam ,Z cam ) is the coordinate in the photoelectric coordinate system, (X world ,Y world ,Zworld ) is the coordinate in the world coordinate system, R is the rotation matrix, T is the translation matrix, f cam is the photoelectric focal length, (x, y) is the coordinate in the image coordinate system, (u, v) is the pixel coordinate, and the radar coordinate system is set to the world coordinate system.

[0023] The step 3 specifically includes the following sub-steps:

[0024] Step 3.1: The set of N radar projection points is First, calculate the quality score of each point in the point set. The calculation method is: traverse each point, calculate the square of its distance to all other points, and then find the inverse sum of these squared distances as the quality score of the point;

[0025] Projection Point The quality score is defined as:

[0026]

[0027] Step 3.2: Assemble the collections according to their quality scores Sort descending as: Ensure that high-quality points are selected as candidate points first. The candidate point set is defined as n=min(t,N), the initial number of candidate point sets is n=1, and increases with the number of iterations t until it reaches the value of the number of iterations t or The number of points is N. In the tth iteration, the first n ordered points are used as candidate points to generate the model and find the inner points, and the first two points are selected to calculate the equation of the line;

[0028] Step 3.3: Calculate the distance from all points to the current line, and determine which points are inliers based on the threshold υ. If the number of inliers found exceeds the current best result, update the best line and inlier set. Assuming that the number of inliers is η and P is the prior probability, the number of iterations t is defined as:

[0029]

[0030] After the straight line is fitted, the straight line is deflected according to the hull attitude angle.

[0031] The step 4 specifically includes the following sub-steps:

[0032] Step 4.1: Center position loss threshold:

[0033] According to the loss of the center position of the bounding box of the YOLO algorithm, the loss threshold λ is generated loss :

[0034]

[0035] Among them, λcoord Represents the corresponding weight term, set λ coord =1, s 2 The s×s grids need to be calculated, B represents the number of box classes, Indicates whether the jth box of the i-th grid predicts the target. If so, otherwise λ loss As the center position loss threshold, it is used to form a coarse correlation wave gate to screen radar points;

[0036] Step 4.2: System Error Threshold

[0037] The track point distance R between radar α and camera β reporting the same target (target i for radar detection and target j for camera detection) at time k ij (k) can be expressed as follows after derivation:

[0038]

[0039] Δr α ,Δr β is the position error between the radar and the camera, Δθ α ,Δθ β is the angle error between the radar and the camera, and the other values ​​are fixed values. According to formula (31), if R ij (k) Take the maximum value R max , Δθ α ,Δθ β ,Δr α ,Δr β The value should be:

[0040] Δθα=Δθ αmax Δθ β =Δθ βmax Δr α =Δr αmax Δr β =-Δr βmax (19)

[0041] The maximum value R of the track point spacing is used max As the system error threshold, the above calculation is performed for all targets at time k, and the maximum value is taken as the system error threshold at that moment;

[0042] Step 4.3: Coarse correlation gate generation

[0043] The system error threshold is located in the radar coordinate system (X world ,Y world ,Z world), the center position loss threshold is located in the pixel coordinate system (u, v), so the system error threshold R max Convert to the pixel coordinate system through formula (27), and define the system error threshold in the pixel coordinate system as: R uv max ;

[0044] The coarse correlation gate size is defined as:

[0045] L max =s*(R uv max +λ loss ) (20)

[0046] s is the proportional factor, which can be adjusted according to the actual situation. If the coarse correlation wave gate is too large, it will increase unnecessary calculation amount; if the coarse correlation wave gate is too small, the correct radar points will be screened out. Set s = 2;

[0047] Rough correlation gate L max As the radius, a circle is drawn with the center point of each YOLO detection frame as the center to generate a coarse correlation gate. For each YOLO detection frame, only the radar points within the gate are fused, and the radar points outside the gate are excluded.

[0048] The step 5 specifically includes the following sub-steps:

[0049] Step 5.1: Edge length similarity index

[0050] The evaluation index is constructed based on the similarity of the lengths of the three sides of the triangle, which is defined as:

[0051]

[0052] represents the edge length similarity at time t, represents the i-th side of radar triangle α, represents the i-th edge of the camera triangle β;

[0053] Step 5.2: Polygon Angle Similarity Metric

[0054] Polygon angle similarity is defined as:

[0055]

[0056] represents the polygon angle similarity at time t, represents the i-th interior angle of radar polygon α, represents the i-th internal angle of the camera polygon β, and n1 is the number of internal angles of the polygon;

[0057] Step 5.3: Center point connection similarity index

[0058] Connect the center point of radar polygon α and the i-th vertex to obtain the center point line L of radar polygon α α , connect the center point of the camera polygon β and the i-th vertex to get the center point line L of the camera polygon β β , the center point connection similarity is defined as:

[0059]

[0060] Represents the similarity of the center point connection at time t, and is the i-th L α The angle and length of and is the i-th L β The angle and length of

[0061] Step 5.4: Comprehensive similarity index

[0062] The comprehensive topological similarity model ω at time t is established by logarithmically transforming the three indicators. t :

[0063]

[0064] The step 6 specifically includes the following sub-steps:

[0065] Step 6.1: Traverse all topological structures, assuming that the number of radar targets is m and the number of camera targets is n, generate the radar and camera fusion matrix F. In F, ζ nm represents the similarity between the radar projection point m and the camera detection frame center point n. If the radar projection point m and the camera detection frame center point n lack correlation, the linkage between them is not considered. nm = 0; if there are multiple fusion evaluations between the radar projection point m and the camera detection frame center point n, the topological structure with the highest similarity among these fusion evaluations is selected to ensure the accuracy of the data and the effectiveness of the system. The similarity is ζ nm =max(ζ nm1 ,ζ nm2 ,…);

[0066]

[0067] Set the topological similarity threshold to ω t = 0.5, for the obtained fusion matrix F, find the maximum value of all elements in F, assuming it is ζ ijmax , and then return the remaining elements of the matrix in the i-th row and j-th column to 0, and repeat the above process until the maximum value ζ in F is reachedijmax ≤0.5, complete the matching between the radar projection point and the center point of the camera detection frame.

[0068] The step 7 specifically includes the following sub-steps:

[0069] Step 7.1: Unfused YOLO detection box

[0070] For the target frame that is not fused in the camera detection, a method of auxiliary fusion using radar data is proposed. The corresponding radar map azimuth line is obtained according to the azimuth information of the unfused camera detection frame. Then, a search operation is performed on the azimuth line to identify the radar detection point closest to the azimuth line. Through this method, the corresponding radar target point can be found for the target frame that was not correctly fused in the camera system detection, thereby realizing the effective fusion of the two sensor data.

[0071] φ r is the radar chart azimuth, φ g is the camera azimuth, φ s is the camera field of view, u is the horizontal coordinate of the center point of the unfused camera detection frame, and u z is the lateral resolution of the camera image, so the radar image azimuth angle formula is:

[0072]

[0073] According to the azimuth information of the radar map determined by formula (39), the corresponding azimuth line is drawn on the radar map, and the nearest radar detection point is located through this line. In this process, the identified radar point is fused with the specific detection frame to achieve more accurate target recognition and positioning;

[0074] Step 7.2: Unfused radar points

[0075] For unfused radar target points, the image interception technology centered on the radar detection point is used to further analyze the surrounding image information and solve the problem of unfused data;

[0076] For the target points identified by the radar, we first determine whether there is a detection frame in the coarse correlation gate. If not, we extract the local area centered on these points from the original image. These local area images are usually small. When this small-size image is enlarged to 640×640 pixels to meet the input requirements of the YOLO network, the proportion of pixels occupied by the target in the image increases, which can highlight the target features and make it easier for the algorithm to perform effective detection.

[0077] The present invention provides a radar-camera fusion detection method for unmanned boats. The radar and camera are used to detect the target respectively, a coarse correlation gate is set to reduce the amount of calculation, a straight line fitting deflection method is used to reduce the influence of the hull shaking, and a topological fusion index is set to fuse the radar and camera, and finally the unrelated data is bidirectionally correlated. The method realizes the detection of sea surface targets by the radar and camera carried by the unmanned boat, and can simultaneously obtain information such as target distance, category, image and orientation, thereby improving the comprehensive situation awareness capability of the unmanned boat. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flow chart of radar camera fusion in the present invention. DETAILED DESCRIPTION

[0079] The present invention proposes a topological bidirectional fusion detection algorithm for unmanned marine boat radar cameras, referring to Figure 1 , including the following steps:

[0080] Step 1: Time alignment of radar and camera. Time alignment is to align the data acquired by radar and camera in time to ensure that they collect data at the same time. This paper uses the "interpolation / extrapolation method" for time alignment.

[0081] To achieve time alignment of radar and camera data, assuming the initial measurements are time consistent, the following steps are performed:

[0082] Step 1.1: If the time difference between the interpolation point and the last radar sampling is less than the sampling interval, use the last measurement data for extrapolation.

[0083] Step 1.2: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is greater than the sampling interval, use the last measurement data closest to the interpolation point for extrapolation.

[0084] Step 1.3: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is equal to the sampling interval, keep the radar data at that moment unchanged.

[0085] Step 1.4: Process the radar and camera data as described above to align them at the same time point to achieve temporal registration.

[0086] Step 2: Spatial registration of radar and camera. Perform external and internal calibration to obtain the geometric parameters of radar and camera and establish the spatial transformation relationship between them. Then, by extracting and matching feature points or feature areas, find the corresponding relationship between radar data and camera data, and finally achieve spatial registration.

[0087]

[0088] Formula (27) is the coordinate transformation matrix, (X cam ,Ycam ,Z cam ) is the coordinate in the photoelectric coordinate system, (X world ,Y world ,Z world ) is the coordinate in the world coordinate system, R is the rotation matrix, and T is the translation matrix. cam is the photoelectric focal length, (x, y) is the coordinate in the image coordinate system, and (u, v) is the pixel coordinate. Set the radar coordinate system to the world coordinate system.

[0089] Step 3: PROSAC algorithm straight line fitting.

[0090] The unmanned boat is equipped with a two-dimensional radar and a camera to detect targets on the sea surface. Due to the influence of the hull on the shaking of the waves, the projection error of the radar detection point is large. The targets in the camera image are all located near the water-sky line. After the radar detection points are projected onto the camera image, they will be distributed near the water-sky line. To address the above problems, a radar point straight line fitting technology based on the PROSAC algorithm is introduced. The algorithm is used to construct a straight line model for the radar data projection points. Furthermore, the obtained straight line model is adjusted through the synchronously acquired unmanned boat attitude information to reduce the radar target positioning deviation caused by the shaking of the unmanned boat hull, thereby optimizing the detection results.

[0091] The PROSAC algorithm straight line fitting process is as follows:

[0092] Step 3.1: The set of N radar projection points is First, the quality score of each point in the point set is calculated by traversing each point, calculating the square of its distance to all other points, and then taking the inverse sum of these squared distances as the quality score of the point.

[0093] Projection Point The quality score is defined as:

[0094]

[0095] Step 3.2: Assemble the collections according to their quality scores Sort descending as: Ensure that high-quality points are selected as candidate points first. The candidate point set is defined as The initial number of candidate point sets is n = 1, and increases with the number of iterations t until it reaches the value of the number of iterations t or The number of points is N. This means that in the tth iteration, the first n sorted points will be used as candidate points to generate the model and find the inliers, and the first two points will be selected to calculate the line equation.

[0096] Step 3.3: Calculate the distances from all points to the current line, and determine which points are inliers based on the threshold v. If the number of inliers found exceeds the current best result, update the best line and inlier set. Assuming the number of inliers is η and P is the prior probability, the number of iterations t is defined as:

[0097]

[0098] After the straight line is fitted, the straight line is deflected according to the hull attitude angle.

[0099] Step 4: Coarse correlation gate generation: A coarse correlation gate threshold is established based on the distance and azimuth error of the sensor system and the boundary box center position loss error of the YOLO algorithm to perform coarse screening of radar points.

[0100] Step 4.1: Center position loss threshold.

[0101] According to the loss of the center position of the bounding box of the YOLO algorithm, the loss threshold λ is generated loss :

[0102]

[0103] λ coord Represents the corresponding weight term. Since this article only uses this loss to generate the loss threshold, we set λ coord =1.s 2 The s×s grids need to be calculated, and B represents the number of box classes. Indicates whether the jth box of the i-th grid predicts the target. If so, otherwise λ loss As the center position loss threshold, it is used to form a coarse correlation wave gate to screen radar points.

[0104] Step 4.2: System Error Threshold

[0105] The track point distance R between radar α and camera β reporting the same target (target i for radar detection and target j for camera detection) at time k ij (k) can be expressed as follows after derivation:

[0106]

[0107] Δr α ,Δr β is the position error between the radar and the camera, Δθ α ,Δθ β is the angle error between the radar and the camera, and the remaining values ​​are fixed values. Therefore, according to formula (31), if R ij (k) Take the maximum value R max , Δθα ,Δθ β ,Δr α ,Δr β The value should be:

[0108] Δθ α =Δθ αmax Δθ β =Δθ βmax Δr α =Δr αmax Δr β =-Δr βmax (32)

[0109] The maximum value R of the track point spacing is used max As the system error threshold. Perform the above calculations for all targets at time k, and take the maximum value as the system error threshold at that moment.

[0110] Step 4.3: Coarse correlation gate generation.

[0111] The system error threshold is located in the radar coordinate system (X world ,Y world ,Z world ), the center position loss threshold is located in the pixel coordinate system (u,v). Therefore, the system error threshold R max Convert to the pixel coordinate system through formula (27), and define the system error threshold in the pixel coordinate system as: R uv max

[0112] The coarse correlation gate size is defined as:

[0113] L max =s*(R uv max +λ loss ) (33)

[0114] s is a proportional factor, which can be adjusted according to actual conditions. If the coarse correlation gate is too large, unnecessary calculations will be increased; if the coarse correlation gate is too small, the correct radar points will be screened out. In this paper, s is set to 2.

[0115] Rough correlation gate L max With radius as the center point of each YOLO detection frame, a circle is drawn to generate a rough correlation gate. For each YOLO detection frame, only the radar points within the gate are fused, and the radar points outside the gate are excluded.

[0116] Step 5: Establishment of topological fusion similarity index. Based on the traditional edge length similarity comparison, a comprehensive topological fusion index is developed. This index adds two new measurement dimensions: polygon angle similarity and line similarity between polygon center points.

[0117] Step 5.1: Edge length similarity indicator.

[0118] The evaluation index is constructed based on the similarity of the lengths of the three sides of the triangle, which is defined as:

[0119]

[0120] represents the edge length similarity at time t, represents the i-th side of radar triangle α, Represents the i-th edge of the camera triangle β.

[0121] Step 5.2: Polygon angle similarity metric.

[0122] Polygon angle similarity is defined as:

[0123]

[0124] represents the polygon angle similarity at time t, represents the i-th interior angle of radar polygon α, Represents the i-th internal angle of the camera polygon β, and n1 is the number of internal angles of the polygon.

[0125] Step 5.3: Center point connection similarity index.

[0126] Connect the center point of radar polygon α and the i-th vertex to obtain the center point line L of radar polygon α α , connect the center point of the camera polygon β and the i-th vertex to get the center point line L of the camera polygon β β , the center point connection similarity is defined as:

[0127]

[0128] Represents the similarity of the center point connection at time t, and is the i-th L α The angle and length of and is the i-th L β Angle and length.

[0129] Step 5.4: Comprehensive similarity index.

[0130] The comprehensive topological similarity model ω at time t is established by logarithmically transforming the three indicators. t :

[0131]

[0132] Step 6: Establish fusion matrix and match radar and camera.

[0133] Traverse all topological structures, assuming that the number of radar targets is m and the number of camera targets is n, generate the radar and camera fusion matrix F. In F, ζ nm represents the similarity between the radar projection point m and the camera detection frame center point n. If the radar projection point m and the camera detection frame center point n lack correlation, the linkage between them is not considered. nm = 0; if there are multiple fusion evaluations between the radar projection point m and the camera detection frame center point n, the topological structure with the highest similarity among these fusion evaluations is selected to ensure the accuracy of the data and the effectiveness of the system. The similarity is ζ nm =max(ζ nm1 ,ζ nm2 ,…).

[0134]

[0135] Set the topological similarity threshold to ω t = 0.5, for the obtained fusion matrix F, find the maximum value of all elements in F, assuming it is ζ ijmax , and then set the remaining elements in the i-th row and j-th column of the matrix to 0. Repeat the above process until the maximum value ζ in F is ijmax ≤0.5. Complete the matching between the radar projection point and the center point of the camera detection frame.

[0136] Step 7: Bidirectional fusion of unfused radar and camera data

[0137] Step 7.1: Unfused YOLO detection box

[0138] For the target frame that is not fused in the camera detection, a method of auxiliary fusion using radar data is proposed. The corresponding radar map azimuth line is obtained according to the azimuth information of the unfused camera detection frame, and then a search operation is performed on the azimuth line to identify the radar detection point closest to the azimuth line. Through this method, the corresponding radar target point can be found for the target frame that was not correctly fused in the camera system detection, thereby realizing the effective fusion of the two sensor data.

[0139] φ r is the radar chart azimuth, φ g is the camera azimuth, φ s is the camera field of view, u is the horizontal coordinate of the center point of the unfused camera detection frame, and u z is the lateral resolution of the camera image, so the radar image azimuth angle formula is:

[0140]

[0141] According to the radar map azimuth information determined by formula (39), the corresponding azimuth line is drawn on the radar map, and the nearest radar detection point is located through this line. In this process, the identified radar point is fused with the specific detection frame to achieve more accurate target recognition and positioning.

[0142] Step 7.2: Unfused radar points

[0143] For unfused radar target points, image capture technology centered on the radar detection point is used to further analyze the surrounding image information and solve the problem of unfused data.

[0144] For the target points identified by the radar, first determine whether there is a detection frame in the coarse correlation gate. If not, then extract the local area centered on these points from the original image. These local area images are usually small. When this small-size image is enlarged to 640×640 pixels to meet the YOLO network input requirements, the pixel ratio of the target in the image increases. It can highlight the target features and make it easier for the algorithm to perform effective detection.

[0145] The present invention proposes a topological bidirectional fusion detection algorithm for radar cameras of unmanned boats at sea, which includes the following technical measures: first, the time alignment of the sensor system is performed to solve the time synchronization problem and obtain the optical image and radar detection point at the same time. After the camera image is detected by YOLOV7-tiny, the center coordinates of the detection frame and the target category are output. According to the maximum detection distance of the camera, the radar points within the range are projected to the optical image through matrix transformation to ensure that the detection results of the sensor system are all located in the pixel coordinate system, so as to realize the system spatial alignment. PROSAC straight line fitting is performed on the projected radar points, and the radar fitting line is deflected according to the hull attitude angle to reduce the influence of the hull shaking. The coarse correlation gate is set by using the system error of the sensor and the position error loss of the YOLO algorithm to screen the radar points and reduce the amount of calculation. Then the topological fusion index is set to perform radar and camera topological fusion. Finally, the unassociated radar points and the detection frame are bidirectionally fused.

Claims

1. A method for topological bidirectional fusion detection of radar cameras of unmanned marine boats, characterized in that: The following steps are involved: Step 1) Time alignment between radar and camera; Step 2) Spatial registration of radar and camera: perform external and internal calibration, obtain the geometric parameters of radar and camera, establish the spatial transformation relationship between them, find the correspondence between radar data and camera data, and realize spatial registration; Step 3) PROSAC algorithm straight line fitting: Use the algorithm to construct a straight line model for the radar data projection points, and adjust the obtained straight line model through the synchronously acquired unmanned boat attitude information; Step 4) Coarse correlation gate establishment: According to the distance and azimuth error of the sensor system and the boundary box center position loss error of the YOLO algorithm, a coarse correlation gate threshold is established to roughly screen the radar points; Step 5) Topological fusion similarity index establishment: Use the edge length similarity, polygon angle similarity and the line similarity between polygon center points to establish a comprehensive fusion similarity index; Step 6) Matrix establishment and radar and camera matching: Establish a fusion matrix and complete the matching of the radar projection point and the center point of the camera detection frame according to the threshold; Step 7) Bidirectional fusion of unfused radar and camera data: perform detection fusion on the image capture near the unfused radar point; for the unfused optical detection frame, select the nearest neighbor radar point of the azimuth line for fusion to achieve bidirectional fusion.

2. A method for detecting a topological bidirectional fusion of a radar camera at sea as claimed in claim 1, characterized in that The step 1 adopts the "interpolation / extrapolation method" to perform time registration, which specifically includes the following sub-steps: Step 1.1: If the time difference between the interpolation point and the last radar sampling is less than the sampling interval, use the last measurement data for extrapolation; Step 1.2: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is greater than the sampling interval, use the last measurement data closest to the interpolation point for extrapolation; Step 1.3: If the time difference between the interpolation point and the last radar sampling time involved in the judgment is equal to the sampling interval, keep the radar data at that moment unchanged; Step 1.4: Process the radar and camera data as described above to align them at the same time point to achieve temporal registration.

3. A method for detecting a topological bidirectional fusion of a radar camera on a marine unmanned boat as claimed in claim 1, characterized in that The step 2 specifically includes the following sub-steps: Step 2.1: Perform external and internal calibration, obtain the geometric parameters of the radar and camera, establish the spatial transformation relationship between them, and then find the correspondence between the radar data and the camera data through the extraction and matching of feature points or feature areas to achieve spatial registration: Formula (27) is the coordinate transformation matrix, (X cam ,Y cam ,Z cam ) is the coordinate in the photoelectric coordinate system, (X world ,Y world ,Z world ) is the coordinate in the world coordinate system, R is the rotation matrix, T is the translation matrix, f cam is the photoelectric focal length, (x, y) is the coordinate in the image coordinate system, (u, v) is the pixel coordinate, and the radar coordinate system is set to the world coordinate system.

4. A method for detecting a topological bidirectional fusion of a radar camera on a marine unmanned boat as claimed in claim 1, characterized in that The step 3 specifically includes the following sub-steps: Step 3.1: The set of N radar projection points is First, calculate the quality score of each point in the point set. The calculation method is: traverse each point, calculate the square of its distance to all other points, and then find the inverse sum of these squared distances as the quality score of the point; Projection Point The quality score is defined as: Step 3.2: Assemble the collections according to their quality scores Sort descending as: Ensure that high-quality points are selected as candidate points first. The candidate point set is defined as The initial number of candidate point sets is n = 1, and increases with the number of iterations t until it reaches the value of the number of iterations t or The number of points is N. In the tth iteration, the first n ordered points are used as candidate points to generate the model and find the inner points, and the first two points are selected to calculate the equation of the line; Step 3.3: Calculate the distance from all points to the current line, and determine which points are inliers based on the threshold υ. If the number of inliers found exceeds the current best result, update the best line and inlier set. Assuming that the number of inliers is η and P is the prior probability, the number of iterations t is defined as: After the straight line is fitted, the straight line is deflected according to the hull attitude angle.

5. A method for detecting a topological bidirectional fusion of a radar camera at sea as claimed in claim 1, characterized in that The step 4 specifically includes the following sub-steps: Step 4.1: Center position loss threshold: According to the loss of the center position of the bounding box of the YOLO algorithm, the loss threshold λ is generated loss : Among them, λ coord Represents the corresponding weight term, set λ coord =1, s 2 The s×s grids need to be calculated, B represents the number of box classes, Indicates whether the jth box of the i-th grid predicts the target. If so, otherwise λ loss As the center position loss threshold, it is used to form a coarse correlation wave gate to screen radar points; Step 4.2: System Error Threshold The track point distance R between radar α and camera β reporting the same target (target i for radar detection and target j for camera detection) at time k ij (k) can be expressed as follows after derivation: Δr α ,Δr β is the position error between the radar and the camera, Δθ α ,Δθ β is the angle error between the radar and the camera, and the other values ​​are fixed values. According to formula (31), if R ij (k) takes the maximum value R max , Δθ α ,Δθ β ,Δr α ,Δr β The value should be: Dth α =Δθ αmax Dth β =Δθ βmax Δr α =Δr αmax Δr β =-Δr βmax (6) The maximum value R of the track point spacing is used max As the system error threshold, the above calculation is performed for all targets at time k, and the maximum value is taken as the system error threshold at that moment; Step 4.3: Coarse correlation gate generation The system error threshold is located in the radar coordinate system (X world ,Y world ,Z world ), the center position loss threshold is located in the pixel coordinate system (u, v), so the system error threshold R max Convert to the pixel coordinate system through formula (27), and define the system error threshold in the pixel coordinate system as: R uv max ; The coarse correlation gate size is defined as: L max =s*(R uv max +λ loss ) (7) s is the proportional factor, which can be adjusted according to the actual situation. If the coarse correlation wave gate is too large, it will increase unnecessary calculation amount; if the coarse correlation wave gate is too small, the correct radar points will be screened out. Set s = 2; Rough correlation gate L max As the radius, a circle is drawn with the center point of each YOLO detection frame as the center to generate a coarse correlation gate. For each YOLO detection frame, only the radar points within the gate are fused, and the radar points outside the gate are excluded.

6. A method for detecting a topological bidirectional fusion of a radar camera at sea as claimed in claim 1, characterized in that The step 5 specifically includes the following sub-steps: Step 5.1: Edge length similarity index The evaluation index is constructed based on the similarity of the lengths of the three sides of the triangle, which is defined as: represents the edge length similarity at time t, represents the i-th side of radar triangle α, represents the i-th edge of the camera triangle β; Step 5.2: Polygon Angle Similarity Metric Polygon angle similarity is defined as: represents the similarity of polygon angles at time t, represents the i-th interior angle of radar polygon α, represents the i-th internal angle of the camera polygon β, and n1 is the number of internal angles of the polygon; Step 5.3: Center point connection similarity index Connect the center point of radar polygon α and the i-th vertex to obtain the center point line L of radar polygon α α , connect the center point of the camera polygon β and the i-th vertex to get the center point line L of the camera polygon β β , the center point connection similarity is defined as: Represents the similarity of the center point connection at time t, and is the i-th L α The angle and length of and is the i-th L β The angle and length of Step 5.4: Comprehensive similarity index The comprehensive topological similarity model ω at time t is established by logarithmically transforming the three indicators. t :

7. A method for detecting a topological bidirectional fusion of a radar camera at sea as claimed in claim 1, characterized in that The step 6 specifically includes the following sub-steps: Step 6.1: Traverse all topological structures, assuming that the number of radar targets is m and the number of camera targets is n, generate the radar and camera fusion matrix F. In F, ζ nm represents the similarity between the radar projection point m and the camera detection frame center point n. If the radar projection point m and the camera detection frame center point n lack correlation, the linkage between them is not considered. nm = 0; if there are multiple fusion evaluations between the radar projection point m and the camera detection frame center point n, the topological structure with the highest similarity among these fusion evaluations is selected to ensure the accuracy of the data and the effectiveness of the system. The similarity is ζ nm =max(ζ nm1 ,ζ nm2 ,…); Set the topological similarity threshold to ω t = 0.5, for the obtained fusion matrix F, find the maximum value of all elements in F, assuming it is ζ ijmax , and then return the remaining elements of the matrix in the i-th row and j-th column to 0, and repeat the above process until the maximum value ζ in F is reached ijmax ≤0.5, complete the matching between the radar projection point and the center point of the camera detection frame.

8. A method for detecting a topological bidirectional fusion of a radar camera at sea as claimed in claim 1, characterized in that The step 7 specifically includes the following sub-steps: Step 7.1: Unfused YOLO detection box For the target frame that is not fused in the camera detection, a method of auxiliary fusion using radar data is proposed. The corresponding radar map azimuth line is obtained according to the azimuth information of the unfused camera detection frame. Then, a search operation is performed on the azimuth line to identify the radar detection point closest to the azimuth line. Through this method, the corresponding radar target point can be found for the target frame that was not correctly fused in the camera system detection, thereby realizing the effective fusion of the two sensor data. φ r is the radar chart azimuth, φ g is the camera azimuth, φ s is the camera field of view, u is the horizontal coordinate of the center point of the unfused camera detection frame, and u z is the lateral resolution of the camera image, so the radar image azimuth angle formula is: According to the azimuth information of the radar map determined by formula (39), the corresponding azimuth line is drawn on the radar map, and the nearest radar detection point is located through this line. In this process, the identified radar point is fused with the specific detection frame to achieve more accurate target recognition and positioning; Step 7.2: Unfused radar points For unfused radar target points, the image interception technology centered on the radar detection point is used to further analyze the surrounding image information and solve the problem of unfused data; For the target points identified by the radar, we first determine whether there is a detection frame in the coarse correlation gate. If not, we extract the local area centered on these points from the original image. These local area images are usually small. When this small-size image is enlarged to 640×640 pixels to meet the input requirements of the YOLO network, the proportion of pixels occupied by the target in the image increases, which can highlight the target features and make it easier for the algorithm to perform effective detection.

Images