A method and system for surface unmanned vehicle environment perception based on vision and millimeter-wave radar fusion under weak observation conditions

Abstract

The present invention belongs to the field of unmanned boat target detection and discloses a surface unmanned boat environment perception method and system based on the fusion of vision and millimeter-wave radar under weak observation conditions. The method uses a visual camera and millimeter-wave radar to collect water surface environment data around the unmanned boat, and uses an inertial measurement unit to collect the unmanned boat's own posture information; clusters the millimeter-wave radar point cloud data, filters out clutter and identifies targets, and simultaneously projects the identified targets into an image coordinate system in combination with the unmanned boat's posture information measured by the inertial measurement unit; detects multi-scale targets in the surface environment through a deep neural network framework, fuses the millimeter-wave radar projected targets, the deep target detection network detection results, and the anchor frame information before non-maximum suppression processing, and outputs the final perception result. The present invention can effectively reduce the adverse effects of incomplete and uncertain sensor information in weak observation environments, providing important information for subsequent autonomous decision-making.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned boat target detection, and specifically relates to a surface unmanned boat environment perception method and system based on the fusion of vision and millimeter-wave radar under weak observation conditions. Background Art

[0002] In recent years, unmanned boats have been widely used to perform various military and non-military tasks, replacing operators to perform dangerous or time-consuming and labor-intensive tasks. They play an important role in more and more aspects, improving work efficiency while reducing casualties among operators.

[0003] For an unmanned aerial vehicle (UAV) to achieve effective fully autonomous or semi-autonomous operation, accurate perception of its surroundings is paramount (identifying static and dynamic obstacles, their types, speed, and heading). Single-sensor perception methods have limitations and are significantly affected by the environment. Multiple sensors of the same or different types can obtain information of varying locality and type, providing a more accurate and comprehensive understanding of the target's direction, environment, and location. Multi-sensor fusion makes the UAV more robust to changes in its operating environment.

[0004] Visual cameras provide rich imaging details and can identify target features, but they lack depth information, have a limited field of view, and are significantly affected by environmental factors and lighting. Millimeter-wave radar, on the other hand, can measure speed and distance, has a long range, is less susceptible to environmental factors, and offers a high cost-effectiveness. The fusion of visual cameras and millimeter-wave radar offers excellent complementary advantages. This fusion can effectively reduce missed detections in low-light conditions and significantly improve the accuracy of long-range small target detection.

[0005] Currently, the high cost of sensor deployment is a challenge for fusion perception in unmanned vehicles. Existing technologies mostly use multiple sensors to achieve a wider sensing range, or expensive lidar to enhance close-range perception. Patent application CN105741234B discloses a visual assistance system for automatic anchoring of unmanned vehicles based on three-dimensional panoramic vision. This system uses four visual cameras and four millimeter-wave radars for environmental perception, providing a wide field of view for automatic anchoring. Patent application CN109444911A discloses a method for detecting, identifying, and localizing surface targets for unmanned vehicles that fuses information from a monocular camera and lidar. This fusion of a visual camera and lidar point cloud improves surface target detection and recognition capabilities, providing the unmanned vehicle with a sound perception foundation for target tracking, path planning, and autonomous navigation. Patent application CN115202366A discloses an autonomous berthing method and system for unmanned vehicles based on environmental perception. This system uses lidar and visual cameras to identify the surrounding environment and assist the unmanned vehicle in berthing operations. Furthermore, existing fusion solutions are mostly based on post-fusion, where the perception of a single sensor limits the overall perception of the fusion system. Summary of the Invention

[0006] The present invention uses a matching visual camera and millimeter-wave radar, and adopts a multi-level fusion solution of front fusion and intermediate fusion. Its purpose is to provide a cost-effective and accurate surface unmanned boat environment perception method and system based on the fusion of vision and millimeter-wave radar under weak observation conditions.

[0007] Based on the above purpose, the present invention adopts the following technical solutions:

[0008] A method for surface unmanned vehicle environment perception based on the fusion of vision and millimeter-wave radar under weak observation conditions includes the following steps:

[0009] Step 1: Install the millimeter-wave radar and the visual camera on the bow of the unmanned boat, and record the position offset and angle offset between the millimeter-wave radar and the visual camera;

[0010] Step 2: At the same time, use the visual camera to obtain camera images, use the millimeter-wave radar to obtain point cloud data, and use the inertial measurement unit to obtain the pitch angle of the unmanned boat;

[0011] Step 3: Use the improved K-Means clustering algorithm to cluster the point cloud data to obtain the point cloud detection target set, and then calculate the length l, width w, center point coordinates (x co ,y co ), rcs o 、v xo 、v yo and center distance; for all points in the point cloud detection target, the radar cross-sectional area and radial velocity are averaged to obtain the velocity v of the point cloud detection target xo 、v yo and radar cross section rcs o .

[0012] Step 4: Project the point cloud detection target into the coordinate system of the camera image to obtain a projected target set;

[0013] In step 5, the camera image is placed into the deep object detection network for object recognition to obtain an image detection target set and a complete anchor box set. The complete anchor box set is the anchor box recognition information before non-maximum suppression processing; the image detection target set includes the category of each image detection target, the coordinates of the upper left and lower right corners of the bounding box, and the confidence; the complete anchor box set includes the confidence of all categories of each anchor box and the coordinates of the upper left and lower right corners of the bounding box.

[0014] Step 6: Fuse the projection target set with the image detection target set to obtain the initial residual projection target set, the residual image detection target set and the first fusion target set. Then fuse the initial residual projection target set with the complete anchor frame set to obtain the secondary residual projection target set and the secondary fusion target set. Finally, the first fusion target set, the secondary fusion target set, the secondary residual projection target set and the residual image detection target set form the final target set.

[0015] Furthermore, in step 6, the method for fusing the projection target set with the image detection target set is: traverse the projection target set, and for each projection target in the projection target set, find the overlap between each image detection target and the current projection target. If there is an image detection target with an overlap greater than 0.5, output a fused target. The bounding box and category of the fused target are the bounding box and category of the image detection target with the largest overlap. The Euclidean distance of the fused target is the Euclidean distance calculated based on the center point coordinates of the current projection target. Finally, the image detection target with the largest overlap and the current projection target are deleted from the image detection target set and the projection target set respectively. All fused targets constitute a fused target set. The traversed projection target set and image detection target set are the initial remaining projection target set and the remaining image detection target set, respectively.

[0016] Furthermore, in step 6, the method of fusing the initial remaining projection target set with the complete anchor frame set is as follows: traverse the initial remaining projection target set, for each projection target in the initial remaining projection target set, find the overlap between each anchor frame in the complete anchor frame set and the current projection target, and select anchor frames with an overlap greater than 0.5 to form a fused anchor frame set, and then multiply the confidence of each anchor frame in the fused anchor frame set by the coefficient ε to obtain the amplified confidence, ε = 2.8*rcs o , rcs o is the radar cross-sectional area of the projected target; if there is an anchor frame with a magnification confidence greater than or equal to 0.5, the anchor frame with a magnification confidence less than 0.5 in the fused anchor frame set is deleted, and the secondary fusion target is output. The category of the secondary fusion target is the category of the anchor frame with the largest magnification confidence in the fused anchor frame set and the category with the maximum output value. The bounding box of the secondary fusion target is the final bounding box obtained by correcting the bounding box of the current projected target using the anchor frame in the fused anchor frame set. The Euclidean distance of the secondary fusion target is the Euclidean distance calculated according to the center point coordinates of the current projected target; finally, the current projected target is deleted from the initial projected target set; the set of all secondary fusion targets constitutes the secondary fusion target set, and the initial remaining projected target set after traversal is the secondary remaining projected target set.

[0017] Furthermore, in step 6, the method for obtaining the final bounding box is:

[0018]

[0019] Where B is the coordinate of the upper left corner and lower right corner of the final bounding box, B 雷达投影 are the upper left and lower right coordinates of the current projection target, is the mean of the upper left corner and lower right corner coordinates of all anchor boxes in the fusion anchor box set, N is the number of anchor boxes in the fusion anchor box set, B n Confidence is the coordinate of the upper left corner and lower right corner of each anchor box in the fusion anchor box set. n is the confidence of each anchor box in the fused anchor box set.

[0020] Furthermore, in step 4, the method of projecting the point cloud detection target into the coordinate system of the camera image includes the following steps:

[0021] The first step is to transfer the center point coordinates of the point cloud detection target to the world coordinate system with the camera center as the origin, calculate the center point coordinates of the point cloud detection target in the world coordinate system, and then calculate the height of the center point projection;

[0022] The second step is to transfer the corrected center point coordinates to the image coordinate system and calculate the center point coordinates of the point cloud detection target in the image coordinate system;

[0023] The third step is to calculate the coordinates of the upper left corner and lower right corner of the target box of the point cloud detection target in the image coordinate system;

[0024] In the fourth step, the center point coordinates, lower left corner coordinates, upper right corner coordinates, speed and radar cross-sectional area information of the point cloud detection target in the image coordinate system are combined to form a projection target. All projection targets form a projection target set.

[0025] Furthermore, in step 4, the center point coordinates (x w ,y w ) is as follows: Where θ is the offset angle between the millimeter-wave radar and the visual camera, Where x is the horizontal lateral offset of the millimeter-wave radar and the vision camera, and y is the horizontal longitudinal offset of the millimeter-wave radar and the vision camera.

[0026] Furthermore, in step 4, the height of the center point projection h is the camera installation height.

[0027] Furthermore, in step 4, the method for calculating the center point coordinates (u, v) of the point cloud detection target in the image coordinate system is: f x and f yis the focal length of the visual camera in the x and y directions, c x and c y are the distortion parameters of the visual camera in the x and y directions.

[0028] Furthermore, in step 4, the coordinates of the upper left corner of the target box of the point cloud detection target in the image coordinate system (u lefttop , v lefttop ) and the lower right corner coordinate (u rightbottom , v rightbottom ) is as follows: a and b are the highest and lowest points of the point cloud detection target.

[0029] A surface unmanned boat environment perception system based on the fusion of vision and millimeter-wave radar under weak observation conditions includes an unmanned boat, a visual camera and a millimeter-wave radar are installed on the bow of the unmanned boat, an inertial measurement unit is installed in the cabin of the unmanned boat, the visual camera is used to obtain images, the millimeter-wave radar is used to obtain point cloud data, and the inertial measurement unit is used to obtain the heading angle, roll angle and pitch angle of the unmanned boat; an industrial computer is installed in the cabin of the unmanned boat, and the industrial computer is connected to the visual camera, millimeter-wave radar and inertial measurement unit; the industrial computer includes a point cloud processing module, an image processing module, a primary target fusion module, a secondary target fusion module and a final target output module; the point cloud processing module The module includes a clustering algorithm unit and a coordinate projection unit; the clustering algorithm unit is used to cluster the point cloud data and obtain a point cloud detection target set composed of point cloud detection targets; the coordinate projection unit is used to project the point cloud detection targets into the image coordinate system and obtain a projection target set composed of projection targets; the image processing module includes a target recognition unit and an anchor frame storage unit, the image detection unit is used to perform target recognition on the image and obtain an image detection target set composed of image detection targets; the anchor frame storage unit is used to save the anchor frame before non-maximum suppression processing and obtain a complete anchor frame set composed of anchor frames; the one-time target fusion module includes an image overlap calculation unit, a one-time fusion output unit and a one-time fusion output unit. Secondary target deletion unit; image overlap calculation unit for calculating the overlap between each projection target and all image detection targets; a primary fusion output unit for fusing the image detection target with the overlap greater than 0.5 and the largest overlap with the projection target to obtain a primary fusion target; a primary target deletion unit for deleting the fused projection target and image detection target from the projection target set and the image detection target set to obtain the initial remaining projection target set and the remaining image detection target set; a final target output module for outputting the final target set and adding the primary fusion target and the remaining image detection targets to the final target set; a secondary target fusion module including an anchor frame overlap calculation unit , anchor frame screening unit, secondary fusion output unit and secondary target deletion unit; the anchor frame overlap calculation unit is used to calculate the overlap of each projection target in the initial residual projection target set and all anchor frames in the complete anchor frame set; the anchor frame screening unit is used to screen out anchor frames with an overlap greater than 0.5 with the projection target, and form a fused anchor frame set; the secondary fusion output unit is used to fuse the fused anchor frame set with the projection target and obtain a secondary fused target; the secondary target deletion unit is used to delete the projection target fused with the fused anchor frame set and obtain a secondary residual projection target set; the final target output module is used to add the secondary fusion target and the secondary residual projection target set to the final target set.

[0030] Compared with the prior art, this application has the following beneficial effects:

[0031] 1. This application simultaneously obtains millimeter-wave radar data, visual camera image data, and inertial measurement unit data to ensure data synchronization between the three sensors, facilitating subsequent processing. Accessing the inertial measurement unit data ensures that the position of the point cloud detection target projected into the image coordinate system is more accurate, reducing the impact of the shaking of the ship caused by water flow, wind, and waves on the projection results.

[0032] 2. Use the improved K-Means clustering algorithm to obtain the point cloud detection target and filter out the clutter caused by environmental problems. Continue to process the point cloud of each point cloud detection target to obtain the size, speed, and RCS of the point cloud detection target. O The data generated by the millimeter radar is fully utilized to better complement the image data generated by the visual camera to achieve better target detection results.

[0033] 3. The point cloud detection target information is appended to the camera image, which increases the richness of the camera image information and makes the unmanned boat target detection more robust.

[0034] 4. Obtain the recognition results of the deep neural network while preserving the anchor frame recognition information before non-maximum suppression to avoid information loss from non-maximum suppression. This information is then integrated with the target identified by the millimeter-wave radar to reduce the uncertainty of the target recognition estimation and enhance robustness. The multi-level combination of millimeter-wave radar information and camera image information, fully utilizing the information from both sensors, can better identify water targets and ensure the navigation safety of the unmanned boat. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the installation positions of the millimeter wave radar and visual camera according to Example 1 of the present invention;

[0036] Figure 2 This is a flow chart of Example 1 of the present invention;

[0037] Figure 3 The camera image of embodiment 1 of the present invention;

[0038] Figure 4 Schematic diagram of projection target and anchor frame information according to embodiment 1 of the present invention;

[0039] Figure 5 This is a schematic diagram of the fusion target of Example 1 of the present invention. DETAILED DESCRIPTION

[0040] Example 1

[0041] A surface unmanned vehicle environment perception method based on the fusion of vision and millimeter-wave radar under weak observation conditions, such as Figure 2 As shown, the following steps are included:

[0042] Step 1: Install the millimeter wave radar and visual camera on the bracket at the bow of the unmanned boat, as shown in the following figure: Figure 1 As shown, the millimeter-wave radar is installed in front of the bracket, and the visual camera is installed just above it. Both the millimeter-wave radar and the visual camera are facing forward, so that the center of the field of view of the visual camera and the millimeter-wave radar are consistent, so as to better complement the information between the two sensors. The inertial measurement unit equipment is installed in the cabin and kept parallel to the shipboard to obtain information on the three shaking angles of the hull. An industrial computer is installed in the cabin of the unmanned boat, and the visual camera is connected to the industrial computer through the serial port of the acquisition card, the millimeter-wave radar is connected to the industrial computer through the CAN card, and the inertial measurement unit is connected to the industrial computer through the serial port. Record the position offset between the millimeter-wave radar and the visual camera Where x is the horizontal offset, y is the horizontal offset, and θ is the angular offset. Since both the visual camera and the millimeter-wave radar are facing straight ahead, their angular offset θ is 0. Mounting the millimeter-wave radar and visual camera on the same bracket at the bow of the unmanned boat aligns their fields of view, further unifying their information. Incorporating inertial measurement unit data allows for more accurate projection of targets detected by the millimeter-wave radar into the camera's image coordinate system.

[0043] Step 2, such as Figure 3 As shown, the camera image IMage is obtained using the visual camera at time t t , using millimeter wave radar to obtain point cloud data PC t , use the inertial measurement unit to obtain data IMU t . n is the number of points in the point cloud, and each point has ten data: including coordinates (x r ,y r ), radar cross section rcs, radial velocity v x ,v y , motion state dynamic_property, target category class_type, possibility of existence prob_of_exist, fuzzy state ambig_state, invalid state invalid_state. IMage t It is a three-channel RGB image data with a resolution of 1920*1280, IMU t Information from three perspectives From top to bottom are the heading angle, roll angle and pitch angle of the unmanned boat at time t.

[0044] Step 3: Use the improved K-Means clustering algorithm to cluster the point cloud data PC t Perform clustering to obtain the point cloud detection target set O wwv , O wwv Detecting the target o from the point cloudwwv Then use the tightest rectangle to frame O wwv Each o wwv . Calculate each o wwv The length l, width w, center point coordinates (x co ,y co ), rcs o 、v xo 、v yo And according to (x co ,y co ) calculated Euclidean distance; for o wwv The radar cross section rcs of all points in the x ,v y Find the average value to get o wwv Radar cross section rcs o and speed v xo 、v yo .

[0045] Step 4: Set all o wwv Projecting into the coordinate system of the camera image to obtain the projection target set includes the following four steps:

[0046] The first step is to get O in step 3. wwv Each o wwv , will o wwv The center point coordinates are transferred to the world coordinate system with the camera center as the origin: according to the mapping relationship Calculate wwv The center point coordinates in the world coordinate system (x w ,y w ), where the offset angle θ between the millimeter-wave radar and the visual camera is 0, so the mapping is a translation transformation. Then use the IMU t The pitch angle information pitch correction projection is used to calculate the height of the center point projection h is the camera installation height (relative to the horizontal plane).

[0047] The second step is to transfer the corrected center point coordinates to the image coordinate system according to the camera's imaging matrix. Calculate wwv The center point in the image coordinate system f x and f y is the focal length of the visual camera in the x and y directions, c x and c y are the distortion parameters of the visual camera in the x and y directions.

[0048] The third step is to calculate o wwvThe coordinates of the upper left corner of the target box in the image coordinate system (u lefttop , v lefttop ) and the lower right corner coordinate u rightbottom , v rightbottom ); let o wwv The highest point is 2m and the lowest point is 0m, then

[0049] The fourth step is to convert (u,v), (u lefttop , v lefttop )、(u rightbottom , v rightbottom ),o wwv The speed v xo , v yo , radar cross section rcs o The center point coordinates (x co ,y co ) information combination forms the projection target w, in O wwv Each o wwv After all projection targets w are formed, all projection targets w form a projection target set W.

[0050] Step 5: IMage the camera image obtained in step 2 t Put it into the deep target detection network image target detection network for target recognition, save its recognition results, and obtain the image detection target set D. The image detection target set D consists of each image detection target d, which includes the category of each image detection target, the image coordinate system coordinates of the upper left and lower right corners of the bounding box, and the confidence of the network output. At the same time, save the anchor box recognition information of the deep target detection network image target detection network before non-maximum suppression processing to obtain a complete anchor box set A. The complete anchor box set A consists of each anchor box a, which includes the confidence of all categories of the anchor box and the image coordinate system coordinates of the upper left and lower right corners of the bounding box. Figure 4 As shown, green is the projected target and red is the anchor box information filtered out by the deep object detection network anchor box detection.

[0051] Step 6, traverse the projection target set W: For each projection target w in the projection target set W, find the overlap between each image detection target and the current projection target. If there is an image detection target with an overlap greater than 0.5, find the image detection target d with the largest overlap with the current projection target. ioumax , and output a fusion target; the bounding box and category of the fusion target are d ioumax The bounding box and category of the fusion target are calculated based on the Euclidean distance of the current projection target. wwv The center point coordinates (x co ,yco ) calculates the Euclidean distance; finally, d ioumax The current projection target is deleted from the image detection target set and the projection target set respectively; if there is no image detection target with an overlap greater than 0.5, the above operation is continued for the next projection target. All the fused targets finally obtained constitute a fused target set. The projection target set and image detection target set after traversal are the initial remaining projection target set and the remaining image detection target set respectively.

[0052] Then traverse the initial remaining projection target set: for each projection target in the initial remaining projection target set, find the overlap between each anchor frame in the complete anchor frame set and the current projection target, and select the anchor frames with an overlap IOU greater than 0.5 to form the fused anchor frame set A cross , then multiply the confidence of each anchor box in the fusion anchor box set by the coefficient ε to obtain the amplified confidence, the default ε=2.8*rcs o , rcs o is the radar cross-section of the current projected target; if there is an anchor box with a magnification confidence greater than or equal to 0.5, select A cross The anchor box a with the highest confidence cross-max , delete the anchor boxes with amplification confidence less than 0.5 in the fusion anchor box set, and output the secondary fusion target as Figure 5 , the category of the secondary fusion target is a cross-max The category outputs the maximum value of the category, and the bounding box of the secondary fusion target is A cross The final bounding box B is obtained by correcting the bounding box of the current projection target with the anchor box in Where B is the coordinate of the upper left corner and lower right corner of the final bounding box, B 雷达投影 are the coordinates of the upper left and lower right corners of the current projection target, To delete the anchor box A after enlarging the confidence level less than 0.5 cross The mean of the upper left and lower right corner coordinates of all anchor boxes in , N is A after deleting the anchor boxes with amplification confidence less than 0.5 cross The number of anchor boxes in B n To delete the anchor box A after enlarging the confidence level less than 0.5 cross The coordinates of the upper left and lower right corners of each anchor box in confidence n To delete the anchor box A after enlarging the confidence level less than 0.5 cross The confidence of each anchor box in the quadratic fusion target is the Euclidean distance corresponding to the current projection target. wwv The center point coordinates (x co ,y co) calculates the Euclidean distance; finally, the current projected target is deleted from the initial projected target set; if there is no anchor box with a magnified confidence level greater than or equal to 0.5, the above operation is repeated for the next projected target. The final set of all secondary fusion targets constitutes the secondary fusion target set, and the set of initial remaining projected targets after traversal is the secondary remaining projected target set.

[0053] Finally, the first fusion target set and the second fusion target set are combined into the final target set O; the remaining w in the second residual projection target set is retained in the pixel coordinate system, and the o corresponding to w is wwv The center point coordinates (x co ,y co ) is added to the final target set O; the remaining d retained output categories and pixel coordinate bounding boxes in the remaining image detection target set are added to the final output target set O, and the final target set O is output. This application processes point cloud data and camera images to preserve anchor box recognition information before non-maximum suppression, obtain more features, and make it easier to find targets.

[0054] Example 2

[0055] A surface unmanned boat environment perception system based on the fusion of vision and millimeter-wave radar under weak observation conditions includes an unmanned boat, a visual camera and a millimeter-wave radar are installed on the bow of the unmanned boat, an inertial measurement unit is installed in the cabin of the unmanned boat, the visual camera is used to obtain images, the millimeter-wave radar is used to obtain point cloud data, and the inertial measurement unit is used to obtain the heading angle, roll angle and pitch angle of the unmanned boat; an industrial computer is installed in the cabin of the unmanned boat, and the industrial computer is connected to the visual camera, millimeter-wave radar and inertial measurement unit; the industrial computer includes a point cloud processing module, an image processing module, a primary target fusion module, a secondary target fusion module and a final target output module; the point cloud processing module The module includes a clustering algorithm unit and a coordinate projection unit; the clustering algorithm unit is used to cluster the point cloud data and obtain a point cloud detection target set composed of point cloud detection targets; the coordinate projection unit is used to project the point cloud detection targets into the image coordinate system and obtain a projection target set composed of projection targets; the image processing module includes a target recognition unit and an anchor frame storage unit, the image detection unit is used to perform target recognition on the image and obtain an image detection target set composed of image detection targets; the anchor frame storage unit is used to save the anchor frame before non-maximum suppression processing and obtain a complete anchor frame set composed of anchor frames; the one-time target fusion module includes an image overlap calculation unit, a one-time fusion output unit and a one-time fusion output unit. Secondary target deletion unit; image overlap calculation unit for calculating the overlap between each projection target and all image detection targets; a primary fusion output unit for fusing the image detection target with the overlap greater than 0.5 and the largest overlap with the projection target to obtain a primary fusion target; a primary target deletion unit for deleting the fused projection target and image detection target from the projection target set and the image detection target set to obtain the initial remaining projection target set and the remaining image detection target set; a final target output module for outputting the final target set and adding the primary fusion target and the remaining image detection targets to the final target set; a secondary target fusion module including an anchor frame overlap calculation unit , anchor frame screening unit, secondary fusion output unit and secondary target deletion unit; the anchor frame overlap calculation unit is used to calculate the overlap of each projection target in the initial residual projection target set and all anchor frames in the complete anchor frame set; the anchor frame screening unit is used to screen out anchor frames with an overlap greater than 0.5 with the projection target, and form a fused anchor frame set; the secondary fusion output unit is used to fuse the fused anchor frame set with the projection target and obtain a secondary fused target; the secondary target deletion unit is used to delete the projection target fused with the fused anchor frame set and obtain a secondary residual projection target set; the final target output module is used to add the secondary fusion target and the secondary residual projection target set to the final target set.

Claims

1. A method for surface unmanned vehicle environment perception based on the fusion of vision and millimeter-wave radar under weak observation conditions, characterized by: The following steps are involved: Step 1: Install the millimeter-wave radar and the visual camera on the bow of the unmanned boat, and record the position offset and angle offset between the millimeter-wave radar and the visual camera; Step 2: At the same time, use the visual camera to obtain camera images, use the millimeter-wave radar to obtain point cloud data, and use the inertial measurement unit to obtain the pitch angle of the unmanned boat; Step 3: Cluster the point cloud data using a clustering algorithm to obtain a point cloud detection target set; Step 4: Project the point cloud detection target into the coordinate system of the camera image to obtain a projected target set; Step 5: Put the camera image into the deep object detection network for target recognition, and obtain the image detection target set and the complete anchor frame set. The complete anchor frame set is the anchor frame recognition information before the deep object detection network performs non-maximum suppression processing; Step 6: Fuse the projection target set with the image detection target set to obtain the initial residual projection target set, the residual image detection target set and the first fusion target set. Then fuse the initial residual projection target set with the complete anchor frame set to obtain the secondary residual projection target set and the secondary fusion target set. Finally, the first fusion target set, the secondary fusion target set, the secondary residual projection target set and the residual image detection target set form the final target set.

2. The method according to claim 1, wherein In step 6, the method for fusing the projection target set with the image detection target set is as follows: traverse the projection target set, and for each projection target in the projection target set, find the overlap between each image detection target and the current projection target. If there is an image detection target with an overlap greater than 0.5, output a fused target. The bounding box and category of the fused target are the bounding box and category of the image detection target with the largest overlap. The Euclidean distance of the fused target is the Euclidean distance calculated based on the center point coordinates of the current projection target. Finally, the image detection target with the largest overlap and the current projection target are deleted from the image detection target set and the projection target set respectively. All fused targets constitute a fused target set. The traversed projection target set and image detection target set are the initial remaining projection target set and the remaining image detection target set, respectively.

3. The method according to claim 2, wherein In step 6, the method of fusing the initial remaining projection target set with the complete anchor frame set is as follows: traverse the initial remaining projection target set, for each projection target in the initial remaining projection target set, find the overlap between each anchor frame in the complete anchor frame set and the current projection target, and select the anchor frames with an overlap greater than 0.5 to form a fused anchor frame set, and then multiply the confidence of each anchor frame in the fused anchor frame set by the coefficient ε to obtain the amplified confidence, ε = 2.8*rcs o , rcs o is the radar cross-sectional area of the projected target; if there is an anchor frame with a magnification confidence greater than or equal to 0.5, then delete the anchor frame with a magnification confidence less than 0.5 in the fusion anchor frame set, and output the secondary fusion target. The category of the secondary fusion target is the category of the anchor frame with the largest magnification confidence in the fusion anchor frame set and the category with the maximum output value. The bounding box of the secondary fusion target is the final bounding box obtained by correcting the bounding box of the current projected target using the anchor frame in the fusion anchor frame set. The Euclidean distance of the secondary fusion target is the Euclidean distance calculated according to the center point coordinates of the current projected target; finally, delete the current projected target from the initial projection target set; The set of all secondary fusion targets constitutes the secondary fusion target set, and the initial residual projection target set after traversal is the secondary residual projection target set.

4. The method according to claim 3, wherein In step 6, the method to obtain the final bounding box is: Where B is the coordinate of the upper left corner and lower right corner of the final bounding box, B 雷达投影 are the coordinates of the upper left and lower right corners of the projection target, is the mean of the upper left corner and lower right corner coordinates of all anchor boxes in the fusion anchor box set, N is the number of anchor boxes in the fusion anchor box set, B n Confidence is the coordinate of the upper left corner and lower right corner of each anchor box in the fusion anchor box set. n is the confidence of each anchor box in the fused anchor box set.

5. The method according to claim 4, wherein In step 4, the method of projecting the point cloud detection target into the coordinate system of the camera image includes the following steps: The first step is to transfer the center point coordinates of the point cloud detection target to the world coordinate system with the camera center as the origin, calculate the center point coordinates of the point cloud detection target in the world coordinate system, and then calculate the height of the center point projection; The second step is to transfer the corrected center point coordinates to the image coordinate system and calculate the center point coordinates of the point cloud detection target in the image coordinate system; The third step is to calculate the coordinates of the upper left corner and lower right corner of the target box of the point cloud detection target in the image coordinate system; In the fourth step, the center point coordinates, lower left corner coordinates, upper right corner coordinates, speed and radar cross-sectional area information of the point cloud detection target in the image coordinate system are combined to form a projection target. All projection targets form a projection target set.

6. The method according to claim 5, wherein In step 4, the center point coordinates (x w ,y w ) is as follows: Where θ is the offset angle between the millimeter-wave radar and the visual camera, Where x is the horizontal lateral offset of the millimeter-wave radar and the vision camera, and y is the horizontal longitudinal offset of the millimeter-wave radar and the vision camera.

7. The method according to claim 6, wherein In step 4, the height of the center point projection h is the camera installation height.

8. The method according to claim 7, wherein In step 4, the method for calculating the center point coordinates (u, v) of the point cloud detection target in the image coordinate system is: f x and f y is the focal length of the visual camera in the x and y directions, c x and c y are the distortion parameters of the visual camera in the x and y directions.

9. The method according to claim 8, wherein In step 4, the coordinates of the upper left corner of the target box of the point cloud detection target in the image coordinate system (u lefttop , v lefttop ) and the lower right corner coordinate (u rightbottom , v rightbottom ) is as follows: a and b are the highest and lowest points of the point cloud detection target.

10. A surface unmanned boat environment perception system based on the fusion of vision and millimeter-wave radar under weak observation conditions, characterized by: The invention comprises an unmanned boat, wherein a visual camera and a millimeter-wave radar are installed at the bow of the unmanned boat, an inertial measurement unit device is installed in the cabin of the unmanned boat, the visual camera is used to acquire images, the millimeter-wave radar is used to acquire point cloud data, and the inertial measurement unit is used to acquire the heading angle, roll angle and pitch angle of the unmanned boat; an industrial computer is installed in the cabin of the unmanned boat, and the industrial computer is connected with the visual camera, the millimeter-wave radar and the inertial measurement unit; the industrial computer comprises a point cloud processing module, an image processing module, a primary target fusion module, a secondary target fusion module and a final target output module; the point cloud processing module comprises a clustering algorithm unit and a coordinate projection unit; the clustering algorithm unit is used to cluster the point cloud data and obtain a point cloud detection target set composed of point cloud detection targets; the coordinate projection unit is used to project the point cloud detection targets into an image coordinate system and obtain a projection target set composed of projection targets; the image processing module The processing module includes a target recognition unit and an anchor frame storage unit, wherein the target recognition unit is used to put the image into the deep target detection network for target recognition and obtain an image detection target set composed of image detection targets; the anchor frame storage unit is used to save the anchor frame before the deep target detection network performs non-maximum suppression processing and obtain a complete anchor frame set composed of anchor frames; the primary target fusion module includes an image overlap calculation unit, a primary fusion output unit and a primary target deletion unit; the image overlap calculation unit is used to calculate the overlap of each projection target with all image detection targets; the primary fusion output unit is used to fuse the image detection target with an overlap greater than 0.5 and the largest overlap with the projection target and obtain a primary fusion target, and the primary target deletion unit is used to delete the fused projection target and image detection target from the projection target set and the image detection target set and obtain an initial remaining projection target set and a remaining image detection target set; The final target output module is used to output the final target set and add the first fusion target and the remaining image detection targets to the final target set; The secondary target fusion module includes an anchor frame overlap calculation unit, an anchor frame screening unit, a secondary fusion output unit, and a secondary target deletion unit; the anchor frame overlap calculation unit is used to calculate the overlap between each projection target in the initial residual projection target set and all anchor frames in the complete anchor frame set; the anchor frame screening unit is used to screen out anchor frames with an overlap greater than 0.5 with the projection target and form a fused anchor frame set; the secondary fusion output unit is used to fuse the fused anchor frame set with the projection target to obtain a secondary fused target; the secondary target deletion unit is used to delete the projection target fused with the fused anchor frame set and obtain a secondary residual projection target set; The final target output module is used to add the secondary fusion target and the secondary residual projection target set to the final target set.

Images