An unmanned ship intelligent environment perception method fusing image segmentation and millimeter wave radar

Abstract

The application discloses a low-cost, high-precision and high-reliability method for unmanned ship intelligent environment perception by fusing image segmentation and millimeter wave radar, which can acquire the environment information of the route and the surrounding of the unmanned ship in real time during the driving of the unmanned ship, including the shoreline, the sailing ship, the navigation mark and the floating object, and provide decision support for the real-time route adjustment and obstacle avoidance of the unmanned ship. The method comprises four parts of the millimeter wave radar, the binocular camera, the image segmentation module and the information fusion module, the image segmentation module is responsible for reading the image data, executing the image segmentation algorithm, identifying the target and generating the minimum fitting rectangle of the target, the information fusion module is responsible for receiving the data of the radar and the image segmentation module, acquiring the polar coordinates of the fusion target, matching the target size and outputting the result to the controller.

Description

Technical Field

[0001] This invention belongs to the field of intelligent unmanned surface vessel (USV) environmental perception, specifically a method for intelligent environmental perception of USVs that integrates image segmentation and millimeter-wave radar. Background Technology

[0002] Unmanned surface vessels (USVs) have been widely used in environmental protection, emergency rescue, scientific research, and military fields due to their advantages such as high maneuverability, efficiency, and unmanned operation. The mission modules carried by USVs are typically of high value. In military and emergency rescue fields, where speed is high and mission timeliness is critical, USVs need to adjust their planned routes in real time based on the surrounding environment. Therefore, obtaining accurate and comprehensive environmental information in real time is particularly essential.

[0003] Millimeter-wave radar has advantages such as long detection range, high accuracy, good resistance to environmental interference, and low cost. However, it is limited to acquiring only the target's position information and cannot reflect the target's outline information. Binocular vision can detect the target's distance and size, but it has large errors and is easily affected by environmental interference. LiDAR has advantages such as long detection range and high detection accuracy, but its high cost limits its application. In summary, there is a need to design a fusion environmental perception sensor to achieve a low-cost, high-accuracy, and highly reliable environmental perception solution for unmanned surface vessels. Attached Figure Description

[0004] Figure 1 A diagram illustrating the structure of an intelligent environmental perception method for unmanned surface vessels that integrates image segmentation and millimeter-wave radar.

[0005] Figure 2 A flowchart illustrating an intelligent environmental perception method for unmanned surface vessels that integrates image segmentation and millimeter-wave radar. Summary of the Invention

[0006] This invention proposes an intelligent environmental perception method for unmanned vessels that integrates image segmentation and millimeter-wave radar. This method is characterized by high accuracy, high reliability, and low cost.

[0007] This method employs a deep learning-based image segmentation algorithm, which, compared to traditional machine vision, has stronger resistance to lighting interference and can achieve multi-target recognition, accurately segmenting targets such as water surfaces, shorelines, ships, and islands.

[0008] The method includes the following steps:

[0009] S1. The image segmentation module acquires data from the stereo camera. Taking the left image as an example, the data is input into the trained image segmentation algorithm to perform background and foreground segmentation calculations. If no target is detected, no data is output to the information fusion module. If target data is detected, the image segmentation algorithm acquires the target category name and the segmented image data, and acquires the pixel data with the target label.

[0010] S2. The number of targets n obtained by the image segmentation algorithm can be any integer greater than 0 and less than 1000;

[0011] S3. Starting from the coordinate point (0, 0), traverse the image pixels. After traversing to the labeled pixel, extract and save the starting coordinate point (X11, Y11) and the last pixel coordinate point (X12, Y12) of the target pixel. Continue traversing and updating the pixels of the next row, extract and save the minimum and maximum values ​​of the X-axis and Y-axis coordinates of all targets.

[0012] S4. Taking target 1 as an example, based on the minimum and maximum coordinates Xmin, Ymin, Xmax, and Ymax of the X and Y axes, generate and save the vertex coordinates of the minimum bounding rectangle of target 1 as (X1min, Y1min), (X1min, Y1max), (X1max, Y1min), and (X1max, Y1max).

[0013] S5. Traverse the image once to obtain the coordinates of the vertices of the minimum bounding rectangle of the contours of target 1 to target n.

[0014] S6. For the right figure, repeat steps 3-5 to generate the vertex coordinates of the minimum bounding rectangle of the contour of target 1-target n.

[0015] S7. Generate the coordinate center point O1-On for target 1 to target n in the left and right images respectively. Calculate the polar radius H1-Hn and polar angle α1-αn of O1-On relative to the sensor. Use the polar radius H as the first sorting criterion and the polar angle α as the second sorting criterion to generate a polar coordinate sorting list (H1, α1), ..., (Hn, αn) from small to large.

[0016] S8. Obtain the polar radius Hr1-Hrn and polar angle αr1-αrn of radar output target 1-target n, use the polar radius H as the first sorting criterion, use the polar angle α as the second sorting criterion, and generate a polar coordinate sorting list (Hr1, αr1), ..., (Hrn', αrn').

[0017] S9. Set the matching threshold between radar target coordinates and image output target center point coordinates to (deltah, deltaα). The value of deltah is in the range of [0m, 2m], and can be an integer multiple of 0.1m; the value of deltaα is in the range of [0°, 1°], and can be an integer multiple of 0.1°.

[0018] S10. Update the polar coordinates of target 1 to target n in the image output to the polar coordinates of the radar (Hr1, αr1), ..., (Hrn, αrn), and save the center coordinate offset (Hr1-H1, αr1-α1), ..., (Hrn-Hn, αrn-αn).

[0019] S11. For target 1 to target n, set the target fusion threshold (Hdelta, αdelta). Hdelta ranges from [0.1m - 5m] and can be a positive integer multiple of 0.1m. αdelta ranges from [0.1° - 2°] and can be a positive integer multiple of 0.1°.

[0020] S12. Compare the polar coordinate differences of H1 and H2. If |H1-H2|≤Hdelta & |α1-α2|≤αdelta, extract the minimum values ​​of (X1min, X2min) and (Y1min, Y2min) respectively, and extract the maximum values ​​of (X1max, Y2max) and (Y1max, Y2max) respectively. Use these as the vertex coordinates of the minimum bounding rectangle of the quadratic fitting target. Use min(H1, H2) as the polar axis coordinates of the target after fusion, and use min(α1, α2) as the polar angle coordinates.

[0021] S13. Compare (H1, α1) with (H2-Hn, α2-αn) sequentially, and compare (H2', α2') with (H3'-Hn', α2'-αn') until (H(n-1)""α(n-1)") and (Hn",αn"), to generate the final quadratic fitting target 1-target n' polar coordinates, which are the center point coordinates of the smallest bounding rectangle contour of the fused target.

[0022] S14. Generate the polar coordinate data of (Xmin, Ymax) and (Xmax, Ymax) of O1-On', calculate the polar coordinate data (H1, α1), ..., (Hn, αn)', and calculate the width data of target 1-target n' according to the principle of similar triangles. Match the polar coordinates and width data of the target center point as the final output. Detailed Implementation

[0023] The method includes the following steps:

[0024] Step 1: The image segmentation module acquires data from the stereo camera. Taking the left image as an example, the data is input into the trained image segmentation algorithm to perform background and foreground segmentation calculations. If no target is detected, no data is output to the information fusion module. If target data is detected, the image segmentation algorithm acquires the target category name and the segmented image data, and obtains the pixel data with the target label.

[0025] During training, the image segmentation algorithm labels water surfaces (blue), ships (red), shorelines (gray), aquatic plants (green), and navigation marks (yellow) to facilitate labeling correction and performance evaluation. More than 2000 images are selected for each target, resulting in good generalization performance.

[0026] Step 2: The number of targets n obtained by the image segmentation algorithm can be any integer greater than 0 and less than 1000;

[0027] The image segmentation algorithm used in this invention can detect 5 target categories. This algorithm can reduce the number of categories to be identified and can also expand the number of target detection categories to a maximum of 1000 categories. The final output category corresponds to the dimension of the output layer. When optimizing the model, only the dimension of the last layer of the network needs to be adjusted.

[0028] Step 3: Starting from the coordinate point (0, 0), traverse the image pixels. After traversing to the labeled pixel, extract and save the starting coordinate point (X11, Y11) and the last pixel coordinate point (X12, Y12) of the target pixel. Continue traversing and updating the pixels of the next row, extract and save the minimum and maximum values ​​of the X-axis and Y-axis coordinates of all targets.

[0029] Step 4: Taking target 1 as an example, based on the minimum and maximum coordinates Xmin, Ymin, Xmax, and Ymax of the X and Y axes, generate and save the vertex coordinates of the minimum bounding rectangle of target 1 as (X1min, Y1min), (X1min, Y1max), (X1max, Y1min), and (X1max, Y1max).

[0030] The image segmentation algorithm returns an image with dimensions of 3 (RGB three channels) * W (width) * H (height). Taking a ship target as an example, the segmented ship image pixel data is [255, 0, 0]. Starting from (0, 0), the image is traversed row by row. The pixel coordinates (X1, Y1) of the first ship target found are saved as the Xmin and Ymin coordinates of the vertex of the minimum bounding rectangle of the ship. The next pixel value found is saved as the Xmax and Ymin coordinates of the vertex of the minimum bounding rectangle. The image is traversed row by row, and the last pixel coordinates are saved as Xmax and Ymin. The Xmin, Ymin, Xmax, and Ymax of the ship target are saved row by row.

[0031] In cases where multiple targets of the same category are detected in an image, the same category is numbered from 0 to n. Taking ships as an example, they are named Ship_0, Ship_1, ..., Ship_n. Since each pixel is a labeled pixel, adding a category judgment condition during the traversal of pixel RGB values ​​can effectively distinguish multiple targets of the same category.

[0032] Step 5: Traverse the image once to obtain the vertex coordinates of the minimum bounding rectangle of the contours of target 1 to target n;

[0033] In this invention, the bounding polygon of the fitted contour is a rectangle, and the number of selected vertices is 4. For other special targets, it can be a triangle, hexagon, or other shapes, and the number of generated vertex coordinates can be 3, 6, or more.

[0034] Step 6: For the right figure, repeat steps 3-5 to generate the vertex coordinates of the minimum bounding rectangle of the contours of target 1 to target n;

[0035] During the inference process of the image segmentation algorithm, inference calculations are performed on the left and right images respectively, the target categories are extracted and the images are processed. According to the performance test results after the algorithm training and the algorithm application experiment, the image segmentation results of the left and right images are consistent and high.

[0036] Step 7: Generate the coordinate center point O1-On for target 1-target n in the left and right images respectively. Calculate the polar radius H1-Hn and polar angle α1-αn of O1-On relative to the sensor. Use the polar radius H as the first sorting criterion and the polar angle α as the second sorting criterion to generate a polar coordinate sorting list (H1, α1), ..., (Hn, αn) from small to large.

[0037] In this invention, the target center point is extracted by generating polar coordinates. Other coordinate systems, such as Cartesian coordinates, can also be used to represent the center point coordinates. The advantage of using polar coordinates is that it reduces the computational load when fusing radar data and image data. The purpose of coordinate sorting is to match it with the radar target coordinate data; other matching methods, such as traversal methods, can be used.

[0038] Step 8: Obtain the polar radius Hr1-Hrn and polar angle αr1-αrn of radar output target 1-target n, use the polar radius H as the first sorting criterion and the polar angle α as the second sorting criterion to generate a polar coordinate sorting list (Hr1, αr1), ..., (Hrn', αrn').

[0039] Radar has a wider sensing range than binocular cameras, and the number of targets detected by radar may be greater than the number of targets identified by image recognition. After generating polar coordinates, the sorting methods for both are consistent, which can quickly match targets.

[0040] Step 9: Set the matching threshold between the radar target coordinates and the target center point coordinates in the image output to (deltah, deltaα). The value of deltah is in the range of [0m, 2m], and can be an integer multiple of 0.1m; the value of deltaα is in the range of [0°, 1°], and can be an integer multiple of 0.1°.

[0041] There are cases where the number of targets n and n' in image segmentation and recognition do not match. For example, floating objects such as duckweed cannot be detected by millimeter-wave radar, but are identified by the image segmentation module, resulting in the number of targets in the image being greater than the number of targets in the radar. If there are no matching coordinates after matching, the coordinates of the unmatched image are used as the coordinates of the target. If the weather is bad, the visibility is low, and the camera detection distance is short, resulting in the number of targets in the radar being greater than the number of targets in the image, the coordinates of the unmatched radar target are used as the coordinates of the target, and the coordinates of the vertices of the minimum bounding rectangle are set to zero.

[0042] Step 10: Update the polar coordinates of target 1 to target n in the image output to the polar coordinates of the radar (Hr1, αr1), ..., (Hrn, αrn), and save the center coordinate offset (Hr1-H1, αr1-α1), ..., (Hrn-Hn, αrn-αn).

[0043] Before application, the coordinate systems of the millimeter-wave radar and the binocular camera are jointly calibrated, and the image polar coordinate system is transformed into the radar coordinate system. The specific coordinate transformation matrix is ​​determined according to the millimeter-wave radar parameters and installation location, and the binocular camera parameters and installation location.

[0044] Step 11: For target 1 to target n, set the target fusion threshold (Hdelta, αdelta). The range of Hdelta is [0.1m - 5m], and Hdelta can be a positive integer multiple of 0.1m. The range of αdelta is [0.1° - 2°], and αdelta can be a positive integer multiple of 0.1°.

[0045] Fusion of detected targets reduces the number of targets output to the unmanned vessel controller, decreasing the controller's computational load and data transmission volume, and improving the robustness of environmental perception. In the practical application of this invention, Hdelta=3m and αdelta=1°, effectively fusing targets that are close to the target while retaining targets that are farther away, thus providing effective input data for the unmanned vessel controller.

[0046] Step 12: Compare the polar coordinate differences of H1 and H2. If |H1-H2|≤Hdelta & |α1-α2|≤αdelta, extract the minimum values ​​of (X1min, X2min) and (Y1min, Y2min) respectively, and extract the maximum values ​​of (X1max, Y2max) and (Y1max, Y2max) respectively. Use these as the vertex coordinates of the minimum bounding rectangle of the quadratic fitting target. Use min(H1, H2) as the polar axis coordinates of the target after fusion, and use min(α1, α2) as the polar angle coordinates.

[0047] The coordinates of the fused vertex are the coordinates of the smallest bounding rectangle of target 1 and target 2. The coordinates of the fused center point will also be updated based on the vertex coordinates.

[0048] Step 13: Compare (H1, α1) with (H2-Hn, α2-αn) sequentially, and compare (H2', α2') with (H3'-Hn', α2'-αn') until (H(n-1)""α(n-1)") and (Hn",αn"), to generate the final quadratic fitting target 1-target n' polar coordinates, which are the center point coordinates of the smallest bounding rectangle contour of the fused target.

[0049] When performing target fusion in images, it is only necessary to obtain the vertex data of the smallest bounding rectangle after target fusion and calculate the polar coordinates of the corresponding vertices in the left and right images. It is not necessary to calculate the polar coordinate data of the midpoint of the coordinates again.

[0050] Step 14: Generate the polar coordinate data of (Xmin, Ymax) and (Xmax, Ymax) of O1-On', calculate the polar coordinate data (H1, α1), ..., (Hn, αn)', and calculate the width data of target 1-target n' according to the principle of similar triangles. Match the polar coordinates and width data of the target center point as the final output.

[0051] The left and right images are merged to capture neighboring targets, generating the vertex coordinates of the minimum bounding rectangle of the final target outline. The distance and width of the target are then calculated and retained after merging. Finally, the obtained polar coordinates of the target midpoint and the target width are sent to the controller, which adjusts and generates a new flight path based on the input data and the current flight path.

Claims

1. An unmanned ship intelligent environment perception method fusing image segmentation and millimeter wave radar, characterized in that , comprising the following steps: S1, the image segmentation module acquires the data of the binocular vision module, inputs the data into the trained image segmentation algorithm for background and foreground segmentation calculation, acquires the identification target class name and segmented image data of the image segmentation algorithm, and acquires the pixel data with target labels; S2, the target number n acquired by the image segmentation algorithm can be any integer greater than 0 and less than 1000; S3, starting from the (0, 0) coordinate point, traversing the image pixel points, after traversing the pixel with a label, the starting coordinate point (X11, Y11) of the target pixel and the last pixel coordinate point (X12, Y12) are extracted and saved; continue to traverse and update the pixel points of the next row, and extract and save the minimum and maximum values of the X axis and Y axis coordinates of all targets; S4, according to the minimum and maximum coordinates Xmin, Ymin, Xmax and Ymax of the X axis and Y axis, the minimum circumscribed rectangle vertex coordinates (X1min, Y1min), (X1min, Y1max), (X1max, Y1min) and (X1max, Y1max) of target 1 are generated and saved; S5, traverse the image once to acquire the contour minimum circumscribed rectangle vertex coordinates of target 1-target n; S6, for the right image, S3-S5 are cyclically executed to generate the contour minimum circumscribed rectangle vertex coordinates of target 1-target n; S7, the coordinate center points O1-On of target 1-target n are generated for the left and right images respectively, the polar radii H1-Hn and the polar angles α1-αn of O1-On relative to the sensor are calculated, the polar radius H is taken as the first sorting basis, the polar angle α is taken as the second sorting basis, and the polar coordinate sorting list (H1, α1), …, (Hn, αn) is generated from small to large; S8, the polar radii Hr1-Hrn and the polar angles αr1-αrn of the radar output target 1-target n are acquired, the polar radius H is taken as the first sorting basis, the polar angle α is taken the second sorting basis, and the polar coordinate sorting list (Hr1, αr1), …, (Hrn', αrn') is generated; S9, set the matching threshold of the radar target coordinate and the image output target center point coordinate to (deltah, deltaα), deltah takes the value range of [0m, 2m], which can be an integer multiple of 0.1m; deltaα takes the value range of [0°, 1°], which can be an integer multiple of 0.1°; S10, update the polar coordinates of the image output target 1-target n to the polar coordinates (Hr1, αr1), …, (Hrn, αrn) of the radar, and save the center coordinate offset (Hr1-H1, αr1-α1), …, (Hrn-Hn, αrn-αn); S11, for target 1-target n, set the target fusion threshold (Hdelta, alpha delta), Hdelta ranges from 0.1m to 5m, Hdelta can be a positive integer multiple of 0.1m, alpha delta ranges from 0.1° to 2°, and alpha delta can be a positive integer multiple of 0.1°. S12, compare the polar coordinate difference values of H1 and H2, if |H1-H2|≤Hdelta & |α1-α2|≤αdelta, extract the minimum values of (X1min, X2min) and (Y1min, Y2min) respectively, extract the maximum values of (X1max, Y2max), (Y1max, Y2max) respectively, as the vertex coordinates of the minimum circumscribed rectangle of the secondary fitting target, take min(H1, H2) as the polar axis coordinate of the target after fusion, and take min(α1, α2) as the polar angle coordinate; S13, compare (H1, α1) and (H2-Hn, α2-αn) in turn, compare (H2', α2') and (H3'-Hn', α2'-αn'), and so on, until (H(n-1)", α(n-1)") and (Hn", αn") are compared, to generate the final obtained polar coordinates of the secondary fitting target 1-target n', that is, the center point coordinates of the fused target minimum circumscribed rectangle contour; S14, generate the polar coordinate data of (Xmin, Ymax) and (Xmax, Ymax) of O1-On', calculate the polar coordinate data (H1, α1), …, (Hn, αn)', according to the principle of similar triangles, calculate the width data of target 1-target n', match the target center point polar coordinate and width data, and output finally.

Images