A multi-sensor adaptive tugboat autonomous escort perception method and equipment

Abstract

The present invention provides a multi-sensor adaptive tugboat autonomous escort perception method and device, relating to the field of escort perception, including: S1: obtaining the distance d1 between the tugboat and the target ship at time t1; if d1 is less than a threshold value D1 and the AIS successfully matches the K-band radar target, the process proceeds to step S2; S2: determining whether the visual camera detects the target; if so, the process proceeds to step S3; S3: obtaining the distance d2 between the tugboat and the target ship at time t2; if d2 is less than the threshold value D2, the visual camera and the laser radar synchronously collect data on the target ship to obtain a camera detection frame and a laser radar detection frame; S4: fusing the camera detection frame and the laser radar detection frame to obtain the current coordinates of the target ship. The present invention uses multiple sensors to compensate for the insufficient perception capability of a single sensor, and gradually completes the escort process in two stages, thereby improving the tugboat's perception of the surrounding environment as it approaches the target.

Description

Technical Field

[0001] The present invention relates to the field of escort perception, and in particular to a multi-sensor adaptive tugboat autonomous escort perception method and equipment. Background Art

[0002] Traditional autonomous navigation systems have limitations when dealing with complex environments and adverse weather conditions. These systems often rely on specific sensors and pre-planned routes and have limited adaptability to various environmental changes. Although existing technologies have improved these problems to a certain extent, they still have some limitations. The recognition requirements for targets at different distances and scales are different. A single sensor often cannot provide sufficient perception information in complex sea conditions, resulting in insufficient perception of the surrounding environment by the navigation system. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multi-sensor adaptive tugboat autonomous escort perception method and equipment to solve the problem of insufficient perception capability of existing navigation systems.

[0004] The present invention provides a multi-sensor adaptive tugboat autonomous escort perception method, comprising:

[0005] S1: Obtain the distance d1 between the tugboat and the target vessel at time t1. If d1 is less than the threshold D1 and the AIS and K-band radar targets are successfully matched, proceed to step S2. Otherwise, repeat step S1.

[0006] S2: Acquire the K-band radar target data of the target ship through AIS and K-band radar fusion matching, and determine whether the visual camera detects the target. If the target is detected, proceed to step S3, otherwise repeat step S2;

[0007] S3: Obtain the distance d2 between the tugboat and the target vessel at time t2; if d2 is less than the threshold D2, the visual camera and lidar perform synchronous data acquisition of the target vessel under the guidance of the K-band radar target data, obtain the camera detection frame and the lidar detection frame respectively, and proceed to step S4; otherwise, return to step S2;

[0008] S4: Fuse the camera detection frame and the lidar detection frame to obtain the current coordinates of the target ship.

[0009] Preferably, the target matching process between AIS and K-band radar is as follows:

[0010] S11: Obtain the distance d of the target ship relative to the tugboat through AIS, GPS and fiber optic compass AIS , azimuth and speed

[0011] S12: Get the distance of the u-th K-wave target through K-band radar Azimuth and speed

[0012] S13: Pass d AIS 、 and Calculate the matching error e of the TTM data of the u-th K-wave target u ;

[0013] S14: Repeat steps S12-S13 to obtain the matching errors of the TTM data of all K-wave targets, and take the TTM data with the smallest matching error as the possible target;

[0014] S15: If the matching error of the possible target is less than 10%, it is determined that the AIS and the K-band radar target are successfully matched, and the successfully matched TTM data is the K-band radar target data of the target ship. Otherwise, it is determined that the AIS and the K-band radar target are not successfully matched.

[0015] Preferred:

[0016] The calculation formula of the matching error E of the possible target is:

[0017]

[0018] Among them, w d is the weight of the distance error, w v is the weight of the velocity error, d AIS-x and d AIS-y d AIS The x-axis and y-axis components, ν AIS-x and ν AIS-y They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of .

[0019] Preferably, step S3 is specifically as follows:

[0020] S31: At the same time, under the guidance of the K-band radar target data, the image data of the target ship is obtained through the visual camera, and the point cloud data of the target ship is obtained through the lidar;

[0021] S32: Input the image data into the improved YOLO v7 model to calculate the camera detection frame;

[0022] S33: Preprocessing the point cloud data to obtain preprocessed point cloud data;

[0023] S34: Clustering the preprocessed point cloud data using a density clustering algorithm based on a variable radius to obtain a lidar detection frame.

[0024] Preferred:

[0025] The improved YOLO v7 model includes: MobileNetV3 feature extraction network and GAM attention module;

[0026] The MobileNetV3 feature extraction network includes sequentially connected convolutional layers, the inverse residual structure of the MobileNetV2 linear bottleneck, depthwise separable convolutional layers, the SE-Net attention model, and the hard-swish activation function module;

[0027] The GAM attention module includes: channel attention submodule and spatial attention submodule;

[0028] The hard-swish activation function module, channel attention submodule and spatial attention submodule are connected in sequence.

[0029] Preferably, step S33 is specifically as follows:

[0030] S331: downsampling the point cloud data using a voxel filtering method to obtain downsampled point cloud data;

[0031] S332: Filter the downsampled point cloud data through a statistical filter to obtain preprocessed point cloud data.

[0032] Preferably, step S34 is specifically as follows:

[0033] S341: Divide the radius into an initial radius R and a clustering radius, obtain core points, and define parameter ρ as the point cloud density threshold;

[0034] S342: Record the point set with the number of points greater than ρ within the initial radius of each core point as the core point cloud, and calculate the Euclidean distance d, d between two adjacent points in the core point cloud. max is the maximum Euclidean distance, d min is the minimum Euclidean distance;

[0035] S343: Let d1 = d max -d min , d2=Rd max ;

[0036] S344: If d1 / d2=1, set the cluster radius r=R;

[0037] If d1 / d2>1, then let the cluster radius r=R+d min ;

[0038] If d1 / d2<1, then let the cluster radius r=Rd min ;

[0039] S345: Cluster the pre-processed point cloud data using a clustering radius r to obtain a lidar detection frame.

[0040] Preferably, step S4 is specifically as follows:

[0041] S41: Calculate the area S of the camera detection frame c and the area S of the lidar detection frame l ;

[0042] S42: Area S of the camera detection frame c and the area S of the lidar detection box projected onto the image l The degree of overlap is calculated based on the degree of coincidence Calculation of

[0043] S43: If If the error is less than 50%, it is considered that the detection objects of the two detection frames are not the same target, and information fusion cannot be performed. The detection results of the visual camera and the lidar are output separately;

[0044] like If the detection rate is greater than or equal to 50%, the detection objects of the two detection frames are considered to be the same target, and the camera detection frame and the lidar detection frame are fused to output the current coordinates of the fused target ship.

[0045] A storage medium stores instructions and data for implementing a multi-sensor adaptive tugboat autonomous escort perception method.

[0046] A multi-sensor adaptive tugboat autonomous escort perception device comprises: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the multi-sensor adaptive tugboat autonomous escort perception method.

[0047] The present invention has the following beneficial effects:

[0048] The tugboat and the target vessel are escorted by multiple sensors, including AIS, K-band radar, visual cameras, and lidar. The AIS and K-band radar collaborate to detect and match data collected by the two devices, completing the first phase of escort. The visual cameras and lidar acquire the target vessel's current coordinates, completing the second phase. This multi-sensor approach compensates for the limited perception capabilities of a single sensor, and the two-stage escort phase enhances the tugboat's awareness of its surroundings as it approaches the target, enabling adaptive autonomous escort. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is the first phase of the collaborative perception matching process between AIS and K-band radar;

[0051] Figure 3 This is the second stage of camera and lidar fusion perception process;

[0052] Figure 4 This is the fusion process of the camera detection frame and the lidar detection frame in the second stage;

[0053] Figure 5 This is a structural diagram of the device according to an embodiment of the present invention;

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Reference Figure 1 The present application provides a multi-sensor adaptive tugboat autonomous escort perception method, comprising:

[0057] S1: Obtain the distance d1 between the tugboat and the target vessel at time t1. If d1 is less than the threshold D1 and the AIS and K-band radar targets are successfully matched, proceed to step S2. Otherwise, repeat step S1.

[0058] Furthermore, a multi-source information data platform is constructed based on different types of sensor equipment, including: AIS, K-band radar, and detailed inspection radar;

[0059] AIS: A new navigation aid system used for maritime safety and communication between ships and shore, and between ships. It typically consists of a VHF communicator, a GPS locator, and a communication controller connected to shipboard displays and sensors. It automatically exchanges important information such as ship position, speed, heading, name, and call sign.

[0060] K-band radar: A K-band FMCW radar with a maximum detection range of 1.5 km, a range resolution of 0.75 m, and an angular resolution of 1.44°.

[0061] Detailed Inspection Radar: Consists of one LiDAR module and two visual camera modules. The LiDAR is a 150-line LiDAR, consisting of a long-range, high-resolution specialized LiDAR and a lightweight, high-precision electro-optical stabilization gimbal. It has a maximum detection range of 500 meters and an angular resolution of 0.18° × 0.24°. The visual cameras are one telephoto industrial camera and one wide-angle industrial camera.

[0062] Steps S1-S2 are the first stage of the autonomous escort process of the present invention. The tugboat's task is to lock the position of the target ship and track it, that is, the perception system identifies the object as an escort ship. When the distance reaches 1500m, the recognition range of the K-wave radar has been reached. The K-wave radar can continuously track and lock the identified target, and output the target number, relative distance, relative azimuth, relative speed, and relative heading. However, the K-wave radar cannot determine which target is the target ship. The AIS data contains the MMSI number of the ship, and the target ship can be filtered out by the MMSI, and the latitude and longitude, heading and speed information of the target ship can be obtained. Therefore, the target ship data of the AIS is matched with the target mapped by the K-wave radar, so that the K-wave radar continuously outputs the information of the target ship.

[0063] refer to Figure 2 ,The target matching process between AIS and K-band radar is as follows:

[0064] S11: Obtain the distance d of the target ship relative to the tugboat through AIS, GPS and fiber optic compass AIS , azimuth and speed

[0065] Specifically, the target ship's coordinates in AIS data are expressed in latitude and longitude, the heading is absolute, and the speed is absolute. K-wave radar data is represented differently, so it must be converted to a relative coordinate system. The target ship's latitude and longitude are calculated with the own ship's latitude and longitude, as well as its heading. The target ship's relative coordinates are then calculated using the own ship's absolute speed and absolute heading to determine the target ship's relative speed and heading.

[0066] Assume that the longitude and latitude coordinates of the target ship measured by AIS are (Lng t , Lat t ), the latitude and longitude coordinates of the ship are obtained from the Global Positioning System (GPS) as (Lng s , Lat s ), then the differences in latitude and longitude of the target ship relative to the own ship are:

[0067] ΔLng=Lng t -Lng s

[0068] ΔLat=Lat t -Lat s

[0069] The distance differences in the longitude and latitude directions are:

[0070]

[0071] Where R is the radius of the Earth.

[0072] With due north as 0° and eastward rotation as positive, the azimuth of the target ship relative to the own ship is and distance d AIS They are:

[0073]

[0074] The heading of the ship due north obtained by the gyrocompass is Then the azimuth of the AIS target relative to the heading is:

[0075]

[0076] The coordinates of the target from the K-wave radar in the Cartesian coordinate system are:

[0077]

[0078] make but:

[0079] P AIS =[d AIS-x , d AIS-y ]

[0080] Assume the absolute heading of the target ship is C AIS , the absolute speed is ν AIS , the absolute heading of own ship is C s , the absolute speed is ν s , then the relative velocity vector of the target ship obtained by AIS relative to the own ship can be expressed as:

[0081]

[0082] Order ν AIS ·cosC AIS -ν s ·cosC s =ν AIS-x , ν AIS sinCAIS -ν s sinC s =d AIS-y ,but:

[0083]

[0084] S12: Get the distance of the u-th K-wave target through K-band radar Azimuth and speed

[0085] Specifically, the information plotted by the K-wave radar includes the relative distance to the target Relative azimuth Relative heading and relative speed

[0086] The coordinates of the target from the K-wave radar in the Cartesian coordinate system are:

[0087]

[0088] make but:

[0089]

[0090] The target velocity vector from the K-wave radar is:

[0091]

[0092] make but:

[0093]

[0094] The target data plotted by the K-wave radar are grouped according to the target number to obtain the data of different targets. Then, the difference between the coordinates and speeds of all targets plotted by the K-wave radar and the coordinates and speeds of the target ship calculated from the AIS is calculated.

[0095] S13: Pass d AIS 、 and Calculate the matching error e of the TTM data of the u-th K-wave target u ;

[0096] S14: Repeat steps S12-S13 to obtain the matching errors of the TTM data of all K-wave targets, and take the TTM data with the smallest matching error as the possible target;

[0097] S15: If the matching error of the possible target is less than 10%, it is determined that the AIS and the K-band radar target are successfully matched, and the successfully matched TTM data is the K-band radar target data of the target ship. Otherwise, it is determined that the AIS and the K-band radar target are not successfully matched.

[0098] Furthermore, the calculation formula of the matching error E of the possible target is:

[0099]

[0100] Among them, w d is the weight of the distance error, w v is the weight of the velocity error, d AIS-x and d AIS-y d AIS The x-axis and y-axis components, ν AIS-x and ν AIS-y They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of .

[0101] Specifically, the K-wave radar target with the smallest difference is selected. If the error is less than 10%, it is considered that the K-wave radar target and the AIS target are both escort vessels. Therefore, the K-wave radar can continuously output information about the target to achieve tracking.

[0102] S2: Acquire the K-band radar target data of the target ship through AIS and K-band radar fusion matching, and determine whether the visual camera detects the target. If the target is detected, proceed to step S3, otherwise repeat step S2;

[0103] S3: Obtain the distance d2 between the tugboat and the target vessel at time t2; if d2 is less than the threshold D2, the visual camera and lidar perform synchronous data acquisition of the target vessel under the guidance of the K-band radar target data, obtain the camera detection frame and the lidar detection frame respectively, and proceed to step S4; otherwise, return to step S2;

[0104] Furthermore, steps S3-S4 represent the second stage of the autonomous navigation process of the present invention. With time t2 greater than time t1, the pan / tilt of the detailed inspection radar in the second stage is guided by the tracking results from step 1, so that the visual camera is always oriented toward the escorted vessel. When the visual camera is able to identify the escorted vessel, it uses it to identify and lock onto the vessel and its features, and uses the laser radar to output its coordinates.

[0105] refer to Figure 3, step S3 is specifically as follows:

[0106] S31: At the same time, under the guidance of the K-band radar target data, the image data of the target ship is obtained through the visual camera, and the point cloud data of the target ship is obtained through the lidar;

[0107] S32: Input the image data into the improved YOLO v7 model to calculate the camera detection frame;

[0108] Furthermore, the improved YOLO v7 model includes: MobileNetV3 feature extraction network and GAM attention module;

[0109] The MobileNetV3 feature extraction network includes sequentially connected convolutional layers, the inverse residual structure of the MobileNetV2 linear bottleneck, depthwise separable convolutional layers, the SE-Net attention model, and the hard-swish activation function module;

[0110] The GAM attention module includes: channel attention submodule and spatial attention submodule;

[0111] The hard-swish activation function module, channel attention submodule and spatial attention submodule are connected in sequence.

[0112] Specifically, the MobileNetV3 feature extraction network is used as the backbone network of the YOLO v7 model to implement feature extraction.

[0113] The operation process of the MobileNetV3 network is as follows: first, use 1×1 convolution to increase the dimension, and then introduce the inverse residual structure of the MobileNetV2 linear bottleneck; then perform 3×3 depth-separable convolution to reduce the computational complexity of the network; then use the attention model SE-Net to make the network focus on more useful channel information to adjust the weight of each channel; finally, use the hard-swish activation function instead of the swish function to reduce the amount of computation and improve the running speed of the model.

[0114] Add the feature fusion network of the GAM attention module. Furthermore, the GAM attention module includes two sub-modules: channel attention and spatial attention. Its mathematical expression is:

[0115]

[0116] Where: F1∈R C×H×W is the input feature map; M c and M s are channel and spatial attention feature maps, respectively; is the matrix dot product.

[0117] The channel attention submodule uses 3D combination to retain information across three dimensions, and then adopts a multilayer perceptron (MLP) to allow feature vectors to fully interact between different dimensions, enabling the model to obtain more nonlinear features and combined feature information; the spatial attention submodule uses two convolutional layers to fuse spatial information, allowing the network to better focus on target areas containing important information.

[0118] After the above steps, the target prediction frame is finally filtered and adjusted in the prediction part to obtain the final camera detection frame.

[0119] S33: Preprocessing the point cloud data to obtain preprocessed point cloud data;

[0120] Furthermore, step S33 is specifically as follows:

[0121] S331: downsampling the point cloud data using a voxel filtering method to obtain downsampled point cloud data;

[0122] Specifically, the original LiDAR point cloud is first downsampled using voxel filtering, reducing the point cloud's computational complexity while preserving its shape and structure. The steps are as follows: The input original point cloud data is completely contained within a cubic grid. Within this grid, a 10cm×10cm×10cm cubic voxel grid is created. The centroid of each voxel is calculated and used to approximate all points within the voxel grid, achieving both downsampling and noise filtering.

[0123] S332: Filter the downsampled point cloud data through a statistical filter to obtain preprocessed point cloud data.

[0124] Specifically, based on the spatial sparsity of outliers, a statistical filter is used to perform statistical analysis on the k-domain of each point in the point cloud data, calculate the average distance from it to all nearby points, and filter out point cloud data that does not meet the requirements based on the statistical information. The specific steps are:

[0125] S3321: Input the point cloud to be processed S0 = {P1, P2, ..., P n}

[0126] S3322: Calculate a point p in a point cloud i (x i ,y i ,z i ,) and the Euclidean distance d of all points in its k-neighborhood ij , and for the distance d ij Find the mean μ ij , the expression is:

[0127]

[0128] S3323: Traverse all points in the point cloud, repeat S3322, and calculate dμ for all points i Form D={dμ i};

[0129] S3324: Calculate the mean μ and variance σ of the set D, expressed as:

[0130]

[0131] S3325: The expression for outlier determination is defined based on the calculated mean μ and variance σ:

[0132] dμ i >μ+εσ

[0133] S3326: When the average distance between a point and all the points in its k-neighborhood satisfies the above expression, the point is considered an outlier.

[0134] S34: Clustering the preprocessed point cloud data using a density clustering algorithm based on a variable radius to obtain a lidar detection frame.

[0135] Furthermore, a density clustering algorithm based on variable radius is proposed to reduce the complexity of the algorithm, quickly select the clustering radius, and complete the clustering processing after point cloud filtering.

[0136] Step S34 is specifically as follows:

[0137] S341: Divide the radius into an initial radius R and a clustering radius, obtain core points, and define parameter ρ as the point cloud density threshold;

[0138] Specifically, in the selection of clustering radius, the radius is divided into initial radius and clustering radius. The initial radius is set when the core point is selected; the clustering radius is the radius used to determine whether it is a clusterable point.

[0139] S342: Use the initial radius R to search, define the parameter ρ to determine the core point in the input point cloud, record the point set with the number of points within the initial radius of each core point greater than ρ as the core point cloud, and calculate the Euclidean distance d = (i, j) between two adjacent points in the core point cloud;

[0140] Specifically, max is the maximum Euclidean distance, d min is the minimum Euclidean distance, and d max The radius of the core point, d min is the point closest to the core point, d1 is the difference between the two, and d2 is the difference between the initial radius and the maximum Euclidean distance;

[0141] d max =Max{d(i,j)|0≤i<n,0≤j<n}

[0142] d min =Min{d(i,j)|0≤i<n,0≤j<n}

[0143] S343: Let d1 = d max -d min , d2=Rd max ;

[0144] S344: If d1 / d2=1, set the cluster radius r=R;

[0145] If d1 / d2>1, then let the cluster radius r=R+d min ;

[0146] If d1 / d2<1, then let the cluster radius r=Rd min ;

[0147] Specifically, when d1 / d2=1, it reflects that the point cloud distribution within the initial radius is relatively uniform, but not necessarily dense, so the clustering radius at this time selects the initial radius.

[0148] When d1 / d2>1, it is considered that the point cloud distribution within the initial radius is relatively scattered, and the neighboring point with the maximum distance is close to the initial radius. At this time, the initial radius is only slightly larger than the core radius that makes it a core point. It cannot contain all the points of the same cluster point cloud. The clustering radius should be expanded. Therefore, the clustering radius is the initial radius plus the minimum distance, that is, r=R+d min .

[0149] When d1 / d2 is less than 1, it is considered that the point cloud within the initial radius is densely distributed around the core point. At this time, if the initial radius is set as the clustering radius, it will cause two or even multiple clusters to merge. Therefore, the clustering radius should be reduced, that is, r = Rd min .

[0150] S345: After accurately selecting the cluster radius r through the above steps, cluster processing is performed on the pre-processed point cloud data to obtain the lidar detection frame.

[0151] S4: Fuse the camera detection frame and the lidar detection frame to obtain the current coordinates of the target ship.

[0152] Furthermore, the time synchronization and spatial matching of the LiDAR and visible light camera are first completed. The specific steps include the following:

[0153] As for time synchronization, the data acquisition frequency of the laser radar used in this design invention is 5-20hz, and the camera frame rate is 30fps. The two sensor data are not synchronized in time. The nearest neighbor time synchronization method is adopted, and the laser radar data with a lower data acquisition frequency is used as the benchmark. When the laser radar completes a sampling, the image data at the nearest neighbor moment to that moment is found, thus completing the time synchronization of the two data.

[0154] The spatial alignment process includes camera intrinsic calibration and combined extrinsic calibration of the LiDAR and camera.

[0155] Step 1: Use Zhang Zhengyou calibration algorithm to calibrate the camera intrinsic parameters. Take 20 to 25 images of a planar target at different angles with the camera, calculate and analyze the corner points of the checkerboard, and solve to obtain the camera's intrinsic parameter matrix. The process formula is:

[0156]

[0157] Among them, O c -X c Y c Z c is the 3D camera coordinate system, is the camera intrinsic parameter matrix, f represents the focal length of the camera, dx and dy represent the pixel conversion units along the x-axis and y-axis respectively, and (u0, v0) represents the coordinates of the origin of the image coordinate system (i.e., the optical center) in the pixel coordinate system.

[0158] Step 2: After completing the camera's intrinsic calibration, in order to achieve the data obtained by the two sensors being expressed in the same coordinate system, the present invention uses the laser radar and camera joint calibration tool in Autoware for calibration, setting the laser radar coordinate system to the world coordinate system, expressed as XYZ, and the camera coordinate system in space O c -X c Y c Z c The conversion relationship to the lidar coordinate system is:

[0159]

[0160] Combined with the camera intrinsic parameter matrix obtained in step 2, the conversion relationship from the lidar coordinate system to the pixel coordinate system is obtained as follows:

[0161]

[0162] Where R is the rotation vector and T is the translation vector.

[0163] refer to Figure 4 , step S4 is specifically as follows:

[0164] S41: Calculate the area S of the camera detection frame c and the area S of the lidar detection frame l ;

[0165] Specifically, the area of ​​the camera detection frame is represented by the total value of the pixels in the target area, and the formula is:

[0166]

[0167] Among them, x i y i It is the pixel value within the area where the camera detection frame is located.

[0168] The area formula of the lidar detection frame is:

[0169] S l =|y max -y min ||z max -z min |

[0170] where y max 、y min are the maximum and minimum vertex values ​​of the y-axis in the 3D target detection frame, and z max 、z min are the maximum and minimum vertex values ​​of the z-axis in the 3D target detection box, respectively.

[0171] S42: Area S of the camera detection frame c The overlap degree is calculated by comparing the area S1 of the laser radar detection frame projected onto the image. Calculation of

[0172] Specifically, if there is no intersection between the lidar detection frame and the camera detection frame, their respective detection results should be retained and they should be determined to be different maritime targets. If there is an intersection between the lidar detection frame and the camera detection frame, the overlap of the two detection frames needs to be calculated to determine whether they are the same maritime target.

[0173] For each lidar detection frame projected onto the image, quickly search and query the nearest camera detection frame by constructing a KD-tree;

[0174] To comprehensively determine the degree of overlap between the lidar detection frame and the camera detection frame, the overlap formula is defined as:

[0175]

[0176] in, is the overlapping area.

[0177] S43: If If the error is less than 50%, it is considered that the detection objects of the two detection frames are not the same target, and information fusion cannot be performed. The detection results of the visual camera and the lidar are output separately;

[0178] like If the detection rate is greater than or equal to 50%, the detection objects of the two detection frames are considered to be the same target, and the camera detection frame and the lidar detection frame are fused to output the current coordinates of the fused target ship.

[0179] See Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically includes: a multi-sensor adaptive tugboat autonomous escort perception device 401, a processor 402 and a storage medium 403.

[0180] A multi-sensor adaptive tugboat autonomous escort perception device 401: The multi-sensor adaptive tugboat autonomous escort perception device 401 implements the multi-sensor adaptive tugboat autonomous escort perception method.

[0181] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the multi-sensor adaptive tugboat autonomous escort perception method.

[0182] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the multi-sensor adaptive tugboat autonomous escort perception method.

[0183] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0184] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.

[0185] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-sensor adaptive tugboat autonomous escort perception method, characterized in that: include: S1: Obtain the distance d1 between the tugboat and the target vessel at time t1. If d1 is less than the threshold D1 and the AIS and K-band radar targets are successfully matched, proceed to step S2. Otherwise, repeat step S1. S2: Acquire the K-band radar target data of the target ship through AIS and K-band radar fusion matching, and determine whether the visual camera detects the target. If the target is detected, proceed to step S3, otherwise repeat step S2; S3: Obtain the distance d2 between the tugboat and the target vessel at time t2; if d2 is less than the threshold D2, the visual camera and lidar perform synchronous data acquisition of the target vessel under the guidance of the K-band radar target data, obtain the camera detection frame and the lidar detection frame respectively, and proceed to step S4; Otherwise, return to step S2; S4: Fuse the camera detection frame and the lidar detection frame to obtain the current coordinates of the target ship.

2. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 1 is characterized in that: The target matching process between AIS and K-band radar is as follows: S11: Obtain the distance d of the target ship relative to the tugboat through AIS, GPS and fiber optic compass AIS , azimuth and speed S12: Get the distance of the u-th K-wave target through K-band radar Azimuth and speed S13: Pass d AIS 、 and Calculate the matching error e of the TTM data of the u-th K-wave target u ; S14: Repeat steps S12-S13 to obtain the matching errors of the TTM data of all K-wave targets, and take the TTM data with the smallest matching error as the possible target; S15: If the matching error of the possible target is less than 10%, it is determined that the AIS and the K-band radar target are successfully matched, and the successfully matched TTM data is the K-band radar target data of the target ship. Otherwise, it is determined that the AIS and the K-band radar target are not successfully matched.

3. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 2, characterized in that: The calculation formula of the matching error E of the possible target is: Among them, w d is the weight of the distance error, w v is the weight of the velocity error, d AIS-x and d AIS-y d AIS The x-axis and y-axis components, ν AIS-x and ν AIS-y They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of and They are The x-axis component and y-axis component of .

4. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 1, characterized in that: Step S3 is specifically as follows: S31: At the same time, under the guidance of the K-band radar target data, the image data of the target ship is obtained through the visual camera, and the point cloud data of the target ship is obtained through the lidar; S32: Input the image data into the improved YOLO v7 model to calculate the camera detection frame; S33: Preprocessing the point cloud data to obtain preprocessed point cloud data; S34: Clustering the preprocessed point cloud data using a density clustering algorithm based on a variable radius to obtain a lidar detection frame.

5. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 4 is characterized by: The improved YOLO v7 model includes: MobileNetV3 feature extraction network and GAM attention module; The MobileNetV3 feature extraction network includes sequentially connected convolutional layers, the inverse residual structure of the MobileNetV2 linear bottleneck, depthwise separable convolutional layers, the SE-Net attention model, and the hard-swish activation function module; The GAM attention module includes: channel attention submodule and spatial attention submodule; The hard-swish activation function module, channel attention submodule and spatial attention submodule are connected in sequence.

6. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 4, characterized in that: Step S33 is specifically as follows: S331: downsampling the point cloud data using a voxel filtering method to obtain downsampled point cloud data; S332: Filter the downsampled point cloud data through a statistical filter to obtain preprocessed point cloud data.

7. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 4, characterized in that: Step S34 is specifically as follows: S341: Divide the radius into an initial radius R and a clustering radius, obtain core points, and define parameter ρ as the point cloud density threshold; S342: Record the point set with the number of points greater than ρ within the initial radius of each core point as the core point cloud, and calculate the Euclidean distance d, d between two adjacent points in the core point cloud. max is the maximum Euclidean distance, d min is the minimum Euclidean distance; S343: Let d1 = d max -d mun , d2=Rd max ; S344: If d1 / d2=1, set the cluster radius r=R; If d1 / d2>1, then let the cluster radius r=R+d min ; If d1 / d2<1, then let the cluster radius r=Rd min ; S345: Cluster the pre-processed point cloud data using a clustering radius r to obtain a lidar detection frame.

8. The multi-sensor adaptive tugboat autonomous escort perception method according to claim 1, characterized in that: Step S4 is specifically as follows: S41: Calculate the area S of the camera detection frame c and the area S of the lidar detection frame l ; S42: Area S of the camera detection frame c and the area S of the lidar detection box projected onto the image I The degree of overlap is calculated based on the degree of coincidence Calculation of S43: If If the error is less than 50%, it is considered that the detection objects of the two detection frames are not the same target, and information fusion cannot be performed. The detection results of the visual camera and the lidar are output separately; like If the detection rate is greater than or equal to 50%, the detection objects of the two detection frames are considered to be the same target, and the camera detection frame and the lidar detection frame are fused to output the current coordinates of the fused target ship.

9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a multi-sensor adaptive tugboat autonomous escort perception method as described in any one of claims 1 to 8.

10. A multi-sensor adaptive tugboat autonomous navigation sensing device, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a multi-sensor adaptive tugboat autonomous escort perception method as described in any one of claims 1 to 8.

Images