Unmanned ship environment sensing method and system based on deep learning

Abstract

The invention relates to an unmanned ship intelligent sensing technology, in particular to an unmanned ship environment sensing method and system based on deep learning, and the system comprises a data collection module, a target detection module, a target tracking module, a target distance measurement module and a target search module. Space-time synchronization of data is carried out by a pan-tilt camera and a laser radar, an improved YOLOv8 network model is adopted by a target detection module, deformable convolution is introduced into a backbone network, and void space pyramid pooling is introduced into a neck layer. The system provides more accurate environment perception information for a decision-making and control system of the unmanned ship, the feature representation capability is improved, the omission ratio is reduced, high-precision feature extraction is ensured through memory access and optimization of a calculation process, meanwhile, the forward calculation speed of the model is remarkably improved, and calculation redundancy and real-time performance reduction are avoided.

Description

Technical Field

[0001] The present invention relates to intelligent perception technology for unmanned boats, and in particular to an unmanned boat environment perception method and system based on deep learning. Background Art

[0002] In recent years, with the development of artificial intelligence, deep learning, and automatic control technologies, unmanned aerial vehicles (UAVs) have been widely used in ocean patrols, environmental monitoring, water search and rescue, border enforcement, and other fields. However, to achieve autonomous navigation and mission execution, UAVs must possess efficient environmental perception capabilities. This means being able to detect, track, and identify targets in the water in real time, while also understanding the surrounding environment, such as shore and berth information, to enable autonomous decision-making.

[0003] Traditional target detection algorithms suffer from inaccurate perception of static convolutional structures and dynamically deforming targets. In complex, unstructured scenes such as strong light reflections, water mist interference, and dynamically changing wave textures, they also suffer from weak feature representation capabilities and high missed detection rates. Furthermore, traditional processing systems often employ a staged processing framework, with target detection, tracking, and berth segmentation functions implemented in separate modules. This leads to computational redundancy and reduced real-time performance. To address this, the inventors proposed a deep learning-based environmental perception method and system for unmanned underwater vehicles. Summary of the Invention

[0004] The purpose of the present invention is to provide an unmanned boat environment perception method and system based on deep learning, which provides more accurate environmental perception information for the decision-making and control system of the unmanned boat through precise target detection, target tracking, target search, distance estimation and berth segmentation, thereby improving the adaptability and safety of the unmanned boat in unstructured complex scenes, improving the feature characterization capability, and reducing the missed detection rate. Through the optimization of memory access and calculation process, while ensuring high-precision feature extraction, the forward calculation speed of the model is significantly improved, avoiding calculation redundancy and real-time degradation, and solving the technical problems raised in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: a method and system for unmanned boat environment perception based on deep learning, the system comprising a data acquisition module, a target detection module, a target tracking module, a target ranging module and a target search module. The data acquisition module acquires environmental data by multi-sensor fusion, and the pan-tilt camera and lidar perform spatiotemporal synchronization of the data. The target detection module adopts an improved YOLOv8 network model, introduces deformable convolution (DCNv4) in the backbone network, introduces atrous spatial pyramid pooling (ASPP) in the neck layer, performs target detection in RGB images, and outputs target category and bounding box information in the image. The target tracking module selects the specified tracking algorithm according to the target state and outputs the real-time position of the target in the image. The selection methods of the tracked target include manual box selection and YOLO prediction box selection. The target ranging module combines the target detection results and the depth information of the lidar, and obtains the point cloud cluster of the target through filtering algorithm and clustering algorithm to calculate the center point coordinates and distance of the target. The target search module combines the target detection results and the structural similarity algorithm. After category filtering, it crops each target in the detection result and calculates the structural similarity with the search target in the sample library. The most suitable target is obtained by comparison and the bounding box information of the target in the image is output;

[0006] The specific steps of this method are as follows: when performing environmental perception, after being processed by the data acquisition module, the target detection module first infers the image to identify the target category and bounding box information in the image, and then calls the target tracking module, target ranging module, target search module or berth segmentation module according to the mission requirements of the unmanned boat. During use, each module can obtain the results of the target detection module or its intermediate information to improve the functions of each module, and output the results to the visualization interface and decision-making end.

[0007] Preferably, the hypothetical output of the deformable convolution (DCNv4) in the target detection module is y(p0), and its specific formula is as follows:

[0008]

[0009] Among them, p0 is the output position, For all fixed sampling positions on the convolution kernel, w(p n ) is the corresponding sampling position p n The convolution kernel weight, Δp(p0,p n ) is the sampling point p at position p0 n The learned offset, m(p0,p n ) is the modulation coefficient learned for each sampling point, where the modulation coefficient is guaranteed to be in the range of [0,1] by the sigmoid function, and the denominator Used to normalize the modulation coefficient to prevent scale offset problem, Δp(p0,p n ) and m(p0,p n ) is as follows:

[0010]

[0011] in, and The sub-networks used to predict offset and modulation respectively, σ(·) is the sigmoid activation function, Θ offset and Θ mod are the corresponding network parameters.

[0012] Preferably, the hypothetical output feature map of the Atrous Spatial Pyramid Pooling (ASPP) in the object detection module is Y, and its specific formula is as follows:

[0013]

[0014] in, is the input feature map, Conv 1×1 (X) is a 1×1 convolution branch, is the dilated convolution branch with a dilation rate of r i , GlobalAvgPool(X) is the global average pooling branch (followed by a 1×1 convolution), Concat(·) is to concatenate the outputs of each branch in the channel dimension, and f(·) is the final fusion layer (1×1 convolution) and is used to integrate the information of each branch.

[0015] Dilated convolution The specific formula is as follows:

[0016]

[0017] Where y(i) is the value of the output feature map at position i, x(k) is the value of the input feature map at position i, w(k) is the value of the convolution kernel (weight) at position k, K is the size of the convolution kernel, r is the expansion rate, which indicates the interval between each element, and i and k are spatial coordinate indices;

[0018] The specific formula of global average pooling GlobalAbgPool(·) is as follows:

[0019]

[0020] Among them, y c is the output value of the cth channel, x c,i,j is the value of the input feature map at the cth channel and position (i, j), H and W are the height and width of the feature map respectively.

[0021] Preferably, the target selection mode of the target tracking module includes manual frame selection and YOLO prediction frame selection;

[0022] In manual selection mode, users can drag the mouse to select the target area in the GUI interface. The system converts the pixel coordinates of the selected area into the normalized image coordinate system, initializes the tracking algorithm based on the center point and width and height parameters, and triggers the anomaly detection mechanism to prevent the selected area from exceeding the image boundary or being too small.

[0023] In YOLO prediction selection mode, based on the detection results of the target detection module, high-confidence targets are prioritized or targets are screened together through category filtering and the target search module. The deep features of the target are then extracted and combined with the tracking algorithm to generate an appearance embedding vector and assign tracking.

[0024] Preferably, the target ranging module selects a ranging target after detecting the target, and the specific steps are as follows:

[0025] ① Obtain the target's two-dimensional bounding box through the target detection module and map it to the lidar point cloud coordinate system;

[0026] ②Within the projection range of the bounding box, intercept the original point cloud data within the bounding box and remove highly abnormal points through straight-through filtering;

[0027] ③ Perform Euclidean clustering on the intercepted point cloud clusters and set the cluster distance threshold. If the distance between points in the same cluster is less than the threshold, they are classified as the same target point cloud. Otherwise, they are divided into different clusters. The specific formula is as follows:

[0028]

[0029] ④ Use statistical filtering to remove outliers and remove points that deviate too much from the mean to achieve noise removal, and then calculate the three-dimensional center point (x c ,y c ,z c ), the specific formula is as follows:

[0030]

[0031] Among them, N is the point cloud cluster point value, (x i ,y i ,z i ) are the coordinate values ​​of each point;

[0032] ⑤ Calculate the target distance. Assuming that the target and the unmanned boat are on the same horizontal plane, a simplified Euclidean distance formula is used. The specific formula is as follows:

[0033]

[0034] When considering the height difference caused by waves, the pitch angle is introduced to correct the sensor coordinate system offset caused by the ups and downs of the unmanned boat. The pitch angle θ of the unmanned boat is obtained in real time through the IMU and aligned with the timestamp of the lidar point cloud data. The actual distance formula after correcting the pitch angle θ is as follows:

[0035]

[0036] Preferably, the target search module selects a target in the sample library and performs a position search in the image, and the specific steps are as follows:

[0037] ① Obtain the bounding box of the target to be matched through the target detection module, and first filter out categories other than the search target category through category filtering;

[0038] ②Crop the target in the bounding box, save the cropped target image to a temporary directory, and uniformly scale it to the same resolution as the sample library to eliminate scale differences;

[0039] ③ Compare the structural similarity of the cropped images with the search targets in the sample library one by one. The specific formula is as follows:

[0040]

[0041] Among them, μ x and μ y are the pixel means of the cropped image and the target image, σ x and σ y is the standard deviation between the cropped image and the target image, σ xy is the covariance value of the two images, C1=(K1L) 2 and C2=(K2Lf 2 is a constant value, K1 is usually 0.01, K2 is usually 0.03, L is the pixel dynamic range (for 8-bit images, L = 255), and the range of SSIM is [-1, 1]. The closer the value is to 1, the more similar the structures of the two images are.

[0042] ④ Output the target information with the highest similarity to the unmanned boat system to help the unmanned boat determine the location and direction of the search target.

[0043] Preferably, the system also includes a berth segmentation module, which combines the target detection results and the waterside segmentation algorithm to extract the RoI area through the target detection results. The YOLOv8-seg model is used in the waterside segmentation algorithm, which can share feature maps with the target detection algorithm, and introduces a lightweight segmentation head based on deep separable convolution to output mask information of the area where the unmanned boat can be berthed.

[0044] Preferably, the berth segmentation module implements berth segmentation by using a RoI region segmentation technology guided by target detection, and the specific steps include:

[0045] ① Obtain the bounding box of the target to be matched through the target detection module and filter out the "port" category;

[0046] ② The image in its bounding box is cropped and input into the water bank segmentation model (YOLOv8-seg) to generate a segmentation mask. The model shares the feature map with the target detection model and adopts a lightweight segmentation head based on depthwise separable convolution. Assuming that the input is F in ∈R H×W×C , output F out The specific formula is as follows:

[0047] F depth =Depth wiseConv(F in ,kernel=3×3)

[0048] F point =PointwiseConv(F depth ,C out )

[0049] F out =UpSample(F point )+Skip(F encoder )

[0050] Among them, Depth wiseConv (·) is the depthwise convolution value, which performs spatial filtering on each channel of the input feature map independently without mixing channel information; PointwiseConv (·) is the pointwise convolution value, which achieves information fusion between channels through weighted summation across channels with a 1×1 convolution kernel, and adjusts the number of output channels to C out ; UpSample(·) bilinear interpolation upsampling value, Skip(·) is the encoder output feature map F encoder Eigenvalues ​​of skip connections;

[0051] The specific formula of depth wiseConv(·) is as follows:

[0052]

[0053] Among them, y m,i,j is the value of the output feature map at the mth channel and position (i, j), x m,i+k,j+l is the value of the input feature map at the mth channel and position (i+k,j+l), w m,k,l is the value of the convolution kernel of the mth channel at position (k, l), and K is the size of the convolution kernel;

[0054] The specific formula of pointwise convolution PointwiseConv(·) is as follows:

[0055]

[0056] Among them, y n,i,j is the value of the output feature map at the nth channel and position (i, j), x m,i,j is the value of the input feature map at the mth channel and position (i, j), w n,m is the convolution kernel with the weight of the 1×1 convolution kernel between the nth channel and the mth input channel, where M is the number of input channels;

[0057] ③ Perform morphological closing operation on the segmentation results to connect the broken edges, and then smooth the jagged edges through Gaussian filtering, finally providing the system with a polygonal outline with a complete topological structure.

[0058] A method for unmanned boat environment perception based on deep learning and its system method, the specific steps of which are as follows:

[0059] ① When performing environmental perception, after being processed by the data acquisition module, the target detection module first infers the image and identifies the target category and bounding box information in the image;

[0060] ② According to the mission requirements of the unmanned boat, the target tracking module, target ranging module, target search module or berth segmentation module are called. During use, each module can obtain the results of the target detection module or its intermediate information to improve the functions of each module and output the results to the visualization interface and decision-making end.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention provides an unmanned boat environment perception method and system based on deep learning. The system includes a data acquisition module, a target detection module, a target tracking module, a target ranging module and a target search module. The target detection module adopts an improved YOLOv8 network model, introduces deformable convolution (DCNv4) in the backbone network, and introduces atrous spatial pyramid pooling (ASPP) in the neck layer. The network structure of YOLOv8 is improved by deformable convolution (DCNv4) and atrous spatial pyramid pooling (ASPP). DCNv4 introduces learnable offsets so that the sampling points of the convolution kernel can be dynamically adjusted according to the local structure of the input data, thereby enhancing the model's accuracy. The robustness of object deformation, rotation, and scale changes is improved by optimizing memory access and computing processes. While ensuring high-precision feature extraction, the forward computing speed of the model is significantly improved, avoiding the problems of computational redundancy and reduced real-time performance. The ASPP module uses multiple dilated convolutions with different expansion rates on the same input feature map, which not only expands the receptive field of the model, but also can extract rich multi-scale features without significantly increasing the number of parameters. It has good adaptability to targets of various sizes. Through multi-scale information fusion, the model can capture the global semantic information of the entire image, thereby modeling the contextual relationship of the scene, reducing excessive reliance on local features, and avoiding false detections due to abnormal local features.

[0063] 2. The target tracking module in the present invention can select a specified tracking algorithm according to the target state and output the real-time position of the target in the image. The target selection method includes manual frame selection and YOLO prediction frame selection. The target selection mode of the target tracking module includes manual frame selection and YOLO prediction frame selection. The dual-algorithm collaborative mechanism is introduced. The two algorithms can play their respective advantages in the perception chain and achieve information complementarity, ensuring that the unmanned boat obtains more complete environmental information. The feature information after data fusion can be used to build a more accurate environmental perception model, realize more advanced mission planning, path optimization and obstacle avoidance control, and at the same time, in harsh environments (such as fog and wave interference) In this case, a single algorithm may fail due to environmental noise. The collaboration of multiple algorithms can use the strengths of different modules to supplement information, improve the fault tolerance of the overall system, and provide a smarter, safer and more efficient technical guarantee for the unmanned boat to perform complex tasks (such as cruising, search and rescue, monitoring, etc.); therefore, the present invention can be widely used in the environmental perception and decision-making system of the unmanned boat. Through precise target detection, target tracking, target search, distance estimation and berth segmentation, it can provide more accurate environmental perception information for the decision-making and control system of the unmanned boat, improve the adaptability and safety of the unmanned boat in complex environments, improve the feature characterization capability, and reduce the missed detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1is a flow chart of the present invention;

[0065] Figure 2 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] See also Figure 1 The present invention provides an embodiment: an unmanned boat environment perception method and system based on deep learning, the system includes a data acquisition module 1, a target detection module 2, a target tracking module 3, a target ranging module 4, a target search module 5 and a berth segmentation module 6.

[0068] See also Figure 2 , where data acquisition module 1 acquires environmental data in a multi-sensor fusion manner, using spatiotemporal synchronization data from the PTZ camera and the LiDAR. The PTZ camera acquires RGB images, and the LiDAR acquires 3D point cloud data. The acquired data is pre-processed to provide the original environmental information for subsequent modules;

[0069] The target detection module 2 adopts the improved YOLOv8 network model, introduces deformable convolution (DCNv4) in the backbone network, introduces atrous spatial pyramid pooling (ASPP) in the neck layer, performs target detection in RGB images, and outputs the target category and bounding box information in the image.

[0070] The hypothetical output of the deformable convolution (DCNv4) in the target detection module 2 is y(p0), and its specific formula is as follows:

[0071]

[0072] Among them, p0 is the output position, For all fixed sampling positions on the convolution kernel, w(p n ) is the corresponding sampling position p n The convolution kernel weight, Δp(p0,p n ) is the sampling point p at position p0 n The learned offset, m(p0,p n ) is the modulation coefficient learned for each sampling point, where the modulation coefficient is guaranteed to be in the range of [0,1] by the sigmoid function, and the denominator Used to normalize the modulation coefficient to prevent scale offset problem, Δp(p0,p n ) and m(p0,p n ) is as follows:

[0073]

[0074] in, and The sub-networks used to predict offset and modulation respectively, σ(·) is the sigmoid activation function, Θ offset and Θ mod are the corresponding network parameters.

[0075] The assumed output feature map of Atrous Spatial Pyramid Pooling (ASPP) in the target detection module 2 is Y, and its specific formula is as follows:

[0076]

[0077] in, is the input feature map, Conv 1×1 (X) is a 1×1 convolution branch, is the dilated convolution branch with a dilation rate of r i , GlobalAvgPool(X) is the global average pooling branch (followed by a 1×1 convolution), Concat(·) is to concatenate the outputs of each branch in the channel dimension, and f(·) is the final fusion layer (1×1 convolution) and is used to integrate the information of each branch.

[0078] Dilated convolution The specific formula is as follows:

[0079]

[0080] Among them, y(i) is the value of the output feature map at position i, x(i) is the value of the input feature map at position i, w(k) is the value of the convolution kernel (weight) at position k, K is the size of the convolution kernel, r is the expansion rate, which indicates the interval between each element, and i and k are spatial coordinate indices;

[0081] The specific formula of global average pooling GlobalAbgPool(·) is as follows:

[0082]

[0083] Among them, y c is the output value of the cth channel, x c,i,j is the value of the input feature map at the cth channel and position (i, j), H and W are the height and width of the feature map respectively.

[0084] The target tracking module 3 selects a specified tracking algorithm according to the target state and outputs the real-time position of the target in the image. The target selection method includes manual frame selection and YOLO prediction frame selection. The target selection mode of the target tracking module 3 includes manual frame selection and YOLO prediction frame selection;

[0085] In manual selection mode, users can drag the mouse to select the target area in the GUI interface. The system converts the pixel coordinates of the selected area into the normalized image coordinate system, initializes the tracking algorithm based on the center point and width and height parameters, and triggers the anomaly detection mechanism to prevent the selected area from exceeding the image boundary or being too small.

[0086] In YOLO prediction selection mode, based on the detection results of the object detection module 2, high-confidence targets are prioritized or targets are screened together through category filtering and the object search module. The deep features of the targets are then extracted and combined with the tracking algorithm to generate an appearance embedding vector and assign tracking;

[0087] The two modes support dynamic switching. When manual selection is triggered, the system will override the current automatic tracking target; if the tracking confidence is lower than the threshold, it will automatically switch to YOLO mode to recapture the target.

[0088] The target ranging module 4 combines the target detection results and the depth information of the laser radar, and obtains the point cloud cluster of the target through filtering algorithm and clustering algorithm, thereby calculating the center point coordinates and distance of the target. After detecting the target, the target ranging module 4 selects the ranging target. The specific steps are as follows:

[0089] ① Obtain the target 2D bounding box through target detection module 2 and map it to the lidar point cloud coordinate system;

[0090] ② Within the projection range of the bounding box, intercept the original point cloud data within the bounding box and remove highly abnormal points (such as water surface reflection noise) through straight-through filtering;

[0091] ③ Perform Euclidean clustering on the intercepted point cloud clusters and set a cluster distance threshold (such as 0.3m). If the distance between points in the same cluster is less than the threshold, they are classified as the same target point cloud. Otherwise, they are divided into different clusters. The specific formula is as follows:

[0092]

[0093] ④ Use statistical filtering to remove outliers and remove points that deviate too much from the mean to achieve noise removal, and then calculate the three-dimensional center point (x c ,y c ,z c ), the specific formula is as follows:

[0094]

[0095] Among them, N is the point cloud cluster point value, (x i ,y i ,z i ) are the coordinate values ​​of each point;

[0096] ⑤ Calculate the target distance. Assuming that the target and the unmanned boat are on the same horizontal plane, a simplified Euclidean distance formula is used. The specific formula is as follows:

[0097]

[0098] When considering the height difference caused by waves, the pitch angle is introduced to correct the sensor coordinate system offset caused by the ups and downs of the unmanned boat. The pitch angle θ of the unmanned boat is obtained in real time through the IMU and aligned with the timestamp of the lidar point cloud data. The actual distance formula after correcting the pitch angle θ is as follows:

[0099]

[0100] The target search module 5 combines the target detection results and the structural similarity algorithm. After category filtering, it crops each target in the detection results and calculates the structural similarity with the search target in the sample library. By comparison, the most suitable target is obtained and the bounding box information of the target in the image is output. The target search module 5 selects the target in the sample library and searches for the position in the image. The specific steps are as follows:

[0101] ① Obtain the bounding box of the target to be matched through the target detection module 2, and first filter out the categories other than the search target category through category filtering;

[0102] ②Crop the target in the bounding box, save the cropped target image to a temporary directory, and uniformly scale it to the same resolution as the sample library to eliminate scale differences;

[0103] ③ Compare the structural similarity of the cropped images with the search targets in the sample library one by one. The specific formula is as follows:

[0104]

[0105] Among them, μ x and μ y are the pixel means of the cropped image and the target image, σ x and σ y is the standard deviation between the cropped image and the target image, σ xy is the covariance value of the two images, C1=(K1L) 2 and C2=(K2L) 2is a constant value, K1 is usually 0.01, K2 is usually 0.03, L is the pixel dynamic range (for 8-bit images, L = 255), and the range of SSIM is [-1, 1]. The closer the value is to 1, the more similar the structures of the two images are.

[0106] ④ Output the target information with the highest similarity to the unmanned boat system to help the unmanned boat determine the position and direction of the search target. This result can be combined with the target ranging module 4 and the target tracking module 3 to complete other tasks.

[0107] The berth segmentation module 6 combines the target detection results and the waterside segmentation algorithm to extract the RoI region based on the target detection results. The YOLOv8-seg model is used in the waterside segmentation algorithm, which can share feature maps with the target detection algorithm. A lightweight segmentation head based on deep separable convolution is introduced to output the mask information of the unmanned boat's berthing area. The berth segmentation module 6 implements berth segmentation through the RoI region segmentation technology guided by target detection. The specific steps include:

[0108] ① Obtain the bounding box of the target to be matched through the target detection module and filter out the "port" category;

[0109] ② The image in its bounding box is cropped and input into the water bank segmentation model (YOLOv8-seg) to generate a segmentation mask. The model shares the feature map with the target detection model and adopts a lightweight segmentation head based on depthwise separable convolution. Assuming that the input is F in ∈R H×W×C , output F out The specific formula is as follows:

[0110] F depth =Depth wiseConv(F in ,kernel=3×3)

[0111] F point =PointeiseConv(F depth ,C out )

[0112] F out =UpSample(F point )+Skip(F encoder )

[0113] Among them, Depth wiseConv (·) is the depthwise convolution value, which performs spatial filtering on each channel of the input feature map independently without mixing channel information; PointwiseConv (·) is the pointwise convolution value, which achieves information fusion between channels through weighted summation across channels with a 1×1 convolution kernel, and adjusts the number of output channels to C out; UpSample(·) bilinear interpolation upsampling value, Skip(·) is the encoder output feature map F encoder Eigenvalues ​​of skip connections;

[0114] The specific formula of depth wiseConv(·) is as follows:

[0115]

[0116] Among them, y m,i,j is the value of the output feature map at the mth channel and position (i, j), x m,i+k,j+l is the value of the input feature map at the mth channel and position (i+k,j+l), w m,k,l is the value of the convolution kernel of the mth channel at position (k, l), and K is the size of the convolution kernel;

[0117] The specific formula of pointwise convolution PointwiseConv(·) is as follows:

[0118]

[0119] Among them, y n,i,j is the value of the output feature map at the nth channel and position (i, j), x m,i,j is the value of the input feature map at the mth channel and position (i, j), w n,m is the convolution kernel with the weight of the 1×1 convolution kernel between the nth channel and the mth input channel, where M is the number of input channels;

[0120] ③ Perform morphological closing operation on the segmentation results to connect the broken edges, and then smooth the jagged edges through Gaussian filtering, finally providing the system with a polygonal outline with a complete topological structure.

[0121] The specific steps of this method are as follows:

[0122] ① Image acquisition and input. During this phase, the unmanned boat uses a high-definition camera to collect real-time images of the water environment, ensuring that the image quality meets the detection requirements. The collected image data serves as the raw input and is uniformly transmitted to the subsequent processing module through the system.

[0123] ② Image preprocessing: The system normalizes, resizes, and denoises the captured images to reduce the effects of ambient lighting and water surface reflections. The preprocessed images ensure data consistency and robustness during subsequent model inference.

[0124] ③ Object detection and result publication: An improved YOLOv8 object detection model is used to perform inference on preprocessed images. This model has been fine-tuned through transfer learning on a dataset of aquatic targets and can accurately identify targets such as ships, buoys, and obstacles.

[0125] ④ Multi-task data processing. Different modules are selected for data processing based on the mission requirements of the unmanned boat. The target tracking module receives the target category, candidate box, and feature vector output by the target detection module. From the KCF, CSRT, and DeepSORT algorithm libraries, it automatically selects the optimal algorithm based on the target's motion speed (low / high speed) and target type (static / dynamic), achieving stable tracking of the target in the image. The target ranging module extracts corresponding point cloud clusters from the target bounding box based on target detection and lidar data fusion, uses Euclidean clustering filtering to remove noise, calculates the three-dimensional coordinates of the target center point, extracts depth information, and calculates the real-time distance between the target and the unmanned boat using a geometric constraint model. The target search module crops the target detected by the target detection module, uses the SSIM structural similarity algorithm to match it with the sample library, and prioritizes targets with high similarity. The berth segmentation module combines the candidate box output by the target detection module and applies the YOLOv8-seg network to perform semantic segmentation of the nearshore scene within the area, outputting the berth boundary and the coordinates of the safe berthing area.

[0126] 5. Result publishing and interaction. After processing, the perception system transmits the detected target information to the visualization interface and decision-making end. Other systems of the UAV can use these messages to achieve data sharing and system collaboration, ensuring that the UAV can navigate, avoid obstacles, and execute tasks based on real-time target information.

[0127] See again Figures 1 to 2 Combined with the above, it can be seen that the deep learning-based environmental perception method and system for unmanned boats provided by the present invention fully solves problems such as insufficient single-modal perception capabilities, low efficiency of heterogeneous sensor collaboration, and poor reliability of autonomous mooring decisions. Through the coordinated optimization of multimodal data fusion and deep learning algorithms, the system maintains stable performance in extreme environments such as surges and fog, significantly improving the perception accuracy and real-time decision-making of unmanned boats in complex water environments. This technology provides reliable technical support for the large-scale application of unmanned boats in scenarios such as military reconnaissance, marine surveys, and port inspections, and has significant economic and strategic value.

[0128] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A deep learning-based unmanned vehicle environment perception method and system, characterized by: The system includes a data acquisition module (1), a target detection module (2), a target tracking module (3), a target ranging module (4) and a target search module (5). The data acquisition module (1) adopts a multi-sensor fusion method to obtain environmental data, and the pan-tilt camera and the laser radar perform spatiotemporal synchronization of the data. The target detection module (2) adopts an improved YOLOv8 network model, introduces deformable convolution (DCNv4) in the backbone network, introduces atrous spatial pyramid pooling (ASPP) in the neck layer, performs target detection in the RGB image, and outputs the target category and bounding box information in the image. The target tracking module (3) performs spatial synchronization based on the target. The state selects a specified tracking algorithm and outputs the real-time position of the target in the image. The selection method of the tracking target includes manual frame selection and YOLO prediction frame selection. The target ranging module (4) combines the target detection result and the depth information of the laser radar, obtains the point cloud cluster of the target through the filtering algorithm and the clustering algorithm, and calculates the center point coordinates and distance of the target. The target search module (5) combines the target detection result and the structural similarity algorithm. After category filtering, each target in the detection result is cropped, and the structural similarity calculation is performed with the search target in the sample library. The most suitable target is obtained by comparison, and the bounding box information of the target in the image is output; The specific steps of the method are as follows: when performing environmental perception, after being processed by a data acquisition module (1), the target detection module (2) first infers the image to identify the target category and bounding box information in the image, and then calls the target tracking module (3), the target ranging module (4), the target search module (5) or the berth segmentation module (6) according to the mission requirements of the unmanned boat. During use, each module can obtain the result of the target detection module (2) or its intermediate information to improve the function of each module, and output the result to a visualization interface and a decision end.

2. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The hypothetical output of the deformable convolution (DCNv4) in the target detection module (2) is y(p0), and its specific formula is as follows: Among them, p0 is the output position, For all fixed sampling positions on the convolution kernel, w(p n ) is the corresponding sampling position p n The convolution kernel weight, Δp(p0,p n ) is the sampling point p at position p0 n The learned offset, m(p0,p n ) is the modulation coefficient learned for each sampling point, where the modulation coefficient is guaranteed to be in the range of [0,1] by the sigmoid function, and the denominator Used to normalize the modulation coefficient to prevent scale offset problem, Δp(p0,p n ) and m(p0,p n ) is as follows: in, and The sub-networks used to predict offset and modulation respectively, σ(·) is the sigmoid activation function, Θ offset and Θ mod are the corresponding network parameters.

3. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The assumed output feature map of the Atrous Spatial Pyramid Pooling (ASPP) in the target detection module (2) is Y, and its specific formula is as follows: in, is the input feature map, Conv 1×1 (X) is a 1×1 convolution branch, is the dilated convolution branch with a dilation rate of r i , GlobalAvgPool(X) is the global average pooling branch (followed by a 1×1 convolution), Concat(·) is to concatenate the outputs of each branch in the channel dimension, and f(·) is the final fusion layer (1×1 convolution) and is used to integrate the information of each branch; Dilated convolution The specific formula is as follows: Among them, y(i) is the value of the output feature map at position i, x(i) is the value of the input feature map at position i, w(k) is the value of the convolution kernel (weight) at position k, K is the size of the convolution kernel, r is the expansion rate, which indicates the interval between each element, and i and k are spatial coordinate indices; The specific formula of global average pooling GlobalAvgPool(·) is as follows: Among them, y c is the output value of the cth channel, x c,i,j is the value of the input feature map at the cth channel and position (i, j), H and W are the height and width of the feature map respectively.

4. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The target selection mode of the target tracking module (3) includes manual frame selection and YOLO prediction frame selection; In manual selection mode, users can drag the mouse to select the target area in the GUI interface. The system converts the pixel coordinates of the selected area into the normalized image coordinate system, initializes the tracking algorithm based on the center point and width and height parameters, and triggers the anomaly detection mechanism to prevent the selected area from exceeding the image boundary or being too small. In the YOLO prediction frame selection mode, based on the detection results of the target detection module (2), high-confidence targets are prioritized or targets are screened together through category filtering and the target search module. Then, the deep features of the target are extracted and combined with the tracking algorithm to generate an appearance embedding vector and assign tracking.

5. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The target distance measurement module (4) selects a distance measurement target after detecting the target, and the specific steps are as follows: ① Obtain the target two-dimensional bounding box through the target detection module (2) and map it to the laser radar point cloud coordinate system; ②Within the projection range of the bounding box, intercept the original point cloud data within the bounding box and remove highly abnormal points through straight-through filtering; ③ Perform Euclidean clustering on the intercepted point cloud clusters and set the cluster distance threshold. If the distance between points in the same cluster is less than the threshold, they are classified as the same target point cloud. Otherwise, they are divided into different clusters. The specific formula is as follows: ④ Use statistical filtering to remove outliers and remove points that deviate too much from the mean to achieve noise removal, and then calculate the three-dimensional center point (x c ,y c ,z c ), the specific formula is as follows: Among them, N is the point cloud cluster point value, (x i ,y i ,z i ) are the coordinate values ​​of each point; ⑤ Calculate the target distance. Assuming that the target and the unmanned boat are on the same horizontal plane, a simplified Euclidean distance formula is used. The specific formula is as follows: When considering the height difference caused by waves, the pitch angle is introduced to correct the sensor coordinate system offset caused by the ups and downs of the unmanned boat. The pitch angle θ of the unmanned boat is obtained in real time through the IMU and aligned with the timestamp of the lidar point cloud data. The actual distance formula after correcting the pitch angle θ is as follows: The target search module (5) combines the target detection results and the structural similarity algorithm, and after category filtering, crops each target in the detection result, and calculates the structural similarity with the search target in the sample library, obtains the most suitable target through comparison, and outputs the target's bounding box information in the image.

6. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The target search module (5) selects a target in the sample library and performs position search in the image. The specific steps are as follows: ① Obtain the bounding box of the target to be matched through the target detection module (2), and first filter out the categories other than the search target category through category filtering; ②Crop the target in the bounding box, save the cropped target image to a temporary directory, and uniformly scale it to the same resolution as the sample library to eliminate scale differences; ③ Compare the structural similarity of the cropped images with the search targets in the sample library one by one. The specific formula is as follows: Among them, μ x and μ y are the pixel means of the cropped image and the target image, σ x and σ y is the standard deviation between the cropped image and the target image, σ xy is the covariance value of the two images, C1=(K1L) 2 and C2=(K2L) 2 is a constant value, K1 is usually 0.01, K2 is usually 0.03, L is the pixel dynamic range (for 8-bit images, L = 255), and the range of SSIM is [-1, 1]. The closer the value is to 1, the more similar the structures of the two images are. ④ Output the target information with the highest similarity to the unmanned boat system to help the unmanned boat determine the location and direction of the search target.

7. The deep learning-based unmanned vehicle environment perception method and system according to claim 1, characterized in that: The invention also includes a berth segmentation module (6), which combines the target detection result and the water bank segmentation algorithm to extract the RoI area through the target detection result. The water bank segmentation algorithm adopts the YOLOv8-seg model, which can share the feature map with the target detection algorithm, and introduces a lightweight segmentation head based on deep separable convolution to output the mask information of the unmanned boat berthing area.

8. The deep learning-based unmanned vehicle environment perception method and system according to claim 7, characterized in that: The berth segmentation module (6) implements berth segmentation by using the RoI region segmentation technology guided by target detection, and its specific steps include: ① Obtain the bounding box of the target to be matched through the target detection module and filter out the "port" category; ② The image in its bounding box is cropped and input into the water bank segmentation model (YOLOv8-seg) to generate a segmentation mask. The model shares the feature map with the target detection model and adopts a lightweight segmentation head based on depthwise separable convolution. Assuming that the input is F in ∈R G×W×C , output F out The specific formula is as follows: F depth =Depth wiseConv(F in ,kernel=3×3) F point =PointwiseConv(F depth ,C out ) F out =UpSample(F point )+Skip(F encoder ) Among them, Depth wiseConv (·) is the depthwise convolution value, which performs spatial filtering on each channel of the input feature map independently without mixing channel information; PointwiseConv (·) is the pointwise convolution value, which achieves information fusion between channels through weighted summation across channels with a 1×1 convolution kernel, and adjusts the number of output channels to C out ; UpSample(·) bilinear interpolation upsampling value, Skip(·) is the encoder output feature map F encoder Eigenvalues ​​of skip connections; The specific formula of depth wiseConv(·) is as follows: Among them, y m,i,j is the value of the output feature map at the mth channel and position (i, j), x m,i+k,j+l is the value of the input feature map at the mth channel and position (i+k,j+l), w m,k,l is the value of the convolution kernel of the mth channel at position (k, l), and K is the size of the convolution kernel; The specific formula of pointwise convolution PointwiseConv(·) is as follows: Among them, y n,i,j is the value of the output feature map at the nth channel and position (i, j), x m,i,j is the value of the input feature map at the mth channel and position (i, j), w n,m is the convolution kernel with the weight of the 1×1 convolution kernel between the nth channel and the wth input channel, and M is the number of input channels; ③ Perform morphological closing operation on the segmentation results to connect the broken edges, and then smooth the jagged edges through Gaussian filtering, finally providing the system with a polygonal outline with a complete topological structure.

9. A method for unmanned boat environment perception based on deep learning and its system method, the specific steps of which include: ① Image acquisition and input: During this stage, the unmanned boat uses a high-definition camera to collect images of the water environment in real time to ensure that the image quality meets the detection requirements; The collected image data is used as the original input and is uniformly transmitted to the subsequent processing module through the system; ② Image preprocessing: the system normalizes, resizes, and reduces noise on the captured images to reduce the effects of ambient lighting and water surface reflections; The preprocessed images ensure the consistency and robustness of the data during subsequent model inference; ③Target detection and result publishing: using the improved YOLOv8 target detection model to reason about the preprocessed images; The model has been fine-tuned through transfer learning on a water target dataset and can accurately identify targets such as ships, buoys, and obstacles. ④ Multi-task data processing: Different modules are selected for data processing according to the mission requirements of the unmanned boat. The target tracking module receives the target category, candidate box and feature vector output by the target detection module, and automatically selects the optimal algorithm from the KCF, CSRT and DeepSORT algorithm libraries according to the target movement speed (low speed / high speed) and target type (static / dynamic) to achieve stable tracking of the target in the image. The target ranging module is based on target detection and lidar data fusion. It extracts corresponding point cloud clusters in the target bounding box, uses Euclidean clustering filtering to remove noise, calculates the three-dimensional coordinates of the target center point, extracts depth information, and calculates the real-time distance between the target and the unmanned boat in combination with the geometric constraint model. The target search module crops the images of the targets detected by the target detection module and uses the SSIM structural similarity algorithm to match them with the sample library, giving priority to targets with high similarity. The berth segmentation module combines the candidate boxes output by the target detection module and applies the YOLOv8-seg network to perform semantic segmentation of the nearshore scene within the area, outputting the berth boundary and the coordinates of the safe berthing area. ⑤Result release and interaction: After processing, the perception system transmits the detected target information to the visualization interface and decision-making end; other systems of the unmanned boat can use these messages to achieve data sharing and system collaboration, ensuring that the unmanned boat can navigate, avoid obstacles and perform tasks based on real-time target information.

Images