An unmanned ship autonomous homeward system and method fusing BEV technology and optimized A* algorithm

Abstract

The application comprises two aspects, the first is an unmanned ship autonomous return port system fusing BEV technology and optimized A* algorithm, comprising: a feature extraction module for extracting multi-view image features and IMU inertial data; a camera and IMU calibration module for completing the calibration of internal and external parameters of each camera, the calibration of external parameters among multiple cameras, and the calibration of external parameters between IMU and camera; an IMU branch expansion module comprising an IMU data preprocessing unit, an IMU state vector construction unit, an IMU time synchronization unit and a multi-modal data fusion unit; a BEV environment modeling module for processing multi-view image features fused with IMU data and generating a BEV feature map; a grid map generation module for dividing the BEV feature map into a grid map; and a path planning module adopting an A* algorithm improved based on a dynamic Voronoi field to plan a path according to the grid map; the second is an unmanned ship autonomous return port method based on the above system; and the above advantages improve the perception accuracy and navigation safety of the unmanned ship.

Description

Technical Field

[0001] This invention relates to the technical field of autonomous navigation for unmanned surface vessels (USVs), and in particular to an autonomous return-to-port system and method for USVs that integrates BEV technology and an optimized A* algorithm. Background Technology

[0002] With the rapid development of marine resource development and unmanned systems technology, unmanned surface vessels (USVs) are increasingly widely used in tasks such as marine environmental monitoring, resource exploration, and near-shore patrol. However, achieving efficient and safe autonomous return to port after missions remains a significant technical challenge. Existing environmental perception technologies largely rely on a fusion of vision and lidar, which improves perception accuracy, but the high cost of lidar limits its large-scale application. Pure vision-based solutions, on the other hand, are prone to continuous changes in camera perspective due to the pitch motion of the vessel caused by waves in the water surface environment, making it difficult to stably extract multi-view image features. Existing methods do not fully integrate motion data such as angular velocity and acceleration from the inertial measurement unit (IMU), leading to accumulated image feature alignment errors and affecting the spatiotemporal consistency of BEV feature modeling. Furthermore, the imperfect temporal synchronization, coordinate transformation, and feature-level fusion mechanisms between IMU and visual data further restrict the detection accuracy of dynamic obstacles.

[0003] Traditional BEV generation methods rely on spatial feature extraction from single-frame images, lacking effective utilization of historical temporal information, leading to biased predictions of dynamic obstacle trajectories. Furthermore, feature distortions caused by UAV motion are not optimized through a time encoder, resulting in insufficient robustness of BEV modeling in dynamic scenarios and difficulty adapting to real-time navigation requirements in complex water environments. In addition, the A* algorithm, widely used in path planning, optimizes path length without quantifying the threat level of dynamic obstacles, leading to planned paths that are too close to obstacles, making it difficult to meet the safety requirements of scenarios with multiple boat flows and complex docks. Although reinforcement learning methods have been attempted for path optimization, their high computational cost and long training cycles limit practical applications, and existing technologies lack a risk assessment mechanism based on dynamic Voronoi fields, failing to integrate the impact of obstacle speed, distance, and motion trends on path safety in real time, resulting in insufficient safety in global path planning.

[0004] In summary, existing technologies suffer from several core shortcomings, including an imperfect multimodal data fusion mechanism, poor spatiotemporal consistency of BEV features, and a lack of dynamic risk assessment for path planning. These deficiencies severely restrict the reliability and safety of autonomous return-to-port for unmanned surface vessels (USVs). To address these issues, this invention proposes an autonomous return-to-port system for USVs that integrates BEV technology with an optimized A* algorithm. This system achieves efficient multimodal data fusion by extending the IMU branch and optimizes path planning using a dynamic Voronoi field risk function, thereby significantly improving perception accuracy and navigation safety in complex water environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned defects in the prior art and provide an unmanned surface vessel (USV) autonomous return-to-port system and method that integrates BEV technology and optimized A* algorithm. It achieves efficient fusion of multimodal data through IMU branch extension module and designs a dynamic Voronoi field risk function to optimize path planning, thereby significantly improving the perception accuracy and navigation safety in complex water environment. This enables the USV autonomous return-to-port system to perceive the marine environment in real time, plan the return-to-port path, and improve the USV's environmental perception accuracy and navigation safety.

[0006] To achieve the above objectives, the present invention is accomplished through the following two aspects.

[0007] Firstly, an autonomous return-to-port system for unmanned surface vessels (USVs) integrating BEV technology and an optimized A* algorithm is provided, comprising:

[0008] The feature extraction module is used to capture multi-view images from multiple cameras of the unmanned surface vessel (USV), extract multi-view image features through a neural network model, and collect multiple different IMU inertial data from the USV's inertial measurement unit (IMU).

[0009] The camera and IMU calibration module is used to perform extrinsic and intrinsic parameter calibration for each camera, extrinsic parameter calibration between multiple cameras, and extrinsic parameter calibration between the IMU and the camera.

[0010] The IMU branch extension module includes an IMU data preprocessing unit, an IMU state vector construction unit, an IMU time synchronization unit, and a multimodal data fusion unit. The multimodal data fusion unit is used to map the acquired IMU data to the multi-view image feature space through coordinate system transformation, and then realize the fusion of IMU data with the feature maps of various viewpoints.

[0011] The BEV environment modeling module is used to input multi-view image features and fused IMU data into the BEV environment modeling module to generate a BEV feature map that characterizes the unmanned surface vessel environment.

[0012] The raster generation module is used to divide the BEV feature map generated by the BEV environment modeling module into a raster map of passable areas and obstacle areas.

[0013] The path planning module uses an A* algorithm based on Voronoi fields to plan the unmanned surface vessel's return path to port based on the grid map.

[0014] Preferably, the camera and IMU calibration module is used to perform the following process:

[0015] In the calibration of a single camera, Zhang Zhengyou's calibration method is used. A checkerboard calibration board is used to obtain the coordinates of three-dimensional points and two-dimensional pixels, corner points are identified, homography constraints are established, and the internal and external parameter matrices are solved to achieve the transformation from the image coordinate system to the world coordinate system.

[0016] In the extrinsic parameter calibration between multiple cameras, a checkerboard calibration board is fixed in the shared field of view area of ​​the unmanned surface vessel (USV), and multiple cameras are simultaneously triggered to acquire calibration images. The initial transformation matrix of the coordinate system of each camera is solved by the perspective n-point algorithm. Angular velocity and acceleration data of the IMU and image streams from multiple cameras are acquired simultaneously.

[0017] In the extrinsic parameter calibration between the IMU and the camera, the rotation matrix in the spatial extrinsic parameters between the IMU and the camera is jointly optimized by maximum likelihood estimation. Translation vector Based on rotation matrix Translation vector Further construct the coordinate system transformation matrix T between the IMU and the camera. imu ;

[0018] The IMU branch extension module includes:

[0019] The IMU data preprocessing unit is used to denoise and calibrate multiple different IMU data using Kalman filtering.

[0020] The IMU state vector construction unit is used to construct the IMU total state vector, which is:

[0021]

[0022] Where X I Let X be the state of the IMU at time k. calib_C X is the external parameter for conversion between the IMU and the camera. C The sliding window for cloning includes the local IMU clone state over the past m image times;

[0023] The IMU time synchronization unit is used to synchronize the camera time offset t. dC ,pass To align the camera time with the IMU time;

[0024] The multimodal data fusion unit, respectively, uses the coordinate system transformation matrix T imu The angular velocity and acceleration data of the unmanned surface vessel collected by the IMU are mapped to the feature space of the multi-view camera image. A fully connected layer is used to encode the total state vector of the IMU into a feature vector that matches the number of image feature channels, and then splices and fuses it with the image features of each view along the channel dimension.

[0025] Preferably, the multimodal data fusion unit is used to perform the following process:

[0026] IMU feature vectors are generated using the fully connected layer weight matrix W and the bias vector b:

[0027]

[0028] Next, let dimension C img *1*1 of F imu With dimension C img *H img *W img Image features F img By concatenating the components, we obtain the dimension (C). img +C imu )*H img *W img Fusion feature F fused .

[0029] Preferably, the BEV environment modeling module includes:

[0030] Temporal self-attention unit is used to extract temporal information from historical BEV features through BEV Queries;

[0031] Spatial attention unit, used to extract spatial information from feature maps of multi-view images;

[0032] A 6-layer stacked encoder is used to fuse spatiotemporal features layer by layer to generate the BEV feature map at the current moment;

[0033] The loss function design module is used to evaluate the loss function of the model prediction by adding an IMU-related loss function term;

[0034] Optimized 3D inspection head for:

[0035] By introducing perspective supervision loss L pers Enhance multi-perspective geometric consistency learning capabilities;

[0036] An improved time encoder was used to eliminate feature offsets caused by the motion of the unmanned surface vessel.

[0037] Preferably, the loss function design module is used to process the following steps:

[0038] IMU Feature Fusion Loss Term Among them BEV pred It is a predicted BEV feature that incorporates IMU information. true These are the true BEV features. This loss function term measures the effect of fusing IMU-encoded features with image features, ensuring that IMU information is correctly incorporated into the BEV features;

[0039] Total loss function L total :L total =L det +L seg +L imu-fuse , where L det L seg These are the model's 3D detection loss function and map segmentation loss function, respectively.

[0040] Preferably, the optimized 3D detection head includes:

[0041] The perspective-supervised loss unit is used to set the loss weights of perspective loss and BEV loss to be equal, i.e., μ. bev =μ pers =1. Then L det =μ bev L bev +μ pers L pers , where L bev It is the loss function of the original 3D detection head, L pers It is a loss of perspective;

[0042] The improved time encoder unit performs bilinear distortion on historical BEV features through the transformation matrix T_k^t∈SE(3) and then concatenates them with the current features;

[0043] The residual dimensionality reduction unit performs channel dimensionality reduction processing on the spliced ​​multi-frame features.

[0044] Preferably, the raster image generation module is used to process the following steps:

[0045] Discretize the BEV feature map generated in the BEV environment modeling module into a matrix: G = {g ij};

[0046] Set the grid value based on the environmental perception results: g ij =1 indicates a passable area, g ij =0 indicates an obstacle area;

[0047] The grid status is updated in real time to reflect the dynamic distribution of obstacles.

[0048] Preferably, the path planning module employs an improved evaluation function.

[0049] The function f(n) = g(n) + h(n) + D(n) is used to evaluate the optimal path node of the unmanned surface vessel in a dynamic environment. Here, g(n) is the actual distance cost from the starting point to the current node, h(n) is the heuristically estimated cost from the current node to the target node, and D(n) is a danger function introduced by a Voronoi field, constructed based on a dynamic Voronoi field.

[0050]

[0051] Where, d o (i,j,t) is the distance from the current node to the obstacle, reflecting the spatial relationship between the node and the obstacle; d v (i,j,t) is the distance from the current node to the Voronoi boundary. This distance measures the node's position in the Voronoi diagram and affects the distribution of the potential field. a(t)>0 is a time-dependent function used to control the potential field decay rate. It is a function that controls the maximum range of the potential field, and the distance d from the current node to the obstacle. o (i,j,t) exceeds When the potential field's influence is significantly reduced or disappears, τ is the penalty factor, v obs (t) is the velocity function of the moving obstacles around the unmanned surface vessel.

[0052] Preferably, the path planning module includes the following steps for path planning execution:

[0053] Initialize the open queue and the close queue;

[0054] When iterating through nodes, an evaluation value containing the danger function D(n) is calculated for each adjacent node;

[0055] When r V (i,j,t)>0 or d v When (i,j,t)≥D, the node evaluation value is set to infinity to exclude dangerous areas where the unmanned surface vessel travels.

[0056] Where D is the minimum distance to be maintained from obstacles, set according to the actual sea surface operation scenario of the unmanned surface vessel;

[0057] The final safe path is generated by tracing back the parent node.

[0058] Secondly, a method for autonomous return to port of unmanned surface vessels (USVs) integrating BEV technology and optimized A* algorithm is provided. This method is implemented based on the autonomous return-to-port system for USVs integrating BEV technology and optimized A* algorithm described in the first aspect. The processing steps include:

[0059] Step S1: Multi-source data acquisition and feature extraction

[0060] Multiple cameras are used to acquire images from multiple perspectives, and features are extracted using the MobileNetV3 neural network model. Meanwhile, the angular velocity and acceleration data of the IMU are acquired simultaneously.

[0061] Step S2: Camera and IMU Calibration

[0062] By calibrating the intrinsic and extrinsic parameters of each camera, calculating the extrinsic parameter relationships between multiple cameras using the perspective n-point algorithm, and jointly optimizing the extrinsic parameters between the IMU and each camera using maximum likelihood estimation, the transformation matrix T between the camera coordinate system and the IMU coordinate system is further derived. imu ;

[0063] Step S3: The IMU data is fused with the feature maps of each viewpoint to denoise the IMU data and construct a state vector. After time synchronization, the state vector is transformed by the transformation matrix T. imu The image features are mapped to the feature space of each viewpoint and then stitched together according to the channel dimension to obtain a multi-view feature map that integrates IMU data.

[0064] Step S4: BEV Environment Modeling

[0065] The fused multi-view image feature map data is input into the BEV model, specifically through a six-layer stack of temporal self-attention units and spatial attention units to generate the BEV feature map. This is achieved by adding an IMU feature fusion loss term L. imu-fuse Designing the overall loss function for the BEV model facilitates iterative optimization during training; introducing perspective-supervised loss L into the 3D detector head of the BEV model. pers Enhance multi-view geometric consistency learning capability and improve the time encoder to eliminate feature offset caused by unmanned surface vessel motion;

[0066] Step S5: Dynamic Grid Map Generation. The BEV features are discretized into a grid map, and passable areas and obstacles are marked in real time.

[0067] Step S6: Safe Path Planning

[0068] Based on the raster graph, an optimized A* algorithm is adopted, which introduces a danger function D(n) containing a dynamic Voronoi field to dynamically eliminate dangerous nodes and backtrack to generate a safe return path.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] 1. This invention provides an autonomous return-to-port system and method for unmanned surface vessels (USVs) that integrates BEV technology and an optimized A* algorithm. Its design aims to innovatively apply BEV technology, widely used in the field of autonomous driving in automobiles, to the field of USVs. Addressing the problem of camera perspective changes caused by wave-induced pitch motion during USV operations on the water surface, making it difficult to stably extract multi-view image features, an IMU branch is extended into the BEV model (i.e., an IMU branch extension module is added), fusing inertial IMU data such as the USV's angular velocity and acceleration.

[0071] Meanwhile, a perspective supervision loss L is introduced into the 3D detector head of the BEV model. pers To enhance multi-view geometric consistency learning capabilities and improve the temporal encoder to eliminate feature offsets caused by unmanned surface vessel motion.

[0072] Furthermore, the A* algorithm is optimized by embedding a dynamic Voronoi field and incorporating a velocity penalty factor. This solves the problem that the traditional A* path planning algorithm, which only uses path length as a heuristic function, results in the planned path being too close to obstacles.

[0073] 2. In summary, the present invention provides an autonomous return-to-port system and method for unmanned surface vessels that integrates BEV technology and optimized A* algorithm. By extending the IMU branch to achieve efficient fusion of multimodal data, improving the 3D detection head to enhance multi-view geometric consistency, and designing a dynamic Voronoi field risk function to optimize path planning, the system significantly improves perception accuracy and navigation safety in complex water environments. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of the system structure of an unmanned surface vessel autonomous return-to-port system that integrates BEV technology and optimized A* algorithm provided by the present invention;

[0076] Figure 2 This is a flowchart of a method for an unmanned surface vessel autonomous return-to-port system that integrates BEV technology and optimized A* algorithm, provided by the present invention. Detailed Implementation

[0077] The technical solution of this embodiment of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiment is one embodiment of the present invention, and not all embodiments thereof. Based on this embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] Example 1

[0079] Please see Figures 1 to 2 Embodiment 1 of the present invention provides an unmanned surface vessel autonomous return-to-port system that integrates BEV technology and optimized A* algorithm, comprising:

[0080] The feature extraction module is used to capture multi-view images from multiple cameras on the unmanned surface vessel (USV), extract multi-view image features through a neural network model, and collect multiple different IMU data from the USV's IMU.

[0081] The specific functions of the feature extraction module include the following steps:

[0082] First, six high-definition cameras are deployed on the bow, port foreboard, starboard foreboard, port aftboard, starboard aftboard, and stern of the unmanned surface vessel.

[0083] Then, the multi-view images captured by the multiple cameras of the unmanned surface vessel are used to extract multi-view image features through the MobileNetV3 neural network model.

[0084] Next, the IMU of the unmanned surface vessel is fixed below the deck of the unmanned surface vessel, close to the centerline and center of gravity of the hull, in order to reduce vibration and external interference;

[0085] Finally, multiple different IMU data were collected from the unmanned surface vessel's IMU, including data such as the angular velocity and acceleration of the hull.

[0086] The camera and IMU calibration module is used to complete the intrinsic parameter calibration of a single camera, the extrinsic parameter calibration between multiple cameras, and the extrinsic parameter calibration between the IMU and the camera using the Zhang Zhengyou calibration method.

[0087] The specific functions of the camera and IMU positioning module include the following steps:

[0088] Through the aforementioned calibration of the intrinsic parameters of a single camera on an unmanned surface vessel (USV), using the Zhang Zhengyou calibration method, a set of known three-dimensional spatial points and corresponding two-dimensional pixel coordinates are obtained in each camera using a checkerboard calibration board. Corner points are identified using image detection algorithms, homography constraints are established to describe the mapping relationship between planar points and image points, and the internal and external parameter matrices relative to the hull are solved, thereby realizing the transformation from the image coordinate system to the world coordinate system.

[0089] Multi-camera extrinsic parameter calibration involves fixing a checkerboard calibration plate within the common field of view of the unmanned surface vessel (USV) while simultaneously triggering the acquisition of calibration images from multiple cameras. The initial transformation matrix of the coordinate system of each camera is solved using a perspective n-point algorithm, and the angular velocity and acceleration data of the IMU and the image stream from the multiple cameras are acquired synchronously.

[0090] IMU and camera extrinsic parameter calibration: Joint optimization of the rotation matrix in the spatial extrinsic parameters between the IMU and camera through maximum likelihood estimation. Translation vector Based on rotation matrix Translation vector Further construct the coordinate system transformation matrix T between the IMU and the camera. imu ;

[0091] Therefore, the camera and IMU calibration module is used to calibrate the intrinsic parameters between a single camera, the extrinsic parameters between multiple cameras, and the transformation matrix T between the camera coordinate system and the IMU coordinate system. imu This ensures the spatiotemporal consistency of data fusion between cameras from different perspectives and between cameras and IMUs.

[0092] The IMU branch extension module is a new addition to the BEV environment modeling module. By adding this IMU branch, data such as angular velocity and acceleration can be processed, and the information such as the angular velocity and acceleration of the ship's hull in the IMU data can be used to assist in the estimation of the ship's motion.

[0093] It includes an IMU data preprocessing unit, an IMU state vector construction unit, an IMU time synchronization unit, and a multimodal data fusion unit. The multimodal data fusion unit is used to map the fused IMU data to a multi-view image feature space through coordinate system transformation, thereby realizing the fusion of multiple different IMU data.

[0094] The IMU branch extension module includes:

[0095] The IMU data preprocessing unit is used to denoise and calibrate multiple different IMU data through Kalman filtering. The specific processing procedure is as follows: IMU data (angular velocity, acceleration, etc. of the unmanned surface vessel) is collected at a specific Hz, denoised by Kalman filtering, and calibrated according to the characteristics and installation conditions of the IMU to ensure data accuracy.

[0096] The IMU state vector construction unit is used to construct the total IMU state vector after the IMU data has been filtered, denoised, and calibrated. The total IMU state vector is:

[0097] Where X I Let X be the state of the IMU at time k. calib_C X is the external parameter for conversion between the IMU and the camera. C The sliding window for cloning includes the local IMU clone state over the past m image times;

[0098]

[0099] (in Indicates at t k At any given moment, the IMU moves from the world coordinate system {G} to the IMU local coordinate system I. k The rotation. and Represents the velocity and t of the IMU in the world coordinate system. k Location at any given time. g and b a It is the bias of the gyroscope and accelerometer. and This represents the rigid body transformation between the camera coordinate system {C} and the IMU coordinate system {I}.

[0100] The IMU time synchronization unit is used to synchronize the camera time offset t. dC ,pass This achieves alignment between camera time and IMU time; the specific processing steps are as follows: IMU data and viewpoint image features are synchronized in time: using IMU time as a reference, and based on the camera's time offset t... dC ,pass (t C To achieve time synchronization (for camera-reported time);

[0101] Multimodal data fusion unit, used to transform coordinate system matrix T imu The data such as angular velocity and acceleration of the unmanned surface vessel measured by the IMU are mapped to the feature space of the current frame multi-view image. A fully connected layer is used to encode the total state vector of the IMU into a feature vector that matches the number of feature channels of the current frame multi-view image, and then the vectors are spliced ​​and fused along the channel dimension.

[0102] The multimodal data fusion unit is used to perform the following processes:

[0103] The specific process of fusing the data such as angular velocity and acceleration of the unmanned surface vessel measured by the IMU with the features of one of the viewpoints is as follows:

[0104] IMU feature vectors are generated using the fully connected layer weight matrix W and the bias vector b:

[0105]

[0106] Next, let dimension C img *1*1 of F imu With dimension C img *H img *W img Current frame current viewpoint image features F img By concatenating the components, we obtain the dimension (C). img +C imu )*H img *W img The fusion feature F of the current frame and current viewpoint fused .

[0107] The IMU data is fused with the feature maps of other viewpoints in the current frame using the above fusion operation.

[0108] More specifically, the process by which the multimodal data fusion unit fuses IMU data with multi-view image features of the current frame is as follows:

[0109] First, align the data coordinate systems of different modalities: based on the relationship between the IMU and camera extrinsic parameters. The IMU pose information is converted into a spatial transformation of the image feature map of the current frame and current viewpoint. The IMU pose transformation matrix is ​​T. imu (Constructed using rotation matrices and translation vectors), then for the current frame image feature map F img The transformation of a point (x, y) in space is (x', y') = T. img (x,y) allows IMU information and viewpoint image features to be represented in the same coordinate system;

[0110] Matching IMU data to the dimensions of image features: using a fully connected layer to convert the IMU state vector (speed) acceleration (etc.) are encoded into a feature vector F with the same number of channels as the image feature map. imu If the weight matrix of the fully connected layer is W and the bias vector is b, then... After encoding F imu Dimension C img *1*1;

[0111] F stitches and fuses the IMU data vector with the image data vector of the current frame and current viewpoint: fused =[F img ,F imu ], about to F imu Along the channel dimension and F img splicing. F img Dimension C img *H img *W img ,F imu Dimension C img *1*1, after splicing F fused Dimension is (C img +C imu )

[0112] *H img *W img ;

[0113] The IMU data is fused with the feature maps of other viewpoints in the current frame using the above fusion operation to obtain the multi-view image features of the current frame that incorporate data such as the angular velocity and acceleration of the unmanned surface vessel.

[0114] The BEV environment modeling module is used to input the multi-view image features of the current frame and the fused IMU data into the BEV model to generate a BEV feature map that represents the unmanned surface vessel environment.

[0115] The BEV environment modeling module includes:

[0116] Temporal self-attention unit is used to extract temporal information from historical BEV features through BEV Queries;

[0117] Spatial attention unit, used to extract spatial information from feature maps of multi-view images;

[0118] A 6-layer stacked encoder is used to fuse spatiotemporal features layer by layer to generate the BEV feature map at the current moment;

[0119] When the IMU branch extension module is added to the BEV model, a novel loss function design module is needed to evaluate the accuracy of the BEV model in generating BEV feature maps that characterize the unmanned surface vessel environment. Therefore, an IMU-related loss function term is added to the total loss function of the BEV model.

[0120] The loss function design module is used to handle the following steps:

[0121] IMU Feature Fusion Loss Term Among them BEV pred It is a predicted BEV feature that incorporates IMU information. true These are the true BEV features. This loss function term measures the effect of fusing IMU-encoded features with image features, ensuring that IMU information is correctly incorporated into the BEV features.

[0122] Total loss function L total :L total =L det +L seg +L imu-fuse , where L det L seg These are the model's 3D detection loss function and map segmentation loss function, respectively.

[0123]

[0124] (where C is the number of categories, P is the set of pixels, and y is the number of categories.) c (p) is the real label. (Predicted labels)

[0125] L det =L cls +L loc +L ori +L size +L vel +L pers

[0126] Classification loss Where N is the number of bounding boxes, C is the number of categories, and y ic It is the true value of the c-th class of the i-th bounding box. It is the predicted probability.

[0127]

[0128] Where (x, y, z) are the center coordinates of the 3D bounding box.

[0129] Directional loss

[0130] Where θ is the heading angle of the unmanned surface vessel.

[0131] Size loss

[0132] Where (τ,w,h) are the length, width, and height of the bounding box.

[0133] speed loss

[0134] in It is the predicted speed. That's the actual speed.

[0135] The total loss function is designed based on the integrated detection loss, segmentation loss, and IMU feature fusion loss. By calculating the error between the model's prediction and the true value, it guides the BEV model to optimize its direction during training, thereby improving the perception accuracy and navigation safety of the unmanned surface vessel in complex marine environments. In existing technologies, the 3D detection head of current models lacks sufficient geometric constraints during multi-view projection and lacks the ability to learn geometric consistency across multiple views, reducing the accuracy of segmenting traversable and obstacle areas. Therefore, existing technologies suffer from the core defect of weak geometric constraints in 3D detection heads.

[0136] This embodiment utilizes an optimized 3D detection head to introduce perspective supervision loss L. pers This enhances the ability to learn geometric consistency from multiple perspectives and further eliminates trait shifts caused by the motion of unmanned surface vessels.

[0137] The perspective-supervised loss unit is used to set the loss weights of perspective loss and BEV loss to be equal, i.e., μ. bev =μ pers =1. Then L det =μ bev L bev +μ pers L pers , where L bev It is the loss function of the original 3D detection head, L pers It is a loss of perspective;

[0138] The improved time encoder unit performs bilinear distortion on historical BEV features through the transformation matrix T_k^t∈SE(3) and then concatenates them with the current features;

[0139] The residual dimensionality reduction unit performs channel dimensionality reduction processing on the spliced ​​multi-frame features;

[0140] An improved time encoder was developed to eliminate feature offsets caused by the motion of the unmanned surface vessel.

[0141] More specifically, the BEV environment modeling module performs the following process: After fusing IMU data with image data, a multi-scale feature map of multiple viewpoint images on the unmanned surface vessel, incorporating IMU information, is obtained at time t: in It is the feature of the i-th viewpoint camera, N view This is the total number of views from multiple cameras, and at the same time, the BEV Features B at time t-1 must also be retained. t-1 In each encoder layer, BEV Queries Q is first passed through the TSA (Temporal Self-Attention) module from B... t-1 The temporal information is queried and fused to obtain the corrected BEV Queries Q′. Then, in the same encoder layer, the TSA-corrected BEV Queries Q′ are processed by the SCA (Spatial Attention) module from the multi-camera FeaturesF. t By querying and fusing spatial information, further refined BEV Queries "Q" are obtained. This encoder layer performs fine-tuning (FF) on the refined BEV features "Q" and outputs the result as input to the next encoder layer. This process is repeated for six layers, resulting in six rounds of fine-tuning, to generate the unified BEV feature map B at time t. t Then input the optimized 3D detection head.

[0142] The improved performance of the 3D inspection head helps generate more accurate BEV characterizations, and optimizations have been made to address the low perception accuracy of the original 3D inspection head in complex dock environments:

[0143] To enhance the learning ability of unmanned surface vessels (USVs) to achieve multi-view geometric consistency, thereby improving the perception accuracy of 3D inspection heads in complex scenes, perspective supervision is introduced through auxiliary perspective loss:

[0144]

[0145] The 2D bounding box projected onto the image plane of the k-th camera by the model's predicted 3D bounding box is Proj. k (3D Box), the actual 2D frame is 2D. N views For the number of cameras, IOULoss measures the positional deviation of the 2D bounding boxes, SegLoss is the semantic segmentation loss of the projected region, and γ is the number of cameras. Iou and γ seg These are the weighting coefficients.

[0146] The loss weights for perspective loss and BEV loss are set to be equal, i.e., μ. bev =μ pers =1. Then L det =μ bev L bev +μ pers L pers , where L bev It is the loss function of the original 3D detection head, L pers It's a loss of perspective.

[0147] To eliminate feature offset caused by unmanned surface vessel (USV) motion and ensure spatial alignment of temporal features, a simple twisting and connection strategy is adopted by improving the temporal encoder, given the BEV features B at different frame k. k First, based on the reference frame transformation matrix between frame t and frame k... Where SE(3) is a special Euclidean group, B k Bilinear warp to the current frame as Then, the previous BEV features are concatenated with the current BEV features along the channel dimension, and residual blocks are used for dimensionality reduction.

[0148] The raster generation module is used to divide the BEV feature map generated by the BEV environment modeling module into a raster map of passable areas and obstacle areas.

[0149] The raster image generation module is used to process the following steps:

[0150] The BEV feature map generated by the BEV environment modeling module is discretized into a matrix: G = {g ij};

[0151] Set the grid value based on the environmental perception results: g ij =1 indicates a passable area, g ij =0 indicates an obstacle area;

[0152] Update the grid status in real time to reflect the dynamic distribution of obstacles;

[0153] More specifically, the process is as follows: The bird's-eye view of the unmanned surface vessel (USV) generated by the BEV environment modeling module is divided into several grids, each grid representing a specific area. Based on the USV's environmental information, each grid is marked as a passable area or an obstacle area. The grid map can be represented as follows:

[0154] G={g ij}

[0155] Among them, g ij G represents the grid cell in the i-th row and j-th column. ij =1 indicates the area where the unmanned surface vessel can pass, g ij =0 indicates an obstacle zone where the unmanned surface vessel cannot pass.

[0156] The path planning module uses an A* algorithm based on Voronoi fields to plan the unmanned surface vessel's return path to port based on the grid map.

[0157] The path planning module employs an improved evaluation function f(n) = g(n) + h(n) + D(n) to evaluate the cost of each node and guide the search for the optimal target node on the grid map where the unmanned surface vessel is located. The hazard function D(n) considers dynamic obstacles on the water surface and is constructed based on a dynamic Voronoi field.

[0158]

[0159] Where, d o (i,j,t) is the distance from the current node to the obstacle, reflecting the spatial relationship between the node and the obstacle; d v (i,j,t) is the distance from the current node to the Voronoi boundary. This distance is used to measure the positional relationship of the node in the Voronoi diagram and affects the distribution of the potential field; a(t)>0 is a time-dependent function used to control the decay rate of the potential field. It is a function that controls the maximum range of the potential field, and the distance d from the current node to the obstacle. o (i,j,t) exceeds When the potential field's influence decreases significantly or disappears, τ is the penalty factor, v obs (t) is the velocity function of the moving obstacles around the unmanned surface vessel.

[0160] The path planning module is used for the following path planning execution steps:

[0161] Initialize the open queue and the close queue;

[0162] When iterating through nodes, an evaluation value containing the danger function D(n) is calculated for each adjacent node;

[0163] When rV (i,j,t)>0 or d v When (i,j,t)≥D, the node evaluation value is set to infinity to exclude dangerous areas where the unmanned surface vessel travels.

[0164] Where D is the minimum distance to be maintained from obstacles, set according to the actual sea surface operation scenario of the unmanned surface vessel;

[0165] The final safe path is generated by tracing back the parent node;

[0166] More specifically, the path planning module is used to perform the following processes:

[0167] A basic path planning program was written based on the traditional A* algorithm to achieve basic obstacle avoidance and path planning capabilities.

[0168] By introducing the danger function D(n) of the Voronoi field to optimize the heuristic function, the path planning process can effectively avoid obstacles in a multi-obstacle dock environment, reduce the risk of collision, and ensure the navigation safety of the unmanned surface vessel.

[0169] The traditional A* algorithm guides the search using an evaluation function f(n) = g(n) + h(n), where g(n) represents the actual distance cost from the starting point to the current node, and h(n) is the estimated distance cost from the current node to the target node. During the search, the algorithm compares the f(n) values ​​of each node and selects the optimal path node for expansion. The specific workflow is as follows: Two queues are initialized: an open queue and a closed queue. The open queue stores nodes to be processed, and the closed queue stores processed nodes. The starting node is added to the open queue. The node n with the smallest f(n) value is taken from the open queue as the current node for processing. If the current node n is the target node, the path search is successful, and the path is backtracked according to the pointer. Otherwise, the current node n is moved from the open queue to the closed queue, and the generation of adjacent nodes for the current node n begins. For each adjacent node, the following operations are performed:

[0170] If an adjacent node is impassable (e.g., due to an obstacle) or is already in the closing queue, skip that node.

[0171] If an adjacent node is not in the open queue, add it to the open queue and calculate its g(n), h(n), and f(n) values. At the same time, set the current node n as the parent node of the adjacent node.

[0172] If the adjacent node is already in the open queue, check if the path to the adjacent node through the current node n is better (i.e., if the value of g(n) is smaller). If so, update the values ​​of g(n), h(n), and f(n) of the adjacent node, and set the current node n as the parent node of the adjacent node.

[0173] Repeat the above steps until the open queue is empty (meaning no path can be found) or the target node is found.

[0174] The A* algorithm continuously updates the heuristic function f(n) of nodes, prioritizing the expansion of nodes most likely to be close to the target, thus efficiently finding the optimal path. However, this traditional algorithm only uses path length as the heuristic function, which may cause the planned path to be too close to obstacles, posing a safety hazard to the navigation of unmanned vessels.

[0175] To address the issue of traditional A* algorithm paths being too close to obstacles in unmanned surface vessel (USV) navigation path planning, this paper improves the A* algorithm's evaluation function by using the concept of establishing navigation boundaries based on the Voronoi field algorithm. The definition of Voronoi field algorithm is as follows:

[0176] f(n) = g(n) + h(n) + D(n)

[0177] Where g(n) is the actual distance cost from the starting point to the current node, h(n) is the heuristic estimated cost from the current node to the target node, and D(n) is the danger function introduced by the Voronoi field.

[0178] By incorporating dynamic factors such as the velocity of moving obstacles on the sea surface into the field formula, the definition of the dynamic Voronoi field is as follows:

[0179]

[0180] Where, d o (i,j,t) is the distance from the current node to the obstacle, reflecting the spatial relationship between the node and the obstacle; d v (i,j,t) is the distance from the current node to the Voronoi boundary. This distance is used to measure the positional relationship of the node in the Voronoi diagram and affects the distribution of the potential field; a(t)>0 is a time-dependent function used to control the decay rate of the potential field. It is a function that controls the maximum range of the potential field, and the distance d from the current node to the obstacle. o (i,j,t) exceeds When the potential field's influence decreases significantly or disappears, τ is the penalty factor, v obs (t) represents the speed of the moving obstacles around the unmanned surface vessel.

[0181] During unmanned surface vessel (USV) navigation, it is necessary to consider not only water depth constraints but also to maintain a safe distance between the generated path and obstacles. After incorporating a hazard function, the evaluation function for each grid is defined as follows:

[0182]

[0183] Where D is a fixed safety constraint distance, set according to the actual application scenario, used to determine whether a node is within the safe range.

[0184] The present invention discloses an autonomous return-to-port system for unmanned surface vessels that integrates BEV technology and an optimized A* algorithm. Its advantages are as follows:

[0185] 1. Utilize advanced BEV technology on unmanned surface vessels to construct three-dimensional perception of the water surface environment through a multi-view camera array, thereby enhancing the real-time perception capability of complex scenes.

[0186] 2. To address the surface environment in which unmanned surface vessels operate, an IMU branch is added to the BEV model. The state vector is designed to synchronize spatiotemporal data, and motion parameters are encoded through a fully connected layer and spliced ​​with visual features. An IMU feature fusion loss term is added to achieve deep fusion optimization of IMU and vision, thus solving the interference from water surface ripples.

[0187] 3. Improve the 3D detection head in the BEV model and introduce perspective loss supervision. pers We balance BEV and perspective space feature learning; we improve the temporal encoder and align features through motion compensation to improve dynamic detection accuracy.

[0188] 4. Improve the A* algorithm by incorporating dynamic Voronoi field quantification of obstacle threats and introducing a velocity penalty factor. With distance attenuation coefficient, it actively avoids high-speed obstacles and ensures navigation safety.

[0189] 5. By combining the BEV model with the IMU of the unmanned surface vessel and improving the 3D detection head to generate BEV feature maps representing the environment of the unmanned surface vessel, and using the BEV feature maps as the basis of the raster map, an optimization planning algorithm based on the A* algorithm is proposed to perform the path planning task of autonomous docking of the unmanned surface vessel. This autonomous docking system of the unmanned surface vessel can perceive the marine environment in real time, plan the return path, and improve the environmental perception accuracy, autonomy and safety of the unmanned surface vessel.

[0190] Example 2

[0191] Please see Figures 1 to 2 Embodiment 2 of the present invention provides an autonomous return-to-port method for unmanned surface vessels (USVs) that integrates BEV technology and an optimized A* algorithm. It is implemented based on the autonomous return-to-port system for USVs that integrates BEV technology and an optimized A* algorithm described in Embodiment 1. The processing steps include:

[0192] S1: Multi-source data acquisition and feature extraction

[0193] Multiple cameras capture images from multiple perspectives, and the MobileNetV3 neural network model extracts features. Simultaneously, data such as the angular velocity and acceleration of the IMU are acquired, as detailed below:

[0194] S11: Deploys 6 high-definition cameras on the bow, port foreside, starboard foreside, port aftside, starboard aftside, and stern of the unmanned surface vessel to simultaneously collect multi-view images.

[0195] S12: The MobileNetV3 neural network model is used to extract multi-scale feature maps of images from various viewpoints, and the angular velocity and acceleration data of the IMU are collected.

[0196] S13: Fix the IMU to the centerline below the deck, calibrate it by denoising using Kalman filtering, and output the filtered data.

[0197] S2: Camera and IMU Calibration

[0198] S21: Single-camera intrinsic parameter calibration: Using a checkerboard calibration board, the internal and extrinsic parameter matrices of the camera relative to the hull are solved based on Zhang Zhengyou's calibration method, realizing the transformation from the world coordinate system to the image coordinate system. S22: Multi-camera extrinsic parameter calibration: A checkerboard is fixed in the common viewing area, and the initial transformation matrix between each camera is solved using a perspective n-point algorithm.

[0199] S23: IMU and Camera Extrinsic Parameter Calibration: Synchronously acquire IMU data and image stream, and jointly optimize the rotation matrix. Translation vector Obtain the transformation matrix T imu .

[0200] S3: IMU data is fused with feature maps from various viewpoints.

[0201] S31: Construct the IMU total state vector X, which includes the IMU pose, extrinsic parameters, and clone states, using the following formula:

[0202] S32: Time synchronization: based on time offset t dC Align IMU and camera data.

[0203] S33: Multimodal fusion: via T imu IMU data is mapped to the image feature space, encoded by a fully connected layer, and then concatenated along the channel dimension to generate a fused feature map F. fused .

[0204] S34: The image feature maps from different perspectives at the current time are fused with the IMU data at the current time using the above fusion process to obtain a multi-view fused feature map that incorporates the IMU data at the current time.

[0205] S4: BEV Environment Modeling

[0206] S41: Input the multi-view fusion feature map that incorporates IMU data at the current time into the BEV model.

[0207] S42: The BEV model uses a 6-layer stacked encoder, a temporal self-attention unit to extract historical BEV features, a spatial attention unit to aggregate multi-view features, and generates a BEV feature map for the current time step. Finally, it is input into an optimized 3D detection head.

[0208] S43: The specific optimizations to the 3D inspection head are as follows:

[0209] S44: Introduce perspective-supervised loss: Project the 3D bounding box onto the image planes of each viewpoint and calculate the perspective loss L. pers BEV loss L bev The weights are set to 1:1.

[0210] S45: Improved time encoder: The historical BEV features are bilinearly distorted using the transformation matrix T_k^t∈SE(3), and the motion offset is eliminated by dimensionality reduction after splicing.

[0211] S5: Dynamic Raster Map Construction

[0212] S51: Discretized BEV features are grid matrix G = {g ij The grid size is set according to the ambient resolution. S52: Mark grid status: g ij =1 indicates a passable area, g ij =0 represents the obstacle area, and the dynamic obstacle distribution is updated in real time.

[0213] S6: Safe Path Planning

[0214] S61: Initialize the open queue and the close queue, and set the start point and the target point.

[0215] S62: Compute the node evaluation function f(n) = g(n) + h(n) + D(n), where D(n) is based on the dynamic Voronoi field formula:

[0216]

[0217] S63: Eliminate dangerous nodes: If r V (i,j,t)>0&or d vGiven (i,j,t)≥D, and D(n)=∞, then f(n)=∞, to avoid paths that approach obstacles. Here, D is the minimum safe distance between the unmanned surface vessel and dynamic obstacles, set according to practical applications.

[0218] S64: Backtrack the parent node to generate a safe path and output the optimal trajectory for the unmanned surface vessel to return to port.

[0219] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An autonomous return-to-port system for unmanned surface vessels integrating BEV technology and optimized A* algorithm, characterized in that, It includes: The feature extraction module is used to capture multi-view images from multiple cameras of the unmanned surface vessel (USV), extract multi-view image features through a neural network model, and collect multiple different IMU inertial data from the USV's inertial measurement unit (IMU). The camera and IMU calibration module is used to perform extrinsic and intrinsic parameter calibration for each camera, extrinsic parameter calibration between multiple cameras, and extrinsic parameter calibration between the IMU and the camera. The IMU branch extension module includes an IMU data preprocessing unit, an IMU state vector construction unit, an IMU time synchronization unit, and a multimodal data fusion unit. The multimodal data fusion unit is used to map the acquired IMU data to the multi-view image feature space through coordinate system transformation, and then realize the fusion of IMU data with the feature maps of various viewpoints. The BEV environment modeling module is used to input multi-view image features and fused IMU data into the BEV environment modeling module to generate a BEV feature map that characterizes the unmanned surface vessel environment. The raster generation module is used to divide the BEV feature map generated by the BEV environment modeling module into a raster map of passable areas and obstacle areas. The path planning module uses an A* algorithm based on Voronoi fields to plan the unmanned surface vessel's return path to port based on the grid map. The path planning module employs an improved evaluation function. To evaluate the optimal path nodes for unmanned surface vessels in a dynamic environment, g(n) is the actual distance cost from the starting point to the current node, h(n) is the heuristically estimated cost from the current node to the target node, and D(n) is a danger function introduced by a Voronoi field, constructed based on a dynamic Voronoi field: ， in, It is the distance from the current node to the obstacle, reflecting the spatial relationship between the node and the obstacle; This is the distance from the current node to the Voronoi boundary. This distance is used to measure the positional relationship of the node in the Voronoi diagram and affects the distribution of the potential field. It is a time-dependent function used to control the decay rate of the potential field. It is a function that controls the maximum range of the potential field, representing the distance from the current node to the obstacle. Exceed At this time, the influence of the potential field will be significantly reduced or disappear. As a penalty factor, Let be the velocity function of the moving obstacles around the unmanned surface vessel.

2. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 1, characterized in that, The camera and IMU calibration module are used to perform the following processes: In the calibration of a single camera, Zhang Zhengyou's calibration method is used. A checkerboard calibration board is used to obtain the coordinates of three-dimensional points and two-dimensional pixels, corner points are identified, homography constraints are established, and the internal and external parameter matrices are solved to achieve the transformation from the image coordinate system to the world coordinate system. In the extrinsic parameter calibration between multiple cameras, a checkerboard calibration board is fixed in the shared field of view area of ​​the unmanned surface vessel (USV), and multiple cameras are simultaneously triggered to acquire calibration images. The initial transformation matrix of the coordinate system of each camera is solved by the perspective n-point algorithm. Angular velocity and acceleration data of the IMU and image streams from multiple cameras are acquired simultaneously. In the extrinsic parameter calibration between the IMU and the camera, the rotation matrix in the spatial extrinsic parameters between the IMU and the camera is jointly optimized by maximum likelihood estimation. Translation vector Based on rotation matrix Translation vector Further construct the coordinate system transformation matrix between the IMU and the camera. ; The IMU branch extension module includes: The IMU data preprocessing unit is used to denoise and calibrate multiple different IMU data using Kalman filtering. The IMU state vector construction unit is used to construct the IMU total state vector, which is: ， in Let k be the state of the IMU. This is the external parameter for conversion between the IMU and the camera. The sliding window for cloning includes the local IMU clone state over the past m image times; The IMU time synchronization unit is used to synchronize the camera time offset. ,pass To align the camera time with the IMU time; Multimodal data fusion units, respectively through coordinate system transformation matrices The angular velocity and acceleration data of the unmanned surface vessel collected by the IMU are mapped to the feature space of the multi-view camera image. A fully connected layer is used to encode the total state vector of the IMU into a feature vector that matches the number of image feature channels, and then splices and fuses it with the image features of each view along the channel dimension.

3. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 1 or 2, characterized in that, The multimodal data fusion unit is used to perform the following processes: IMU feature vectors are generated using the fully connected layer weight matrix W and the bias vector b: ， Next, the dimension is of With dimension Image features By piecing them together, we can obtain the dimensions. Fusion characteristics .

4. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 1, characterized in that, The BEV environment modeling module includes: Temporal self-attention unit is used to extract temporal information from historical BEV features through BEV Queries; Spatial attention unit, used to extract spatial information from feature maps of multi-view images; A 6-layer stacked encoder is used to fuse spatiotemporal features layer by layer to generate the BEV feature map at the current moment; The loss function design module is used to evaluate the loss function of the model prediction by adding an IMU-related loss function term; Optimized 3D inspection head for: By introducing perspective supervision loss Enhance multi-perspective geometric consistency learning capabilities; An improved time encoder was used to eliminate feature offsets caused by the motion of the unmanned surface vessel.

5. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 4, characterized in that, The loss function design module is used to handle the following steps: IMU Feature Fusion Loss Term ,in It is a predicted BEV feature that incorporates IMU information. These are the true BEV features. This loss function term measures the effect of fusing IMU-encoded features with image features, ensuring that IMU information is correctly incorporated into the BEV features. Total loss function ,in , These are the model's 3D detection loss function and map segmentation loss function, respectively.

6. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 4, characterized in that, The optimized 3D detection head includes: The perspective-supervised loss unit is used to set the loss weights of perspective loss and BEV loss to be equal, i.e. .but ,in It is the loss function of the original 3D detection head. It is a loss of perspective; The time encoder improvement unit improves upon the transformation matrix. The historical BEV features are bilinearly distorted and then concatenated with the current features. The residual dimensionality reduction unit performs channel dimensionality reduction processing on the spliced ​​multi-frame features.

7. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 1, characterized in that, The raster image generation module is used to process the following steps: Discretize the BEV feature map generated in the BEV environment modeling module into a matrix: Set the grid value based on the environmental perception results: This is a passable area. This is an obstacle area; The grid status is updated in real time to reflect the dynamic distribution of obstacles.

8. The unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm according to claim 1, characterized in that, The path planning module is used for the following path planning execution steps: Initialize the open queue and the close queue; When iterating through nodes, an evaluation value containing the danger function D(n) is calculated for each adjacent node; when or At that time, the node evaluation value is set to infinity to eliminate dangerous areas where the unmanned surface vessel can travel; Where D is the minimum distance to be maintained from obstacles, set according to the actual sea surface operation scenario of the unmanned surface vessel; The final safe path is generated by tracing back the parent node.

9. A method for autonomous return to port of an unmanned surface vessel integrating BEV technology and optimized A* algorithm, characterized in that, It is implemented according to any one of claims 1-8, which is an unmanned surface vessel autonomous return-to-port system integrating BEV technology and optimized A* algorithm, and its processing steps include: Step S1: Multi-source data acquisition and feature extraction. Multiple perspective images are acquired through multiple cameras and features are extracted using the MobileNetV3 neural network model. Meanwhile, the angular velocity and acceleration data of the IMU are acquired simultaneously. Step S2: Camera and IMU Calibration. By calibrating the intrinsic and extrinsic parameters of each camera, the extrinsic parameter relationship between multiple cameras is calculated using the perspective n-point algorithm. The extrinsic parameters between the IMU and each camera are jointly optimized using maximum likelihood estimation, and the transformation matrix between the camera coordinate system and the IMU coordinate system is further obtained. ; Step S3: The IMU data is fused with the feature maps of each viewpoint to denoise the IMU data and construct a state vector. After time synchronization, the data is transformed using a transformation matrix. The image features are mapped to the feature space of each viewpoint and then stitched together according to the channel dimension to obtain a multi-view feature map that integrates IMU data. Step S4: BEV Environment Modeling. The fused multi-view image feature map data is input into the BEV model. Specifically, it is generated by passing through a temporal self-attention unit and a spatial attention unit in sequence, and then stacked through six layers to generate the BEV feature map. An IMU feature fusion loss term is added during this process. Designing the overall loss function for the BEV model facilitates iterative optimization during training; introducing perspective-supervised loss into the 3D detector head in the BEV model. Enhance multi-view geometric consistency learning capability and improve the time encoder to eliminate feature offset caused by unmanned surface vessel motion; Step S5: Dynamic Grid Map Generation. The BEV features are discretized into a grid map, and passable areas and obstacles are marked in real time. Step S6: Based on the grid map, the safe path planning adopts the optimized A* algorithm. By introducing the danger function D(n) with dynamic Voronoi field, dangerous nodes are dynamically eliminated and safe return paths are generated backtracking.

Images