Multi-source information fusion-based tug autonomous navigation accompanying real-time sensing method and equipment
By improving the YOLOv8 model and combining multi-source information fusion technology, the problems of high data fusion cost and poor real-time performance in tugboat accompaniment are solved, and high-precision real-time perception of tugboat autonomous accompaniment is achieved, which improves the recognition accuracy.
Patent Information
- Application Number
- CN202411148003.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-29
AI Technical Summary
The existing tugboat navigation method has high data fusion cost, poor real-time performance, and poor recognition accuracy, especially when the tugboat navigation position is selected.
Using the improved YOLOv8 model, time synchronization and spatial calibration are performed through multi-source information fusion, including visible light image data and lidar point cloud data, MLLA module and RepDSC module enhance model feature attention capability and prediction speed, and design CRCLS loss function to Improve recognition accuracy.
High-precision 3D target frame detection is realized, which improves the real-time perception and accuracy of the tugboat autonomous navigation, reduces the cost of data fusion, and improves the recognition accuracy.
Smart Images

Figure CN119048881A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous tugboat escort, and in particular to a real-time perception method and device for autonomous tugboat escort based on multi-source information fusion. Background Art
[0002] In the existing tugboat escort process, the crew usually relies on experience to select the escort position. However, this method is too dependent on manual labor, time-consuming and labor-intensive, and has a high risk. In the ship detection related technology, the fusion recognition method currently used is mostly for offline data, which has the problems of low accuracy and low real-time performance. In response to these technical challenges, no effective solutions have been proposed, especially when the tugboat selects the escort position. The data fusion cost of the existing escort method is high, the real-time performance is poor, and the recognition accuracy is not ideal. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a real-time perception method and device for tugboat autonomous escort based on multi-source information fusion, which is used to solve the problems of high data fusion cost, poor real-time performance and unsatisfactory recognition accuracy of existing escort methods.
[0004] The present invention provides a real-time perception method for tugboat autonomous escort based on multi-source information fusion, comprising:
[0005] S1: Build an improved YOLOv8 model, train the improved YOLOv8 model through a sample data set, and obtain a trained improved YOLOv8 model;
[0006] S2: The tugboat obtains the visible light image data and the LiDAR point cloud data of the target ship, performs time synchronization and spatial calibration on the visible light image data and the LiDAR point cloud data, and obtains the calibration result;
[0007] S3: Input the visible light image data into the trained improved YOLOv8 model to obtain the visual target detection results, and obtain the frustum point cloud through the visual target detection results, lidar point cloud data and calibration results;
[0008] S4: Cluster the view frustum point cloud to obtain a 3D target frame, and the tugboat accompanies the target ship through the 3D target frame.
[0009] Preferred:
[0010] The improved YOLOv8 model includes: Backbone network, Neck network, Head network, first RepDSC module, second RepDSC module, third RepDSC module and MLLA module;
[0011] The head network includes: a first detection head, a second detection head and a third detection head;
[0012] Backbone network is connected with Neck network;
[0013] The Neck network, the first RepDSC module and the first detection head are connected in sequence;
[0014] The Neck network, the second RepDSC module and the second detection head are connected in sequence;
[0015] The Neck network, the third RepDSC module, the MLLA module and the third detection head are connected in sequence.
[0016] Preferably, step S1 specifically comprises:
[0017] S11: Input the sample data set into the improved YOLOv8 model to obtain the prediction box. According to the intersection and union ratio between the prediction box and the true target box, the prediction box is divided into the positive sample box set Pos and the negative sample box set Neg, and the cross entropy loss is constructed. Among them, n is the index of the sample frame, p n and are the predicted score and true score of sample frame n respectively;
[0018] S12: Construct weighting function w n (x), calculate the first classification loss L cls-1 , the calculation formula is:
[0019]
[0020] Where N is the total number of sample frames;
[0021] S13: Calculate the second classification loss L cls-2 , the calculation formula is:
[0022]
[0023] Among them, i is the index of the positive sample box, M is the total number of positive sample boxes, iou i is the intersection-and-union ratio of the positive sample box i and the true target box, j is the index of the negative sample box, and K is the total number of negative sample boxes;
[0024] S14: Through L cls-1 and L cls-2 Calculate the third classification loss L cls ;
[0025] S15: According to the positive sample frame and the negative sample frame, L cls Split and weight, and calculate the fourth classification loss L cls-pos And the fifth classification loss L cls-neg , L cls-posand L cls-neg Add to get the CRCLS loss function L cls ′;
[0026] S16: During the training process of the improved YOLOv8 model, steps S11-S15 are repeated until the CRCLS loss function reaches a convergence condition or is less than a set threshold, thereby obtaining a trained improved YOLOv8 model.
[0027] Preferred:
[0028] Weighting function w n The calculation formula for (x) is:
[0029]
[0030] Among them, μ is the hyperparameter to be trained, μ∈(0,1).
[0031] Preferred:
[0032] The third classification loss L cls The calculation formula is:
[0033]
[0034] Among them, a and b are hyperparameters to be trained, a, b∈(0,1), and a+b=1.
[0035] Preferably, step S15 is specifically as follows:
[0036] S151: The third classification loss L cls Split into and
[0037] S152: Yes and Weighted calculation to obtain the fourth classification loss L cls-pos , the calculation formula is:
[0038]
[0039] S153: Yes Weighted calculation to obtain the fifth classification loss L cls-neg , the calculation formula is:
[0040]
[0041] S154: CRCLS loss function L cls ′=L cls-pos +L cls-neg .
[0042] Preferably, step S2 specifically comprises:
[0043] S21: Time synchronization is performed on the visible light image data and the lidar point cloud data through the PTP time synchronization protocol to obtain the visible light image data and the lidar point cloud data with the same timestamp;
[0044] S22: Filter and denoise the visible light image data to obtain filtered image data, and calculate the camera intrinsic parameter matrix through the filtered image data;
[0045] S23: Select the point cloud located at the center of the calibration plate in the laser radar point cloud data to obtain the filtered point cloud data, perform joint calibration with the filtered point cloud data and the filtered image data to obtain a joint external parameter matrix, and the calibration result is composed of the camera internal parameter matrix and the joint external parameter matrix.
[0046] Preferably, step S3 specifically comprises:
[0047] S31: Input the visible light image data into the trained improved YOLOv8 model to obtain visual target detection results, including: category one-hot vector and target box boundary;
[0048] S32: Obtain the camera intrinsic parameter matrix and the joint extrinsic parameter matrix of the calibration result, and obtain the view cone point cloud corresponding to the 2D area of interest according to the category unique hot vector, the target box boundary, the lidar point cloud data, the camera intrinsic parameter matrix and the joint extrinsic parameter matrix.
[0049] A storage medium, wherein the storage medium stores instructions and data for realizing the real-time perception method for tugboat autonomous escort based on multi-source information fusion.
[0050] A tugboat autonomous navigation real-time perception device based on multi-source information fusion, comprising: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the tugboat autonomous navigation real-time perception method based on multi-source information fusion.
[0051] The present invention has the following beneficial effects:
[0052] An improved YOLOv8 model is constructed to obtain visual target detection results. The MLLA module and RepDSC module are introduced into the existing YOLOv8 model. The MLLA module can enhance the model's ability to focus on features and improve its recognition accuracy of targets. The RepDSC module effectively reduces the number of model parameters and improves the prediction speed, making the model more efficient. At the same time, a CRCLS loss function is constructed for training the improved YOLOv8 model. The CRCLS loss function can more effectively deal with the sample imbalance problem, improve the model's recognition ability for difficult samples, and enhance the correlation between classification tasks and positioning tasks, thereby improving the overall detection performance. The visual target detection results obtained by improving the YOLOv8 model can obtain a higher-precision 3D target box, realizing high-precision real-time perception during autonomous escort. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;
[0054] Figure 2 To improve the structure diagram of the YOLOv8 model;
[0055] Figure 3 It is the structural diagram of MLLA module;
[0056] Figure 4 is the structural diagram of the RepDSC module;
[0057] Figure 5 Flowchart for time synchronization and storage of visible light image data and lidar point cloud data;
[0058] Figure 6 This is a system structure diagram of an embodiment of the present invention;
[0059] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0060] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0061] Reference Figure 1 The present invention provides a real-time perception method for tugboat autonomous escort based on multi-source information fusion, comprising:
[0062] S1: Build an improved YOLOv8 model, train the improved YOLOv8 model through a sample data set, and obtain a trained improved YOLOv8 model;
[0063] Furthermore, the improvements of the YOLOv8 model are mainly reflected in the following three aspects: (1) Introduction of the MLLA module: The Mamba-Like Linear Attention (MLLA) module is used to combine the Mamba model with the linear attention mechanism, aiming to enhance the model's ability to focus on features and improve its recognition accuracy of targets. (2) Design of the RepDSC module: The depthwise separable convolution (DSC) is introduced to replace the traditional 3×3 convolution, effectively reducing the number of model parameters while improving the prediction speed, making the model more efficient. (3) Optimization of the loss function: A new loss function (Cls-Reg Coupled Loss Function, CRCLS) is designed to solve the sample imbalance problem of the existing loss function when dealing with difficult samples, as well as the low correlation between classification and positioning tasks, enhance the model's attention to difficult samples, and improve the correlation between classification loss and positioning loss. The structure of the improved YOLOv8 model is as follows: Figure 2 shown.
[0064] The improved YOLOv8 model includes: Backbone network, Neck network, Head network, first RepDSC module, second RepDSC module, third RepDSC module and MLLA module;
[0065] The head network includes: a first detection head, a second detection head and a third detection head;
[0066] Backbone network is connected with Neck network;
[0067] The Neck network, the first RepDSC module and the first detection head are connected in sequence;
[0068] The Neck network, the second RepDSC module and the second detection head are connected in sequence;
[0069] The Neck network, the third RepDSC module, the MLLA module and the third detection head are connected in sequence.
[0070] Specifically, (1) MLLA module such as Figure 3 As shown in the figure, the MLLA module combines Mamba with the linear attention mechanism to improve the inference speed of the model. The MLLA module is mainly composed of an input gate, a forget gate, a linear attention mechanism, and a residual connection.
[0071] Input Gate: The input gate module is trained to generate higher input gate values for information-rich areas such as foreground objects, while suppressing less useful areas to generate lower input gate values. Using an input gate can reduce the throughput of the network model.
[0072] Forget gate: The main function of the forget gate is to control the retention and update of information in the network, so that the model can selectively retain or discard information. Since image data is not causal data, there is no need to adopt the traditional recursive design method. In order to reduce complexity and increase computing speed, this module uses Rotary Positional Embedding (RoPE) as an alternative to the traditional forget gate.
[0073] Linear Attention: Use the linear layer to convert the input into query and key, without changing the value. Use the linear attention mechanism to capture the relationship and dependency between different elements in the input, helping the model to fully understand the data.
[0074] Residual connection: Residual connection allows the network to directly transfer information between different layers. At each layer, the input will not only undergo standard forward calculations, but will also be added to the input of the previous layer to ensure the continuity of information and improve the flow and retention of information. In deep networks, as the number of layers increases, the gradient may gradually weaken during the back propagation process, making training difficult, and residual connections can effectively alleviate the problem of gradient disappearance.
[0075] (2) The structure diagram of the RepDSC module is shown in Figure 4 shown.
[0076] A RepDSC structure is proposed, which embeds depthwise separable convolution in RepConv to effectively balance the accuracy and speed of the model. Based on the idea of model reparameterization, this method converts the multi-branch model in the training phase into a single-branch structure in the prediction phase, thereby transferring the high accuracy advantage in training to the high speed in prediction.
[0077] Depthwise convolution is divided into two convolution processes: channel-wise convolution and pointwise convolution. In channel-wise convolution, each convolution kernel is responsible for processing only one channel, so each channel is convolved by only one convolution kernel. The number of feature map channels generated by this process is the same as the number of input channels. The convolution kernel size of pointwise convolution is 1×1×M, where M is the number of channels of the previous layer. The pointwise convolution operation generates a new feature map by weightedly combining all feature maps of the previous layer in the depth direction. The number of convolution kernels determines the number of output feature maps. Compared with traditional 3×3 convolution, depthwise separable convolution significantly reduces the number of parameters and improves the prediction speed and effect of the model.
[0078] Compared with traditional 3×3 convolution, depthwise separable convolution significantly reduces the number of parameters during model reparameterization, thereby improving the prediction speed and effect of the model. The RepDSC structure not only simplifies the number of model parameters, but also ensures that the feature representation obtained from previous training can be efficiently used in the inference stage.
[0079] Furthermore, in order to solve the problem of imbalance between difficult and easy samples in image datasets and the low correlation between classification and localization tasks, the CRCLS loss function (Cls-Reg Coupled Loss Function, CRCLS) was designed.
[0080] In most cases, the number of samples of different categories in the training dataset varies significantly, so there is an imbalance between simple samples and difficult samples. Simple samples are more numerous, easy to identify, and have obvious features, while difficult samples are relatively sparse, difficult to identify, and may have problems such as occlusion, deformation, blur, lighting changes, small size, and low contrast with the background.
[0081] In YOLOv8, the classification loss function used is the standard cross entropy loss (CE), which cannot handle the problem of sample imbalance. In addition, the detection head of YOLOv8 adopts a decoupled design of classification and positioning, resulting in a low correlation between the classification task and the positioning task. Therefore, the use of the CRCLS loss function can more effectively deal with the problem of sample imbalance and improve the model's ability to recognize difficult samples. At the same time, the CRCLS loss function can also enhance the correlation between the classification task and the positioning task, thereby improving the overall detection performance.
[0082] Step S1 is specifically as follows:
[0083] S11: Input the sample data set into the improved YOLOv8 model to obtain the prediction box. According to the intersection and union ratio between the prediction box and the true target box, the prediction box is divided into the positive sample box set Pos and the negative sample box set Neg, and the cross entropy loss is constructed. Among them, n is the index of the sample frame, p n and are the predicted score and true score of sample frame n respectively;
[0084] S12: Construct weighting function w n (x), calculate the first classification loss L cls-1 , the calculation formula is:
[0085]
[0086] Where N is the total number of sample frames;
[0087] Furthermore, for the cross entropy loss (CE), samples near the boundary tend to suffer a large loss. Therefore, it is hoped that the model can learn to optimize these samples and use them more fully to train the network. However, the number of such samples is relatively small; therefore, try to assign higher weights to difficult samples and construct a weighting function w n (x) Emphasize the sample frame at the boundary;
[0088] Weighting function w n The calculation formula for (x) is:
[0089]
[0090] Among them, μ is the hyperparameter to be trained, μ∈(0,1).
[0091] S13: The YOLOv8 model uses a parallel classification subnet and a positioning subnet to complete the classification and positioning tasks respectively. They rely on independent classification loss and positioning loss to train the model. This structure leads to the independence between the classification task and the positioning task, which affects the positioning accuracy of the model. In addition, the weak correlation between the classification task and the positioning task will affect the performance of the model when calculating AP (average precision). To solve this problem, without sacrificing efficiency, the IoU (intersection over union) of the positive sample box is directly used to calculate the second classification loss.
[0092] Calculate the second classification loss L cls-2 , the calculation formula is:
[0093]
[0094] Among them, i is the index of the positive sample box, M is the total number of positive sample boxes, iou i is the intersection-and-union ratio of the positive sample box i and the true target box, j is the index of the negative sample box, and K is the total number of negative sample boxes;
[0095] S14: Through L cls-1 and L cls-2 Calculate the third classification loss L cls ;
[0096] Furthermore, the third classification loss L cls The calculation formula is:
[0097]
[0098] Among them, a and b are hyperparameters to be trained, a, b∈(0,1), and a+b=1.
[0099] S15: According to the positive sample frame and the negative sample frame, L clsSplit and weight, and calculate the fourth classification loss L cls-pos And the fifth classification loss L cls-neg , L cls-pos and L cls-neg Add to get the CRCLS loss function L cls ′;
[0100] Furthermore, simply applying weights to the classification loss of each sample box will change the size of the total loss, destroy the balance between the positive and negative sample losses, and reduce the detection performance; therefore, a normalization strategy is used to weight the classification loss so that the sum of the classification losses of positive and negative samples remains unchanged.
[0101] Step S15 is specifically as follows:
[0102] S151: The third classification loss L cls Split into and
[0103] S152: Yes and Weighted calculation to obtain the fourth classification loss L cls-pos , the calculation formula is:
[0104]
[0105] S153: Yes Weighted calculation to obtain the fifth classification loss L cls-neg , the calculation formula is:
[0106]
[0107] S154: CRCLS loss function L cls ′=L cls-pos +L cls-neg .
[0108] S16: During the training process of the improved YOLOv8 model, steps S11-S15 are repeated until the CRCLS loss function reaches a convergence condition or is less than a set threshold, thereby obtaining a trained improved YOLOv8 model.
[0109] S2: The tugboat obtains the visible light image data and the LiDAR point cloud data of the target ship, performs time synchronization and spatial calibration on the visible light image data and the LiDAR point cloud data, and obtains the calibration result;
[0110] Furthermore, step S2 is specifically as follows:
[0111] S21: Time synchronization is performed on the visible light image data and the lidar point cloud data through the PTP time synchronization protocol to obtain the visible light image data and the lidar point cloud data with the same timestamp;
[0112] Specifically, refer to Figure 5 , time synchronization and storage of visible light image data and lidar point cloud data through the PTP time synchronization protocol includes the following steps:
[0113] Step 211: Using the GPS timestamp as a reference for hardware triggering, synchronizing the time of the computing platform to the GPS time;
[0114] Step 212: Connect the computing platform, the switch, the camera, and the lidar through network cables, and ensure that all four support the PTP protocol;
[0115] Step 213: Make the computing platform a PTP master, make the camera and the lidar a PTP slave, and complete the time synchronization between the camera and the lidar and the computing platform;
[0116] Step 214: Create a top-level folder on the computing platform. If the top-level folder path exists, use the current folder; if the folder path does not exist, create a new folder at the specified path;
[0117] Step 215: Create two thematic folders under the top-level folder path, respectively used to store synchronized image data and point cloud data;
[0118] Step 216: Create subfolders according to time under each thematic folder;
[0119] Step 217: In the subfolder, the image data is saved in png format and the point cloud data is saved in pcd format;
[0120] Step 218: Determine whether the storage time of the subfolder reaches the preset value M hours. If it has reached, perform lossless compression: compress all image data in the subfolders under the image theme folder into mp4 video and delete the atomic folder and original data; compress all point cloud data in the subfolders under the point cloud theme folder into zip compressed packages and delete the atomic folder and original data; if it has not reached, continue to save the synchronized image data and point cloud data in the subfolders;
[0121] The compression operation process is as follows:
[0122] Step 1: Read the number of subfolders under the image theme folder and point cloud theme folder;
[0123] Step 2: Determine whether there are uncompressed subfolders under the topic folder;
[0124] Step 3: Dynamically set the maximum number of threads in the thread pool according to the number of uncompressed subfolders under each thematic folder;
[0125] Step 4: Determine whether the subfolder contains image data or point cloud data, and perform compression operations in the thread pool;
[0126] Step 5: If the subfolder contains image data, use HEVC lossless compression mode to compress multiple png images into mp4 videos and delete the atomic folder and original data; if the subfolder contains point cloud data, use zip lossless compression mode to compress the point cloud data subfolder into a zip compressed package and delete the atomic folder and original data;
[0127] Step 6: After waiting for all threads in the thread pool to complete the compression operation, go to step 1.
[0128] Step 219: Determine whether the number of sub-files under the subject folder has reached a preset value N, where sub-files include sub-folders, mp4 videos, and zip compressed packages; if so, delete the sub-file with the earliest date and go to step 216; if not, go directly to step 216.
[0129] S22: Filter and denoise the visible light image data to obtain filtered image data, and calculate the camera intrinsic parameter matrix through the filtered image data;
[0130] S23: Select the point cloud located at the center of the calibration plate in the laser radar point cloud data to obtain the filtered point cloud data, perform joint calibration with the filtered point cloud data and the filtered image data to obtain a joint external parameter matrix, and the calibration result is composed of the camera internal parameter matrix and the joint external parameter matrix.
[0131] S3: Input the visible light image data into the trained improved YOLOv8 model to obtain the visual target detection results, and obtain the frustum point cloud through the visual target detection results, lidar point cloud data and calibration results;
[0132] Furthermore, step S3 is specifically as follows:
[0133] S31: Input the visible light image data into the trained improved YOLOv8 model to obtain visual target detection results, including: category one-hot vector and target box boundary;
[0134] S32: Obtain the camera intrinsic parameter matrix and the joint extrinsic parameter matrix of the calibration result, and obtain the view cone point cloud corresponding to the 2D area of interest according to the category unique hot vector, the target box boundary, the lidar point cloud data, the camera intrinsic parameter matrix and the joint extrinsic parameter matrix.
[0135] S4: Cluster the view frustum point cloud to obtain a 3D target frame, and the tugboat accompanies the target ship through the 3D target frame.
[0136] Specifically, the DBSCAN clustering algorithm is used to cluster the view frustum point cloud to obtain the 3D target box detection result.
[0137] See also Figure 6 , Figure 6 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a real-time perception device 401 for autonomous tugboat navigation based on multi-source information fusion, a processor 402 and a storage medium 403.
[0138] A real-time perception device 401 for autonomous navigation of a tugboat based on multi-source information fusion: The real-time perception device 401 for autonomous navigation of a tugboat based on multi-source information fusion implements the real-time perception method for autonomous navigation of a tugboat based on multi-source information fusion.
[0139] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the real-time perception method for tugboat autonomous escort based on multi-source information fusion.
[0140] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the real-time perception method for tugboat autonomous escort based on multi-source information fusion.
[0141] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0142] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.
[0143] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A real-time perception method for tugboat autonomous escort based on multi-source information fusion, characterized in that: include: S1: Build an improved YOLOv8 model, train the improved YOLOv8 model through a sample data set, and obtain a trained improved YOLOv8 model; S2: The tugboat obtains the visible light image data and the laser radar point cloud data of the target ship, performs time synchronization and spatial calibration on the visible light image data and the laser radar point cloud data, and obtains the calibration result; S3: Input the visible light image data into the trained improved YOLOv8 model to obtain the visual target detection results, and obtain the frustum point cloud through the visual target detection results, lidar point cloud data and calibration results; S4: Cluster the view frustum point cloud to obtain a 3D target frame, and the tugboat accompanies the target ship through the 3D target frame.
2. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 1 is characterized by: The improved YOLOv8 model includes: Backbone network, Neck network, Head network, first RepDSC module, second RepDSC module, third RepDSC module and MLLA module; The head network includes: a first detection head, a second detection head and a third detection head; Backbone network is connected with Neck network; The Neck network, the first RepDSC module and the first detection head are connected in sequence; The Neck network, the second RepDSC module and the second detection head are connected in sequence; The Neck network, the third RepDSC module, the MLLA module and the third detection head are connected in sequence.
3. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 1 is characterized in that: Step S1 is specifically as follows: S11: Input the sample data set into the improved YOLOv8 model to obtain the prediction box. According to the intersection and union ratio between the prediction box and the true target box, the prediction box is divided into the positive sample box set Pos and the negative sample box set Neg, and the cross entropy loss is constructed. Among them, n is the index of the sample frame, p n and are the predicted score and true score of sample frame n respectively; S12: Construct weighting function w n (x), calculate the first classification loss L cls-1 , the calculation formula is: Where N is the total number of sample frames; S13: Calculate the second classification loss L cls-2 , the calculation formula is: Among them, i is the index of the positive sample box, M is the total number of positive sample boxes, iou i is the intersection-and-union ratio of the positive sample box i and the true target box, j is the index of the negative sample box, and K is the total number of negative sample boxes; S14: Through L cls-1 and L cls-2 Calculate the third classification loss L cls ; S15: According to the positive sample frame and the negative sample frame, L cls Split and weight, and calculate the fourth classification loss L cls-pos And the fifth classification loss L cls-neg , L cls-pos and L cls-neg Add to get the CRCLS loss function L cls ′; S16: During the training process of the improved YOLOv8 model, steps S11-S15 are repeated until the CRCLS loss function reaches a convergence condition or is less than a set threshold, thereby obtaining a trained improved YOLOv8 model.
4. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 3 is characterized by: Weighting function w n The calculation formula for (x) is: Among them, μ is the hyperparameter to be trained, μ∈(0,1).
5. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 3 is characterized by: The third classification loss L cls The calculation formula is: Among them, a and b are hyperparameters to be trained, a, b∈(0,1), and a+b=1.
6. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 5 is characterized in that: Step S15 is specifically as follows: S151: The third classification loss L cls Split into and S152: Yes and Weighted calculation to obtain the fourth classification loss L cls-pos , the calculation formula is: S153: Yes Weighted calculation to obtain the fifth classification loss L cls-neg , the calculation formula is: S154: CRCLS loss function L cls ′=L cls-pos +L cls-neg .
7. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 1 is characterized in that: Step S2 is specifically as follows: S21: Time synchronization is performed on the visible light image data and the lidar point cloud data through the PTP time synchronization protocol to obtain the visible light image data and the lidar point cloud data with the same timestamp; S22: Filter and denoise the visible light image data to obtain filtered image data, and calculate the camera intrinsic parameter matrix through the filtered image data; S23: Select the point cloud located at the center of the calibration plate in the laser radar point cloud data to obtain the filtered point cloud data, perform joint calibration with the filtered point cloud data and the filtered image data to obtain a joint external parameter matrix, and the calibration result is composed of the camera internal parameter matrix and the joint external parameter matrix.
8. The real-time perception method for tugboat autonomous escort based on multi-source information fusion according to claim 1 is characterized in that: Step S3 is specifically as follows: S31: Input the visible light image data into the trained improved YOLOv8 model to obtain visual target detection results, including: category one-hot vector and target box boundary; S32: Obtain the camera intrinsic parameter matrix and the joint extrinsic parameter matrix of the calibration result, and obtain the view cone point cloud corresponding to the 2D area of interest according to the category unique hot vector, the target box boundary, the lidar point cloud data, the camera intrinsic parameter matrix and the joint extrinsic parameter matrix.
9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the real-time perception method for autonomous tugboat escort based on multi-source information fusion as described in any one of claims 1 to 8.
10. A real-time perception device for autonomous tugboat escort based on multi-source information fusion, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the real-time perception method for autonomous tugboat escort based on multi-source information fusion as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Water surface target movement detection and positioning method based on LSTM network fusion
CN117853958A
Multispectral target detection model training method, target detection method and system
CN117911710A
Sensor fusion method based on binocular camera guidance
WO2024114119A1
Cited By
Unmanned vibration roller sensing method, device and system and storage medium
CN119516449A
Unmanned vibrating roller sensing method, device, system and storage medium
CN119516449B
