Port unmanned vehicle obstacle detection and identification method based on multi-sensor fusion

Abstract

The wharf unmanned vehicle obstacle detection and identification method based on multi-sensor fusion of the application relates to the technical field of intelligent driving, by setting the maximum threshold and the minimum threshold, comparing the sensor coverage with the maximum threshold and the minimum threshold, executing the sensor increase and decrease strategy, and time synchronizing the sensor data; acquiring the intrinsic matrix, the distortion coefficient and the extrinsic matrix, projecting the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, converting the two-dimensional coordinates in the image data into three-dimensional coordinates under the global coordinate system; labeling the sensor data to obtain the two-dimensional boundary box and the three-dimensional bounding box, and obtaining the category and pose label of the obstacle; building a multi-modal fusion perception model, taking the sensor data and the category and pose label of the obstacle as training data, training the multi-modal fusion perception model; deploying the optimized multi-modal fusion perception model to output the category and pose information of the obstacle, which is suitable for complex driving environment.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion. Background Technology

[0002] In autonomous driving applications, environmental perception systems need to cope with changing weather, lighting conditions, and complex dynamic traffic scenarios. Therefore, combining the advantages of different sensors to improve the accuracy and robustness of the system's environmental perception has become a key research focus. LiDAR and cameras, as two main sensors in autonomous driving perception systems, each possess unique advantages. LiDAR can provide accurate distance information and 3D spatial structure, while cameras can capture rich texture and color information. However, single-sensor perception methods have limitations in practical applications. For example, LiDAR performance is easily affected by rain and snow, while camera perception effectiveness is limited at night or in strong light conditions. Therefore, multi-sensor fusion perception technology has emerged.

[0003] Patent application CN117233791A discloses an unmanned vehicle target detection method based on camera fusion solid-state LiDAR. The method involves setting up a camera and a solid-state LiDAR at four positions: front, rear, left, and right of the vehicle. The cameras and LiDARs at each of the four positions are jointly calibrated, and the image data and point cloud data are spatiotemporally synchronized. The image data and point cloud data are then enhanced. Neural networks are used to extract image features and point cloud features, and the image features and point cloud features at each single position are fused. The fused information from the four positions is then subjected to secondary feature fusion. The extracted fused features are input into a prediction network to identify the type, location, and pose information of obstacles in the mine.

[0004] Existing multi-sensor fusion perception methods are mostly based on simple sensor data registration and fusion strategies, typically employing traditional point cloud and image feature matching, simple geometric transformations, and other methods. However, these methods are prone to problems such as perception blind spots, target recognition errors, or data processing delays when facing dynamic and complex driving environments, making it difficult to meet the real-time and high-precision requirements of autonomous driving. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the prior art. To this end, one objective of this application is to propose a method for obstacle detection and recognition of unmanned vehicles at docks based on multi-sensor fusion, thereby improving the accuracy of obstacle detection for unmanned vehicles.

[0006] One aspect of this application provides a method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion, including:

[0007] Step S100: Set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time.

[0008] Step S200: Obtain the intrinsic parameter matrix and distortion coefficients of the camera, obtain the extrinsic parameter matrix between the lidar and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system.

[0009] Step S300: Mark obstacles in the sensor data to obtain the two-dimensional bounding boxes of obstacles in the image data and the corresponding three-dimensional bounding boxes in the point cloud data. Obtain the category and pose label of the obstacle based on the three-dimensional bounding boxes and obstacle markings.

[0010] Step S400: Build a multimodal fusion perception model, using sensor data and obstacle category and pose labels as training data to train the multimodal fusion perception model;

[0011] Step S500: Convert the trained multimodal fusion perception model into an ONNX model, load the ONNX model into the TensorRT inference engine for acceleration, and obtain the optimized multimodal fusion perception model.

[0012] Step S600: Deploy the optimized multimodal fusion perception model, the unmanned vehicle collects real-time sensor data to detect obstacles, and outputs the category and pose information of obstacles;

[0013] The specific method for setting the maximum and minimum thresholds for sensor coverage, comparing the current sensor coverage with the maximum and minimum thresholds, executing corresponding sensor addition / reduction strategies, acquiring sensor data using the sensor array, and synchronizing the sensor data in time is as follows:

[0014] Step S110: Divide the dock operation area into grid areas, and preset the maximum threshold η of sensor coverage η. max and minimum threshold η min Sensor addition and reduction strategies include addition strategies and reduction strategies;

[0015] The specific addition strategy is as follows:

[0016] Step S111: When the sensor coverage is less than the minimum threshold, calculate the sensor coverage density ρ of each grid in the grid area. grid The sensor coverage density is less than the coverage density threshold ρ. minThe grid is used as a candidate grid;

[0017] Step S112: For each candidate grid, count the number N of existing LiDARs within it. L And the number of cameras N C If the ratio of the number of lidars to the number of cameras Less than the preset ratio threshold α of LiDAR and camera LC If so, prioritize adding LiDAR; otherwise, prioritize adding cameras.

[0018] Step S113: Within the candidate grid, randomly generate M potential sensor locations. For each potential sensor location, calculate the local coverage improvement Δη before and after adding the sensor. increase Select the potential sensor location p that offers the greatest improvement in local coverage. max As an additional sensor location;

[0019] Step S114: Add the newly added sensor locations to the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras;

[0020] Step S115: Repeat steps S111 to S114 until the sensor coverage is greater than or equal to the minimum threshold η. min ;

[0021] The reduction strategy is specifically as follows:

[0022] Step S116: When the sensor coverage rate is greater than the maximum threshold, calculate the average sensor coverage density of all grids within the grid area. and the sensor coverage density ρ of each grid grid Grids with a sensor coverage density greater than the average sensor coverage density are designated as redundant grids.

[0023] Step S117: Calculate the contribution Δη of each sensor within the redundant grid to the local coverage. decrease The contribution refers to the decrease in sensor coverage before and after removing the sensor in the redundant grid;

[0024] Step S118: Select the sensor that contributes the least to the local coverage as the sensor to be removed, remove the sensor from the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras.

[0025] Step S119: Repeat steps S116 to S118 until the sensor coverage is less than or equal to the maximum threshold η. max ;

[0026] Step S120: Collect all sensor data in the dock operation area based on the corresponding sensor addition and reduction strategy, and use a combination of high-precision GPS time synchronization and ROS time synchronization to synchronize all sensor data in time to obtain multi-sensor data.

[0027] The high-precision GPS timing synchronization refers to using a high-precision GPS timing module to provide a unified time reference for all sensors, with each sensor data data accompanied by a GPS timestamp t. GPS , representing the absolute time of sensor data acquisition, the resolution of the GPS timestamp is Δt. GPS .

[0028] ROS time synchronization refers to: within the ROS framework, using the message_filters package to align timestamps of data from different sensors, and defining a synchronization window Δt. sync For sensor data whose timestamp difference is less than the synchronization window, they are considered to have been collected at the same time and are synchronized in time.

[0029] The formula for calculating the synchronization window is: Among them, f L f C The data frequencies of the lidar and camera are Δt, respectively. realtime The time threshold corresponding to the real-time requirements of time synchronization;

[0030] The specific methods for obtaining the camera's intrinsic parameter matrix and distortion coefficients, and for obtaining the extrinsic parameter matrix between the lidar and the camera, are as follows:

[0031] Step S201: Take n1 images of the calibration board from different angles and positions using a camera. In each image of the calibration board, extract the coordinates p of the feature points of the calibration board. img = (ui, vi), calculate the corresponding world coordinates P based on the physical dimensions of the calibration plate. world =(X w ,Y w Z w );

[0032] Step S202: Establish a camera local coordinate system with the camera as the origin. Transform the world coordinates of the calibration board in the world coordinate system to the camera local coordinate system using the camera's extrinsic parameter matrix, and obtain the corresponding coordinates (X, Y, F, Z) of the feature points of the calibration board in the camera local coordinate system. c ,Y c Z c Establish normalized feature point coordinates (ui) n vi n The pinhole model between the camera and the local coordinate system;

[0033] Step S203: Establish a distortion model to map the normalized calibration board feature point coordinates to the distorted feature point coordinates (u d ,v d );

[0034] Step S204: Assign the corresponding coordinates (X, Y, Z) of the feature point to the camera's local coordinate system. c ,Y c Z c The normalized feature point coordinates (ui) are projected from the pinhole model. n vi n Using a distortion model, the normalized feature point coordinates (ui) are... n vi n Converted to distorted feature point coordinates (u) d ,v d The distorted feature point coordinates are converted into actual feature point coordinates based on the camera's intrinsic parameter matrix. These coordinates are then used as the predicted feature point coordinates. The Euclidean distance between the reprojection error and the actual measured feature point coordinates (ui,vi) is calculated as the reprojection error. By minimizing the reprojection error, the camera's intrinsic parameter matrix and distortion coefficients are optimized.

[0035] Step S205: Construct a local laser coordinate system with the lidar as the origin, place a calibration board within the camera's field of view, and use the lidar to scan and obtain the 3D point cloud data of the calibration board in the local laser coordinate system.

[0036] Step S206: For each point P in the 3D point cloud lidar =(X l ,Y l Z l Using the extrinsic parameter matrix T l Transform it to the camera local coordinate system. The extrinsic parameter matrix includes a rotation matrix and a translation vector. Then, use the pinhole model to project the points in the camera local coordinate system onto the calibration board image plane. Find the feature point coordinates corresponding to each point in the 3D point cloud in the calibration board image and establish the correspondence between the 3D point cloud and the feature point coordinates of the calibration board.

[0037] Step S207: The Euclidean distance between the predicted feature point coordinates in the calibration board image and the actual feature point coordinates is used as the projection error. By minimizing the projection error, the optimal extrinsic parameter matrix is ​​obtained.

[0038] The specific method for projecting the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane using intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices, based on point cloud data and image data from sensor data, and converting the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system, is as follows:

[0039] Step S210: Define the origin and coordinate axes of the dock operation area, establish a global coordinate system, establish a local coordinate system for the laser radar with each lidar position as the origin, and establish a local coordinate system for the camera with each camera position as the origin;

[0040] Step S220: The sensor data includes point cloud data measured by the lidar and image data measured by the camera. The three-dimensional point coordinates P of the point cloud data are... lidar Through the external parameter matrix T l =[R l |t l Transform to the camera's local coordinate system to obtain the 3D point coordinates P in the camera's local coordinate system. camera Among them, R l t l These are the rotation matrix and the translation vector, respectively.

[0041] Step S230: Project the 3D point coordinates in the camera's local coordinate system into distorted 2D coordinates on the image plane using the camera's intrinsic parameter matrix and distortion coefficients. d ,v d This converts the distorted two-dimensional coordinates into two-dimensional coordinates (ui, vi);

[0042] Step S240: For each two-dimensional coordinate (uj, vj) in the image data, transform it into a normalized two-dimensional coordinate (uj) using the intrinsic parameter matrix and distortion coefficients. n ,vj n );

[0043] Step S250: Obtain the depth information Z based on the correspondence between the normalized two-dimensional coordinates and the points in the point cloud data;

[0044] Step S260: Combine the normalized two-dimensional coordinates with the depth information to obtain the normalized two-dimensional coordinates in the three-dimensional coordinates P in the global coordinate system. global ;

[0045] The specific method for annotating sensor data to obtain two-dimensional bounding boxes of obstacles in image data and corresponding three-dimensional bounding boxes in point cloud data, and obtaining the category and pose labels of obstacles based on the three-dimensional bounding boxes and obstacle annotations, is as follows:

[0046] Step S310: Draw two-dimensional bounding boxes of obstacles on the image data, record the two-dimensional coordinates of each two-dimensional bounding box, and label the obstacle category within the two-dimensional bounding box;

[0047] Step S320: Based on the two-dimensional bounding boxes marked in the image data, find the corresponding three-dimensional point coordinates in the point cloud data to obtain point cloud clusters, calculate the Euclidean distance between points in the point cloud clusters, and aggregate points whose Euclidean distance is less than the distance threshold into a cluster to form an obstacle point cloud;

[0048] Step S330: Obtain the center coordinates of the obstacle point cloud Calculate the covariance matrix Cz of the obstacle point cloud, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, λ3 and the corresponding eigenvalue vectors v1, v2, v3;

[0049] Step S340: Select the eigenvector v1 with the largest eigenvalue as the length direction of the 3D bounding box, select the eigenvector v2 with the second largest eigenvalue as the width direction of the 3D bounding box, and select the eigenvector v3 with the smallest eigenvalue as the height direction.

[0050] Step S350: For each point pz in the obstacle point cloud i Calculate its projected length l in the length, width, and height directions. i w i h i ;

[0051] Step S360: Find the maximum projected length l of all points along the length direction. max and minimum projection length l min Maximum projected length w in the width direction max and minimum projection length w min Maximum projected length h in the height direction max and minimum projection length h min ;

[0052] Step S370: Take the difference between the maximum and minimum projected length in the length direction as the length of the 3D bounding box, take the difference between the maximum and minimum projected length in the width direction as the width of the 3D bounding box, and take the maximum and minimum projected length in the height direction as the height of the 3D bounding box.

[0053] Step S380: Take the center coordinates of the obstacle point cloud as the position of the obstacle, take the length, width and height directions of the 3D bounding box as the orientation of the obstacle, take the length, width and height of the 3D bounding box as the size of the obstacle, and take the position, orientation and size of the obstacle as the pose of the obstacle. Combine the category of the obstacle label to obtain the category and pose label of the obstacle within the 3D bounding box.

[0054] The method for verifying the annotation of the three-dimensional bounding box is as follows: the three-dimensional bounding box is annotated and verified by projecting the three-dimensional bounding box onto the image plane to generate its two-dimensional projection box, and comparing the degree of fit between the two-dimensional projection box and the two-dimensional bounding box annotated in the image data. If the degree of fit is greater than the fit threshold, the annotation is considered correct.

[0055] The method for calculating the degree of fit is as follows: calculate the intersection area and the union area between the two-dimensional projection frame and the two-dimensional bounding box marked in the image data, and calculate the ratio between the intersection area and the union area as the degree of fit between the two-dimensional projection frame and the two-dimensional bounding box.

[0056] The specific method for training the multimodal fusion perception model, using sensor data and obstacle category and pose labels as training data, is as follows:

[0057] Step S410: Acquire labeled sensor data, including point cloud data. and image data Where, p m N represents the point cloud data of the m-th frame. P For point cloud frame count, I j For the image data of the j-th frame, N I The number of image frames;

[0058] Step S420: Preprocess the sensor data of each frame to obtain voxelized point cloud data and normalized image data;

[0059] Step S430: Build a multimodal fusion perception model, which includes three parts: feature extraction, feature fusion, and classification and regression.

[0060] The multimodal fusion perception model is built on the PyTorch framework and includes three parts: feature extraction, feature fusion, and classification / regression. Feature extraction includes point cloud feature extraction and image feature extraction. The point cloud feature extraction uses a 3D sparse convolutional neural network to extract features from voxelized point cloud data, obtaining a point cloud feature map. Where L′, W′, and H′ represent the voxel space size after downsampling, and C′ is the extracted feature dimension; the image feature extraction uses a 2D convolutional neural network to process the image data I. j Feature extraction is performed to obtain the image feature map. Among them, H f W f C represents the height and width of the image feature map. f The dimension of the image features; the feature fusion part first combines the image feature map Projecting the point cloud onto a bird's-eye view yields... Then it is compared with the point cloud feature map. By splicing the images in the spatial dimension, a fused feature map is obtained. The classification and regression part includes a classification branch and a regression branch, and consists of convolutional layers and fully connected layers.

[0061] Step S440: Extract features from the voxelized point cloud data and normalized image data to obtain point cloud feature maps and image feature maps. Then, fuse the point cloud feature maps and image feature maps to obtain a fused feature map.

[0062] Step S450: The classification and regression part will fuse the feature maps. The mapping is done as a probability map, and for each grid cell in the probability map, the probability of it belonging to each category is predicted; the feature maps are then fused. Mapping to a regression map, predict the pose of obstacles in each grid cell of the regression map;

[0063] Step S460: Minimize the cross-entropy loss function between the predicted probability of a grid cell belonging to each category and the true category label. Minimize the Smooth L1 loss function between the predicted and true poses of obstacles in a grid cell. The weighted sum of the cross-entropy loss function and the Smooth L1 loss function is used as the loss function of the multimodal fusion sensing model.

[0064] Step S470: Use sensor data and obstacle category and pose labels as training data, sensor data as input data, and corresponding obstacle category and pose labels as output data. Use the Adam optimizer to minimize the value of the loss function as the optimization objective to optimize the model parameters. When the value of the loss function converges, the trained multimodal fusion perception model is obtained.

[0065] The specific method for converting the trained multimodal fusion sensing model into an ONNX model, loading the ONNX model into the TensorRT inference engine for acceleration, and obtaining the optimized multimodal fusion sensing model is as follows:

[0066] Step S510: Convert the trained multimodal fusion perception model into an ONNX model to obtain a general model representation for deployment in different inference environments;

[0067] Step S520: Optimize the ONNX model. The optimization process includes model pruning, quantization, and layer fusion.

[0068] Step S530: Use the TensorRT inference engine to load the optimized ONNX model for acceleration, and obtain the optimized multimodal fusion perception model;

[0069] Step S540: Integrate the optimized multimodal fusion perception model into the perception system of the autonomous vehicle.

[0070] One aspect of this application provides an obstacle detection and recognition system for unmanned dock vehicles based on multi-sensor fusion, including:

[0071] The sensor dynamic addition / reduction module is used to set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time.

[0072] The sensor data conversion module is used to obtain the camera's intrinsic parameter matrix and distortion coefficients, obtain the extrinsic parameter matrix between the lidar and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system.

[0073] The obstacle information annotation module is used to annotate the sensor data to obtain the two-dimensional bounding box of the obstacle in the image data and the corresponding three-dimensional bounding box in the point cloud data. Based on the three-dimensional bounding box and the obstacle annotation, the category and pose label of the obstacle are obtained.

[0074] The multimodal model training module is used to build a multimodal fusion perception model. It uses sensor data and obstacle category and pose labels as training data to train the multimodal fusion perception model.

[0075] The multimodal model acceleration module is used to convert the trained multimodal fusion perception model into an ONNX model, load the ONNX model into the TensorRT inference engine for acceleration, and obtain an optimized multimodal fusion perception model.

[0076] The autonomous vehicle real-time detection module is used to deploy an optimized multimodal fusion perception model. The autonomous vehicle collects real-time sensor data to detect obstacles and outputs the category and pose information of the obstacles.

[0077] One aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement steps in a method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion.

[0078] One aspect of this application provides a readable storage medium storing a computer program adapted for loading by a processor to perform steps in a method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion.

[0079] The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion proposed in this application has the following advantages over existing technologies:

[0080] The proposed dynamic deployment strategy based on sensor coverage adjusts the deployment of LiDAR and cameras in real time by setting a coverage threshold to adapt to the real-time needs of the dock operation area. This strategy evaluates sensor coverage density using a grid-based approach and considers the different characteristics of LiDAR and cameras. While meeting coverage requirements, it optimizes the type and location distribution of sensors, improving the efficiency and quality of data acquisition.

[0081] This application adopts a combination of hardware and software synchronization. The high-precision time synchronization module realizes the timestamp alignment of sensor data, ensuring the consistency of data timing.

[0082] This application's multimodal fusion perception model takes point cloud and image data as input, and directly outputs obstacle category and pose information through feature extraction, feature fusion, and classification regression. The model achieves adaptive fusion of features from different modalities within a deep neural network, fully leveraging the complementarity of LiDAR and camera data, simplifying the processing flow, and improving perception performance.

[0083] This application employs model conversion, optimization, and acceleration to convert the trained multimodal fusion perception model into the general ONNX format, and uses the TensorRT inference engine for optimization and acceleration. Through model pruning, quantization, and operator fusion, the inference speed of the model is significantly improved while ensuring perception accuracy, thus meeting the real-time perception requirements of autonomous vehicles.

[0084] This application proposes a complete multi-sensor fusion perception solution suitable for autonomous driving in port terminals. The solution covers the entire process from sensor deployment, data acquisition, model training to model deployment, achieving full-domain environmental perception coverage, spatiotemporal synchronization of data, deep feature fusion, and real-time obstacle detection and tracking, providing strong support for the automated operation of unmanned vehicles in the complex environment of ports. Attached Figure Description

[0085] Figure 1 A flowchart illustrating the obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion provided in this application;

[0086] Figure 2The flowchart of the method for the augmentation strategy provided in this application;

[0087] Figure 3 A flowchart of the method for the reduction strategy provided in this application;

[0088] Figure 4 The functional block diagram of the obstacle detection and recognition system for unmanned dock vehicles based on multi-sensor fusion provided in this application. Detailed Implementation

[0089] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0090] In the accompanying drawings, the size, dimensions, and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustrative purposes only and are not strictly to scale. As used herein, the terms “approximately,” “about,” and similar terms are used to indicate approximation, not degree, and are intended to illustrate inherent deviations in measured or calculated values ​​that will be recognized by one of ordinary skill in the art. Furthermore, the order in which the steps are described in this application does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0091] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0092] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0093] It should be noted that, where there is no conflict, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0094] Example 1

[0095] like Figure 1 As shown, the obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion provided in this application includes:

[0096] Step S100: Set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time.

[0097] The sensors include lidar and cameras.

[0098] The specific method for setting the maximum and minimum thresholds for sensor coverage, comparing the current sensor coverage with the maximum and minimum thresholds, executing corresponding sensor addition / reduction strategies, acquiring sensor data using the sensor array, and synchronizing the sensor data in time is as follows:

[0099] Step S110: Divide the dock operation area into grid areas, and preset the maximum threshold η of sensor coverage η. max and minimum threshold η min Sensor addition and reduction strategies include addition strategies and reduction strategies;

[0100] The formula for calculating the sensor coverage η is: Among them, A covered A represents the area covered by the sensor. total This refers to the total area of ​​the dock's operational area.

[0101] See Figure 2 The specific addition strategy is as follows:

[0102] Step S111: When the sensor coverage is less than the minimum threshold, calculate the sensor coverage density ρ of each grid in the grid area. grid The sensor coverage density is less than the coverage density threshold ρ. min The grid is used as a candidate grid;

[0103] The formula for calculating the sensor coverage density is: Where N sensor A represents the number of sensors within the grid. grid The area is the grid area.

[0104] Step S112: For each candidate grid, count the number N of existing LiDARs within it. L And the number of cameras N C If the ratio of the number of lidars to the number of cameras Less than the preset ratio threshold α of LiDAR and camera LC If so, prioritize adding LiDAR; otherwise, prioritize adding cameras.

[0105] Step S113: Within the candidate grid, randomly generate M potential sensor locations. For each potential sensor location, calculate the local coverage improvement Δη before and after adding the sensor. increase Select the potential sensor location p that offers the greatest improvement in local coverage. max As an additional sensor location;

[0106] The formula for calculating the improvement in local coverage is: Δη increase =η now -η past , where η past η represents the sensor coverage before adding the sensor. now Sensor coverage after adding sensors;

[0107] The potential sensor can be a lidar or a camera. The specific type of sensor selected as the potential sensor depends on the relationship between the ratio of the number of lidars to the number of cameras in step S112 and the preset ratio threshold of lidars and cameras.

[0108] Step S114: Add the newly added sensor locations to the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras;

[0109] Step S115: Repeat steps S111 to S114 until the sensor coverage is greater than or equal to the minimum threshold η. min ;

[0110] See Figure 3 The reduction strategy is specifically as follows:

[0111] Step S116: When the sensor coverage rate is greater than the maximum threshold, calculate the average sensor coverage density of all grids within the grid area. and the sensor coverage density ρ of each grid grid Grids with a sensor coverage density greater than the average sensor coverage density are designated as redundant grids.

[0112] Step S117: Calculate the contribution Δη of each sensor within the redundant grid to the local coverage. decrease The contribution refers to the decrease in sensor coverage before and after removing the sensor in the redundant grid;

[0113] The specific calculation method for the contribution of each sensor in the redundant grid to the local coverage is as follows: Δη decrease =|η now -η past |;

[0114] Step S118: Select the sensor that contributes the least to the local coverage as the sensor to be removed, remove the sensor from the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras.

[0115] Step S119: Repeat steps S116 to S118 until the sensor coverage is less than or equal to the maximum threshold η. max ;

[0116] The maximum threshold η max and minimum threshold η min Coverage density threshold ρ min The ratio threshold α between lidar and camera LC The value is set by those skilled in the art based on experience.

[0117] Step S120: Collect all sensor data in the dock operation area based on the corresponding sensor addition and reduction strategy, and use a combination of high-precision GPS time synchronization and ROS time synchronization to synchronize all sensor data in time to obtain multi-sensor data.

[0118] The high-precision GPS timing synchronization refers to using a high-precision GPS timing module to provide a unified time reference for all sensors, with each sensor data data accompanied by a GPS timestamp t. GPS , representing the absolute time of sensor data acquisition, the resolution of the GPS timestamp is Δt. GPS .

[0119] The resolution of the GPS timestamp is set by those skilled in the art based on experience, and is typically at the microsecond level.

[0120] ROS time synchronization refers to: within the ROS framework, using the message_filters package to align timestamps of data from different sensors, and defining a synchronization window Δt. sync For sensor data whose timestamp difference is less than the synchronization window, they are considered to have been collected at the same time and are synchronized in time.

[0121] The formula for calculating the synchronization window is: Among them, f L f C The data frequencies of the lidar and camera are Δt, respectively. realtimeThe time threshold corresponding to the real-time requirements of time synchronization;

[0122] The time threshold corresponding to the real-time requirement of the time synchronization is set by those skilled in the art based on experience.

[0123] Step S200: Obtain the intrinsic parameter matrix and distortion coefficients of the camera, obtain the extrinsic parameter matrix between the lidar and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system.

[0124] The specific methods for obtaining the camera's intrinsic parameter matrix and distortion coefficients, and for obtaining the extrinsic parameter matrix between the lidar and the camera, are as follows:

[0125] Step S201: Take n1 images of the calibration board from different angles and positions using a camera. In each image of the calibration board, extract the coordinates p of the feature points of the calibration board. img = (ui, vi), calculate the corresponding world coordinates P based on the physical dimensions of the calibration plate. world =(X w ,Y w Z w );

[0126] The specific method for calculating the corresponding world coordinates based on the physical dimensions of the calibration plate is as follows:

[0127] Step S201.1: Measure the actual size of each square on the calibration plate. size And the number of rows and columns of the calibration plate;

[0128] Step S201.2: Define the top left corner of the calibration plate as the origin of the world coordinate system, define the plane on which the calibration plate is located as the Z=0 plane, and align the X-axis and Y-axis with the two edges of the calibration plate respectively;

[0129] Step S201.3: For each feature point coordinate (ui, vi) on the calibration plate, calculate its world coordinate P in the world coordinate system. world =(X w ,Y w Z w );

[0130] The formula for calculating world coordinates is: X w =ui×square size Y w =vi×square size Z w =0;

[0131] For example, assuming a 6×8 calibration board with each square measuring 30 mm, using the corner coordinates of the calibration board as feature point coordinates, the corner coordinates on the calibration board can be calculated as follows: top left corner (0,0), top right corner (0,7), bottom left corner (5,0), bottom right corner (5,7). In this way, we can calculate the coordinates of all corner points on the calibration board in the world coordinate system. These coordinates, along with the corner coordinates detected in the image, are used to estimate the camera's intrinsic and extrinsic parameters.

[0132] Step S202: Establish a camera local coordinate system with the camera as the origin. Transform the world coordinates of the calibration board in the world coordinate system to the camera local coordinate system using the camera's extrinsic parameter matrix, and obtain the corresponding coordinates (X, Y, F, Z) of the feature points of the calibration board in the camera local coordinate system. c ,Y c Z c Establish normalized feature point coordinates (ui) n vi n The pinhole model between the camera and the local coordinate system;

[0133] The calculation formula for the pinhole model is as follows: Where, λ c K is the scaling factor. c Let X be the intrinsic parameter matrix of the camera. c ,Y c Z c ) represents the corresponding coordinates of the feature points on the calibration board in the camera's local coordinate system, (ui) n vi n () represents the normalized coordinates of the feature points;

[0134] The intrinsic parameter matrix of the camera is: Among them, f x f y These are the camera's focal lengths, c and c, respectively. x c y These are the coordinates of the camera's principal point;

[0135] The principal point coordinates of the camera refer to the point where the optical axis of the camera lens intersects the image plane of the calibration board.

[0136] Step S203: Establish a distortion model to map the normalized calibration board feature point coordinates to the distorted feature point coordinates (u d ,v d );

[0137] The expression for the distortion model is: Where r is the radial distance from the normalized feature point coordinates to the principal point. k1, k2, k3, p1, and p2 are distortion coefficients, where k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0138] Step S204: Assign the corresponding coordinates (X, Y, Z) of the feature point to the camera's local coordinate system. c ,Y c Z c The normalized feature point coordinates (ui) are projected from the pinhole model. n vi n Using a distortion model, the normalized feature point coordinates (ui) are... n vi n Converted to distorted feature point coordinates (u) d ,v d The distorted feature point coordinates are converted into actual feature point coordinates based on the camera's intrinsic parameter matrix. These coordinates are then used as the predicted feature point coordinates. The Euclidean distance between the reprojection error and the actual measured feature point coordinates (ui,vi) is calculated as the reprojection error. By minimizing the reprojection error, the camera's intrinsic parameter matrix and distortion coefficients are optimized.

[0139] The conversion formula for converting distorted feature point coordinates to feature point coordinates based on the camera's intrinsic parameter matrix is ​​as follows:

[0140] Step S205: Construct a local laser coordinate system with the lidar as the origin, place a calibration board within the camera's field of view, and use the lidar to scan and obtain the 3D point cloud data of the calibration board in the local laser coordinate system.

[0141] Step S206: For each point P in the 3D point cloud lidar =(X l ,Y l Z l Using the extrinsic parameter matrix T l Transform it to the camera local coordinate system. The extrinsic parameter matrix includes a rotation matrix and a translation vector. Then, use the pinhole model to project the points in the camera local coordinate system onto the calibration board image plane. Find the feature point coordinates corresponding to each point in the 3D point cloud in the calibration board image and establish the correspondence between the 3D point cloud and the feature point coordinates of the calibration board.

[0142] Step S207: The Euclidean distance between the predicted feature point coordinates in the calibration board image and the actual feature point coordinates is used as the projection error. By minimizing the projection error, the optimal extrinsic parameter matrix is ​​obtained.

[0143] The specific method for projecting the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane using intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices, based on point cloud data and image data from sensor data, and converting the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system, is as follows:

[0144] Step S210: Define the origin and coordinate axes of the dock operation area, establish a global coordinate system, establish a local coordinate system for the laser radar with each lidar position as the origin, and establish a local coordinate system for the camera with each camera position as the origin;

[0145] Step S220: The sensor data includes point cloud data measured by the lidar and image data measured by the camera. The three-dimensional point coordinates P of the point cloud data are... lidar Through the external parameter matrix T l =[R l |t l Transform to the camera's local coordinate system to obtain the 3D point coordinates P in the camera's local coordinate system. camera Among them, R l t l These are the rotation matrix and the translation vector, respectively.

[0146] The three-dimensional point coordinates P of the point cloud data lidar Through the external parameter matrix T l =[R l |t l The transformation formula for converting to the camera's local coordinate system is: P camera =R l ×P lidar +t l ;

[0147] Step S230: Project the 3D point coordinates in the camera's local coordinate system into distorted 2D coordinates on the image plane using the camera's intrinsic parameter matrix and distortion coefficients. d ,v d This converts the distorted two-dimensional coordinates into two-dimensional coordinates (ui, vi);

[0148] Steps S210 to S230 above are used to transform the points in the laser local coordinate system to their projected positions on the image plane.

[0149] Step S240: For each two-dimensional coordinate (uj, vj) in the image data, transform it into a normalized two-dimensional coordinate (uj) using the intrinsic parameter matrix and distortion coefficients. n ,vj n );

[0150] The transformation formula for converting it into normalized two-dimensional coordinates using the intrinsic parameter matrix and distortion coefficients is as follows:

[0151] Step S250: Obtain the depth information Z based on the correspondence between the normalized two-dimensional coordinates and the points in the point cloud data;

[0152] The depth information Z represents the Z coordinate value of the normalized two-dimensional coordinate in the camera's local coordinate system, that is, the distance between the two-dimensional coordinate and the camera. Therefore, it is necessary to find the point in the point cloud data corresponding to the two-dimensional coordinate, and directly obtain the depth of the corresponding point from the point cloud data, which is the depth information of the two-dimensional coordinate.

[0153] Step S260: Combine the normalized two-dimensional coordinates with the depth information to obtain the normalized two-dimensional coordinates in the three-dimensional coordinate system of the global coordinate system;

[0154] The formula for calculating the normalized two-dimensional coordinates in the three-dimensional coordinate system is as follows:

[0155] Steps S240 to S260 above convert the two-dimensional coordinates of each pixel in the image data to three-dimensional coordinates in the global coordinate system.

[0156] Step S300: Mark obstacles in the sensor data to obtain the two-dimensional bounding boxes of obstacles in the image data and the corresponding three-dimensional bounding boxes in the point cloud data. Obtain the category and pose label of the obstacle based on the three-dimensional bounding boxes and obstacle markings.

[0157] The specific method for annotating sensor data to obtain two-dimensional bounding boxes of obstacles in image data and corresponding three-dimensional bounding boxes in point cloud data, and obtaining the category and pose labels of obstacles based on the three-dimensional bounding boxes and obstacle annotations, is as follows:

[0158] Step S310: Draw two-dimensional bounding boxes of obstacles on the image data, record the two-dimensional coordinates of each two-dimensional bounding box, and label the obstacle category within the two-dimensional bounding box;

[0159] The two-dimensional coordinates of the two-dimensional bounding box are represented as: (ui) min vi min ,ui max vi max ), where (ui min vi min () represents the top-left corner of the 2D bounding box, (ui) max vi max () represents the bottom right corner of a two-dimensional bounding box;

[0160] Step S320: Based on the two-dimensional bounding boxes marked in the image data, find the corresponding three-dimensional point coordinates in the point cloud data to obtain point cloud clusters, calculate the Euclidean distance between points in the point cloud clusters, and aggregate points whose Euclidean distance is less than the distance threshold into a cluster to form an obstacle point cloud;

[0161] The distance threshold is set by those skilled in the art based on experience. Points in a point cloud cluster whose Euclidean distance is less than the distance threshold can be considered as points of the same object. Separating these points from other points in the point cloud cluster yields the obstacle point cloud Pz = {pz}. i =(x i ,y i ,z i )|i=1,2,...,N}, where pz i =(x i ,y i ,z i ) represents the coordinates of each point in the obstacle point cloud, and N represents the number of points in the obstacle point cloud;

[0162] Step S330: Obtain the center coordinates of the obstacle point cloud Calculate the covariance matrix Cz of the obstacle point cloud, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, λ3 and the corresponding eigenvalue vectors v1, v2, v3;

[0163] The formula for calculating the center coordinates of the obstacle point cloud is:

[0164] The formula for calculating the covariance matrix of the obstacle point cloud is: Where T is the transpose of the matrix;

[0165] Step S340: Select the eigenvector v1 with the largest eigenvalue as the length direction of the 3D bounding box, select the eigenvector v2 with the second largest eigenvalue as the width direction of the 3D bounding box, and select the eigenvector v3 with the smallest eigenvalue as the height direction.

[0166] Step S350: For each point pz in the obstacle point cloud i Calculate its projected length l in the length, width, and height directions. i w i h i ;

[0167] The formulas for calculating the projected lengths in the length, width, and height directions are as follows: i =(pz i -cz)×v1,w i =(pz i -cz)×v2,h i =(pz i -cz)×v3;

[0168] Step S360: Find the maximum projected length l of all points along the length direction. max and minimum projection length l min Maximum projected length w in the width direction max and minimum projection length w min Maximum projected length h in the height direction max and minimum projection length h min ;

[0169] Step S370: Take the difference between the maximum and minimum projected length in the length direction as the length of the 3D bounding box, take the difference between the maximum and minimum projected length in the width direction as the width of the 3D bounding box, and take the maximum and minimum projected length in the height direction as the height of the 3D bounding box.

[0170] Step S380: Take the center coordinates of the obstacle point cloud as the position of the obstacle, take the length, width and height directions of the 3D bounding box as the orientation of the obstacle, take the length, width and height of the 3D bounding box as the size of the obstacle, and take the position, orientation and size of the obstacle as the pose of the obstacle. Combine the category of the obstacle label to obtain the category and pose label of the obstacle within the 3D bounding box.

[0171] Optionally, the 3D bounding box is labeled and verified by projecting the 3D bounding box onto the image plane to generate its 2D projection box. The degree of fit between the 2D projection box and the 2D bounding box labeled in the image data is compared. If the degree of fit is greater than the fit threshold, the labeling is considered correct.

[0172] The matching threshold is set by those skilled in the art based on experience.

[0173] The method for calculating the degree of fit is as follows: calculate the intersection area and the union area between the two-dimensional projection frame and the two-dimensional bounding box marked in the image data, and calculate the ratio between the intersection area and the union area as the degree of fit between the two-dimensional projection frame and the two-dimensional bounding box.

[0174] Step S400: Build a multimodal fusion perception model, using sensor data and obstacle category and pose labels as training data to train the multimodal fusion perception model;

[0175] The specific method for training the multimodal fusion perception model, using sensor data and obstacle category and pose labels as training data, is as follows:

[0176] Step S410: Acquire labeled sensor data, including point cloud data. and image data Where, pm N represents the point cloud data of the m-th frame. P For point cloud frame count, I j For the image data of the j-th frame, N I The number of image frames;

[0177] Step S420: Preprocess the sensor data of each frame to obtain voxelized point cloud data and normalized image data;

[0178] The preprocessing of sensor data for each frame refers to: downsampling the point cloud data of each frame to unify the number of points to N1, normalizing the point cloud to a unit sphere centered at the origin, and converting it into voxelized point cloud data. Where L, W, and H represent the length, width, and height of the voxel space, respectively, and C represents the feature dimension of each voxel;

[0179] Scale each frame of image data to a uniform size H. I ×W I The pixel values ​​are then normalized to the range [0,1] to obtain normalized image data;

[0180] Step S430: Build a multimodal fusion perception model, which includes three parts: feature extraction, feature fusion, and classification and regression.

[0181] The multimodal fusion perception model is built on the PyTorch framework and includes three parts: feature extraction, feature fusion, and classification / regression. Feature extraction includes point cloud feature extraction and image feature extraction. The point cloud feature extraction uses a 3D sparse convolutional neural network to extract features from voxelized point cloud data, obtaining a point cloud feature map. Where L′, W′, and H′ represent the voxel space size after downsampling, and C′ is the extracted feature dimension; the image feature extraction uses a 2D convolutional neural network to process the image data I. j Feature extraction is performed to obtain the image feature map. Among them, H f W f C represents the height and width of the image feature map. f The dimension of the image features; the feature fusion part first combines the image feature map Projecting the point cloud onto a bird's-eye view yields... Then it is compared with the point cloud feature map. By splicing the images in the spatial dimension, a fused feature map is obtained. The classification and regression part includes a classification branch and a regression branch, and consists of convolutional layers and fully connected layers.

[0182] The 3D sparse convolutional neural network consists of a series of 3D sparse convolutional layers and pooling layers, which can effectively capture the spatial structure information of point clouds.

[0183] The 2D convolutional neural network consists of multiple convolutional layers, pooling layers, and residual blocks, and is capable of extracting texture and semantic information from images.

[0184] Step S440: Extract features from the voxelized point cloud data and normalized image data to obtain point cloud feature maps and image feature maps. Then, fuse the point cloud feature maps and image feature maps to obtain a fused feature map.

[0185] The fused feature map contains geometric information from point cloud data and appearance information from image data, which can more comprehensively represent obstacles in the scene.

[0186] Step S450: The classification and regression part will fuse the feature maps. The mapping is done as a probability map, and for each grid cell in the probability map, the probability of it belonging to each category is predicted; the feature maps are then fused. Mapping to a regression map, predict the pose of obstacles in each grid cell of the regression map;

[0187] Step S460: Minimize the cross-entropy loss function between the predicted probability of a grid cell belonging to each category and the true category label. Minimize the Smooth L1 loss function between the predicted and true poses of obstacles in a grid cell. The weighted sum of the cross-entropy loss function and the Smooth L1 loss function is used as the loss function of the multimodal fusion sensing model.

[0188] The formula for calculating the cross-entropy loss function is as follows: Where, N pos y is the number of grid cells in the probability map that contain obstacles. na,nc The true category label for the na-th raster cell belonging to category nc. This is a predicted value for the probability that a grid cell belongs to each category;

[0189] The formula for calculating the Smooth L1 loss function is as follows: Where the number 7 represents the number of parameters in the pose. This represents the true value of the j-th parameter of the obstacle pose in the na-th grid cell. This represents the predicted value of the j-th parameter of the obstacle pose in the na-th grid cell;

[0190] The pose has 7 parameters, namely the position (x, y, z), size (l, w, h), and orientation θ of the obstacle;

[0191] The loss function The calculation formula is: Where λ2 is the weighting coefficient of the balanced loss function;

[0192] The weighting coefficients of the balance loss function are set by those skilled in the art based on experience.

[0193] Step S470: Use sensor data and obstacle category and pose labels as training data, sensor data as input data, and corresponding obstacle category and pose labels as output data. Use the Adam optimizer to minimize the value of the loss function as the optimization objective to optimize the model parameters. When the value of the loss function converges, the trained multimodal fusion perception model is obtained.

[0194] Step S500: Convert the trained multimodal fusion perception model into an ONNX model, load the ONNX model into the TensorRT inference engine for acceleration, and obtain the optimized multimodal fusion perception model.

[0195] The specific method for converting the trained multimodal fusion sensing model into an ONNX model, loading the ONNX model into the TensorRT inference engine for acceleration, and obtaining the optimized multimodal fusion sensing model is as follows:

[0196] Step S510: Convert the trained multimodal fusion perception model into an ONNX model to obtain a general model representation for deployment in different inference environments;

[0197] Step S520: Optimize the ONNX model. The optimization process includes model pruning, quantization, and layer fusion.

[0198] The model pruning refers to removing redundant and unimportant neuron connections, reducing model size and accelerating inference speed. The quantization refers to converting model parameters and activation functions from floating-point numbers to fixed-point numbers, reducing memory usage and computational overhead. The layer fusion refers to merging multiple consecutive operations into an equivalent operation, reducing data transmission and memory access.

[0199] Step S530: Use the TensorRT inference engine to load the optimized ONNX model for acceleration, and obtain the optimized multimodal fusion perception model;

[0200] Step S540: Integrate the optimized multimodal fusion perception model into the perception system of the autonomous vehicle;

[0201] The ONNX model is an open model representation standard that allows interaction between different deep learning frameworks and inference engines. By converting the multimodal fusion perceptual model to the ONNX format, it can be easily deployed to various inference environments.

[0202] The process of loading the ONNX model into the TensorRT inference engine for acceleration includes: operator fusion, memory reuse, low-precision inference, and dynamic size support;

[0203] The operator fusion refers to merging multiple compatible operations into a single CUDA kernel to reduce kernel startup overhead.

[0204] The memory reuse mentioned above refers to reusing the memory space of intermediate results to reduce the number of memory allocations and releases;

[0205] The low-precision inference refers to quantizing the model into INT8 or FP16 format while ensuring accuracy, thereby reducing computational and storage overhead.

[0206] The dynamic size support refers to allowing the multimodal fusion sensing model to dynamically adjust under different input sizes, thereby improving flexibility.

[0207] Step S600: Deploy the optimized multimodal fusion perception model, the unmanned vehicle collects real-time sensor data to detect obstacles, and outputs the category and pose information of obstacles;

[0208] The optimized multimodal fusion perception model, in which the autonomous vehicle collects real-time sensor data for obstacle detection and outputs obstacle category and pose information, is specifically implemented as follows:

[0209] Step S610: The autonomous vehicle's perception system subscribes to the LiDAR topic and the camera image topic to obtain real-time sensor data. The perception system transmits the real-time sensor data to the optimized multimodal fusion perception model.

[0210] Step S620: Preprocess the real-time sensor data, input the preprocessed real-time sensor data into the optimized multimodal fusion perception model, and output the obstacle category and pose information;

[0211] Step S630: Encapsulate the category and pose information of the detected obstacles and send them to other systems of the autonomous vehicle for decision-making and navigation.

[0212] The above steps, through model design and optimization strategies, achieve high-precision, low-latency multimodal obstacle perception, providing reliable environmental perception capabilities for unmanned vehicles to operate autonomously in dock scenarios.

[0213] Example 2

[0214] like Figure 4 As shown, the obstacle detection and recognition system for unmanned dock vehicles based on multi-sensor fusion provided in this application includes:

[0215] The sensor dynamic addition / reduction module is used to set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time.

[0216] The sensor data conversion module is used to obtain the camera's intrinsic parameter matrix and distortion coefficients, obtain the extrinsic parameter matrix between the lidar and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system.

[0217] The obstacle information annotation module is used to annotate the sensor data to obtain the two-dimensional bounding box of the obstacle in the image data and the corresponding three-dimensional bounding box in the point cloud data. Based on the three-dimensional bounding box and the obstacle annotation, the category and pose label of the obstacle are obtained.

[0218] The multimodal model training module is used to build a multimodal fusion perception model. It uses sensor data and obstacle category and pose labels as training data to train the multimodal fusion perception model.

[0219] The multimodal model acceleration module is used to convert the trained multimodal fusion perception model into an ONNX model, load the ONNX model into the TensorRT inference engine for acceleration, and obtain an optimized multimodal fusion perception model.

[0220] The autonomous vehicle real-time detection module is used to deploy an optimized multimodal fusion perception model. The autonomous vehicle collects real-time sensor data to detect obstacles and outputs the category and pose information of the obstacles.

[0221] Example 3

[0222] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion.

[0223] This application provides a readable storage medium storing a computer program adapted for loading by a processor to execute steps in a method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion.

[0224] The methods, apparatus, and devices of this application may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of this application are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this application may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to this application. Thus, this application also covers recording media storing programs for performing the method according to this application.

[0225] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0226] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for obstacle detection and recognition of unmanned dock vehicles based on multi-sensor fusion, characterized in that, include: Set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time. Obtain the camera's intrinsic parameter matrix and distortion coefficients, obtain the extrinsic parameter matrix between the LiDAR and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system. Obstacles are labeled on sensor data to obtain two-dimensional bounding boxes of obstacles in image data and corresponding three-dimensional bounding boxes in point cloud data. The category and pose label of the obstacle are obtained based on the three-dimensional bounding boxes and obstacle labels. A multimodal fusion perception model was built, and sensor data and obstacle category and pose labels were used as training data to train the multimodal fusion perception model. The trained multimodal fusion perception model is converted into an ONNX model, and the ONNX model is loaded into the TensorRT inference engine for acceleration, resulting in an optimized multimodal fusion perception model. The optimized multimodal fusion perception model is deployed, and the unmanned vehicle collects real-time sensor data to detect obstacles and outputs the category and pose information of the obstacles. The specific method for setting the maximum and minimum thresholds for sensor coverage, comparing the current sensor coverage with the maximum and minimum thresholds, executing corresponding sensor addition / reduction strategies, acquiring sensor data using the sensor array, and synchronizing the sensor data in time is as follows: The dock operation area is divided into grid areas, and a maximum threshold for sensor coverage η is preset. and minimum threshold Sensor addition and reduction strategies include addition strategies and reduction strategies; Based on the corresponding sensor addition and reduction strategy, all sensor data in the dock operation area are collected. The time synchronization of all sensor data is performed by a combination of high-precision GPS time synchronization and ROS time synchronization to obtain multi-sensor data.

2. The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in claim 1, characterized in that, The specific addition strategy is as follows: Step S111: When the sensor coverage is less than the minimum threshold, calculate the sensor coverage density of each grid in the grid area. The sensor coverage density is less than the coverage density threshold. The grid is used as a candidate grid; Step S112: For each candidate grid, count the number of existing LiDARs within it. And the number of cameras If the ratio of the number of lidars to the number of cameras Less than the preset ratio threshold of LiDAR and camera If so, prioritize adding LiDAR; otherwise, prioritize adding cameras. Step S113: Within the candidate grid, randomly generate M potential sensor locations. For each potential sensor location, calculate the local coverage improvement before and after adding the sensor. Select potential sensor locations that offer the greatest improvement in local coverage. As an additional sensor location; Step S114: Add the newly added sensor locations to the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras; Step S115: Repeat steps S111 to S114 until the sensor coverage is greater than or equal to the minimum threshold. .

3. The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in claim 2, characterized in that, The reduction strategy is specifically as follows: Step S116: When the sensor coverage rate is greater than the maximum threshold, calculate the average sensor coverage density of all grids within the grid area. and sensor coverage density of each grid Grids with a sensor coverage density greater than the average sensor coverage density are designated as redundant grids. Step S117: Calculate the contribution of each sensor within the redundant grid to the local coverage. The contribution refers to the decrease in sensor coverage before and after removing the sensor in the redundant grid; Step S118: Select the sensor that contributes the least to the local coverage as the sensor to be removed, remove the sensor from the corresponding LiDAR location set or camera location set, and update the number of LiDARs and cameras. Step S119: Repeat steps S116 to S118 until the sensor coverage is less than or equal to the maximum threshold. .

4. The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in claim 3, characterized in that, The specific method for projecting the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane using intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices, based on point cloud data and image data from sensor data, and converting the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system, is as follows: Define the origin and coordinate axes of the dock operation area, establish a global coordinate system, establish a local coordinate system for each lidar position as the origin, and establish a local coordinate system for each camera position as the origin. The sensor data includes point cloud data measured by lidar and image data measured by camera, and the three-dimensional point coordinates of the point cloud data are... Through the external parameter matrix Transform to the camera's local coordinate system to obtain the 3D point coordinates in the camera's local coordinate system. ;in, , These are the rotation matrix and the translation vector, respectively. The camera's intrinsic parameters and distortion coefficients are used to project the 3D point coordinates in the camera's local coordinate system into distorted 2D coordinates on the image plane. Convert the distorted two-dimensional coordinates to two-dimensional coordinates. ; For each two-dimensional coordinate in the image data The intrinsic parameter matrix and distortion coefficients are used to transform it into normalized two-dimensional coordinates. ; Based on the correspondence between normalized two-dimensional coordinates and points in point cloud data, the depth information Z is obtained; By combining the normalized 2D coordinates with depth information, we obtain the 3D coordinates of the normalized 2D coordinates in the global coordinate system. .

5. The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in claim 4, characterized in that, The specific method for annotating sensor data to obtain two-dimensional bounding boxes of obstacles in image data and corresponding three-dimensional bounding boxes in point cloud data, and obtaining the category and pose labels of obstacles based on the three-dimensional bounding boxes and obstacle annotations, is as follows: Draw two-dimensional bounding boxes of obstacles on the image data, record the two-dimensional coordinates of each two-dimensional bounding box, and label the obstacle category within the two-dimensional bounding box; Based on the two-dimensional bounding boxes marked in the image data, the corresponding three-dimensional point coordinates are found in the point cloud data to obtain point cloud clusters. The Euclidean distance between points in the point cloud clusters is calculated, and points with Euclidean distances less than the distance threshold are aggregated into a cluster to form an obstacle point cloud. Obtain the center coordinates of the obstacle point cloud Calculate the covariance matrix Cz of the obstacle point cloud, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, λ3 and the corresponding eigenvalue vectors v1, v2, v3; Select the eigenvector v1 with the largest eigenvalue as the length direction of the 3D bounding box, select the eigenvector v2 with the second largest eigenvalue as the width direction of the 3D bounding box, and select the eigenvector v3 with the smallest eigenvalue as the height direction. For each point in the obstacle point cloud Calculate its projected length in the length, width, and height directions. , , ; Find the maximum projected length of all points along the length direction. and minimum projection length Maximum projected length in the width direction and minimum projection length Maximum projected length in the height direction and minimum projection length ; The difference between the maximum and minimum projected lengths in the length direction is taken as the length of the 3D bounding box, the difference between the maximum and minimum projected lengths in the width direction is taken as the width of the 3D bounding box, and the maximum and minimum projected lengths in the height direction are taken as the height of the 3D bounding box. The center coordinates of the obstacle point cloud are taken as the position of the obstacle. The length, width, and height directions of the 3D bounding box are taken as the orientation of the obstacle. The length, width, and height of the 3D bounding box are taken as the size of the obstacle. The position, orientation, and size of the obstacle are collectively referred to as the pose of the obstacle. Combined with the category of the obstacle label, the category and pose label of the obstacle within the 3D bounding box are obtained.

6. The obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in claim 5, characterized in that, The specific method for training the multimodal fusion perception model, using sensor data and obstacle category and pose labels as training data, is as follows: Acquire labeled sensor data, including point cloud data. and image data ,in, This represents the point cloud data of the m-th frame. For point cloud frame count, For the image data of the j-th frame, The number of image frames; The sensor data for each frame is preprocessed to obtain voxelized point cloud data and normalized image data. A multimodal fusion perception model is constructed, which includes three parts: feature extraction, feature fusion, and classification and regression. Feature extraction is performed on voxelized point cloud data and normalized image data to obtain point cloud feature maps and image feature maps. Feature fusion is then performed on the point cloud feature maps and image feature maps to obtain fused feature maps. ;in, , , This indicates the size of the voxel space after downsampling. The dimension of the image features; The classification and regression part will integrate feature maps. The mapping is done as a probability map, and for each grid cell in the probability map, the probability of it belonging to each category is predicted; the feature maps are then fused. Mapping to a regression map, predict the pose of obstacles in each grid cell of the regression map; The cross-entropy loss function minimizes the predicted probability of a grid cell belonging to each category and the true category label. Minimize the Smooth L1 loss function between the predicted and true poses of obstacles in a grid cell. The weighted sum of the cross-entropy loss function and the Smooth L1 loss function is used as the loss function of the multimodal fusion sensing model. ; Sensor data and obstacle category and pose labels are used as training data, sensor data as input data, and corresponding obstacle category and pose labels as output data. The Adam optimizer is used to optimize the model parameters by minimizing the value of the loss function. When the value of the loss function converges, the trained multimodal fusion perception model is obtained.

7. A multi-sensor fusion-based obstacle detection and recognition system for unmanned dock vehicles, used to implement the multi-sensor fusion-based obstacle detection and recognition method for unmanned dock vehicles according to any one of claims 1-6, characterized in that, include: The sensor dynamic addition / reduction module is used to set the maximum and minimum thresholds for sensor coverage, compare the current sensor coverage with the maximum and minimum thresholds, execute the corresponding sensor addition / reduction strategy, acquire sensor data using the sensor array, and synchronize the sensor data in time. The sensor data conversion module is used to obtain the camera's intrinsic parameter matrix and distortion coefficients, obtain the extrinsic parameter matrix between the lidar and the camera, and based on the point cloud data and image data in the sensor data, use the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix to project the three-dimensional point coordinates of the point cloud data into two-dimensional coordinates of the image plane, and convert the two-dimensional coordinates in the image data into three-dimensional coordinates in the global coordinate system. The obstacle information annotation module is used to annotate the sensor data to obtain the two-dimensional bounding box of the obstacle in the image data and the corresponding three-dimensional bounding box in the point cloud data. Based on the three-dimensional bounding box and the obstacle annotation, the category and pose label of the obstacle are obtained. The multimodal model training module is used to build a multimodal fusion perception model. It uses sensor data and obstacle category and pose labels as training data to train the multimodal fusion perception model. The multimodal model acceleration module is used to convert the trained multimodal fusion perception model into an ONNX model, load the ONNX model into the TensorRT inference engine for acceleration, and obtain an optimized multimodal fusion perception model. The autonomous vehicle real-time detection module is used to deploy an optimized multimodal fusion perception model. The autonomous vehicle collects real-time sensor data to detect obstacles and outputs the category and pose information of the obstacles.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program adapted for loading by a processor to perform the steps in the obstacle detection and recognition method for unmanned dock vehicles based on multi-sensor fusion as described in any one of claims 1-6.

Images