A 3D target detection device for unmanned delivery vehicles
By combining an onboard Ethernet camera and LiDAR 3D target detection device on an unmanned delivery vehicle, high-precision target recognition in complex environments is achieved, solving the problem of low recognition rate in existing technologies and improving the safety of path planning and obstacle avoidance.
Patent Information
- Application Number
- CN202210960692.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-08-11
AI Technical Summary
The existing 3D target recognition technology for unmanned delivery vehicles has a low recognition rate under conditions of occlusion, small targets, or adverse weather, which cannot meet the needs of path planning and obstacle avoidance, and poses safety hazards.
A 3D target detection device combining an in-vehicle Ethernet camera and a LiDAR is used to perform 3D target detection through field-of-view correction, synchronous data matching, and multi-layer neural networks. The device includes a field-of-view correction module, a synchronous data matching module, and a 3D target detection module. The neural network is used for image data processing and point cloud data matching to achieve synchronization and field-of-view matching of LiDAR and camera data. The LSTM network is used for feature extraction and regression of multi-frame data.
It improves the accuracy of target recognition for unmanned delivery vehicles in complex environments, ensures the safety of path planning and obstacle avoidance, and reduces safety hazards caused by target obstruction or weather conditions.
Smart Images

Figure CN117649655B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned delivery vehicles, and in particular to a 3D target detection device for unmanned delivery vehicles. Background Technology
[0002] With the rapid development of the logistics industry, a large amount of goods are transported worldwide via land, air, and sea. However, upon arrival at the delivery point, couriers need to deliver packages to each household, which is a waste of manpower and resources, significantly reducing delivery efficiency. If the location of the delivery point remains unchanged, the distance from the delivery station to the residential area is fixed. Current technology uses unmanned delivery vehicles for automated goods delivery. Compared to autonomous vehicles under development, unmanned delivery vehicles are relatively safer due to different application scenarios. They can deliver goods at low speeds, such as less than 20 km / h, in fixed environments. However, because they are unmanned, driving on busy roads and in residential areas requires recognizing targets ahead in order to plan a path accordingly. Traditional image-based target recognition is based on 2D target recognition. Although 2D target recognition algorithms are relatively mature, they do not provide distance information, making it difficult to provide better path planning services for drone delivery vehicles. Therefore, 2D target recognition cannot meet the requirements of unmanned delivery vehicles. Target information needs to be obtained at the 3D level to acquire the target in front of the vehicle and its position, enabling path planning and obstacle avoidance. However, current 3D target recognition is still under development, and its recognition rate is low under conditions of occlusion, small objects, or bad weather. In residential environments, due to the influence of streetlights, mobile facilities, and dense trees, existing algorithms struggle to identify targets when they are temporarily or partially obscured, potentially leading to safety hazards for unmanned delivery vehicles. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention provides a 3D target detection device for unmanned delivery vehicles, comprising at least:
[0004] Vehicle-mounted Ethernet cameras and LiDAR installed on the vehicle body;
[0005] An autonomous driving host that receives image data from an in-vehicle Ethernet camera and point cloud data from a LiDAR and performs 3D target detection;
[0006] The autonomous driving host includes: a field of view correction module, a synchronous data matching module, and a 3D target detection module. The field of view correction module is used to match the field of view of the LiDAR and the vehicle camera based on their installation positions and imaging relationship.
[0007] The synchronous data matching module is used to synchronize and match the frame data of the LiDAR with the frame data of the camera, so that the frame data of the LiDAR and the frame data of the camera are captured at the same time.
[0008] The 3D target detection module is used to perform 3D recognition of targets ahead during driving.
[0009] A 3D target detection device for unmanned delivery vehicles, further optionally, includes a synchronous data matching module for registering LiDAR and vehicle-mounted camera acquisitions, specifically including: calculating the time delay and frequency offset of the vehicle-mounted Ethernet camera and LiDAR to the autonomous driving host according to the gPTP protocol;
[0010] The vehicle-mounted Ethernet camera and LiDAR are time-synchronized and calibrated according to their respective time delays and frequency offsets.
[0011] Using the time point of the point cloud data sent by the LiDAR received by the autonomous driving host as a reference benchmark, the image data of the previous frame and the image data of the next frame corresponding to the point cloud data of the current LiDAR are obtained. The registered image data is obtained by linear interpolation and the current reference timestamp is marked on the registered image data.
[0012] A 3D target detection device for unmanned delivery vehicles, further optionally, includes a field-of-view correction module for achieving field-of-view matching between the lidar and the vehicle-mounted camera, specifically including:
[0013] A checkerboard method is used to obtain the calibration matrix, which is then used to crop the input point cloud. Each point p(x,y,z) is projected onto the image plane using the calibration matrix, retaining only points within the image plane. This ensures that the data from both the LiDAR and camera sensors have the same field of view. The projection formula is as follows:
[0014]
[0015] Where K is the intrinsic parameter calibration matrix of the camera, R and t are the rotation matrix and translation vector describing the position between the lidar and the camera, and (u,v) are the image coordinates corresponding to point p(x,y,z).
[0016] A 3D target detection device for unmanned delivery vehicles, further optionally, the method for the 3D target detection module to implement 3D target detection specifically includes:
[0017] The images captured by the vehicle-mounted camera are processed by the first neural network model to obtain the location and category of the two-dimensional targets in the images;
[0018] Based on the acquired position of the two-dimensional target, the target position is mapped to the three-dimensional point cloud image of the lidar imaging and the point cloud data corresponding to the target is selected;
[0019] After inputting the category and point cloud data into the second neural network for processing, the global features of the 3D target are segmented from the point cloud data;
[0020] The global features of the 3D target in multiple consecutive frames are input into the third neural network, and the position of the 3D target is regressed through the network.
[0021] A 3D target detection device for unmanned delivery vehicles, wherein, optionally, the third neural network comprises: at least one LSTM network and at least one fully connected layer;
[0022] The LSTM network is used to process the global feature vectors of the 3D target in multiple consecutive frames to extract the temporary enhanced feature vector of the target of interest. The temporary enhanced feature vector is then input into a fully connected layer to regress the precise location of the 3D target.
[0023] A 3D target detection device for unmanned delivery vehicles, wherein the hidden layers are all multi-layer LSTM units, and the optimal values of the number of hidden layers and the number of neurons are determined by a grid search method;
[0024] Features are extracted using LSTM units and then input into a fully connected layer to regress detailed parameters of the 3D target bounding box.
[0025] A 3D target detection device for unmanned delivery vehicles, which optionally further controls the loss function of similarity on the global feature vectors of 3D targets in consecutive frames, so as to provide that the feature embeddings of the same object are similar in consecutive frames;
[0026] The similarity difference between two feature vectors of the same object in consecutive frames is used for calculation.
[0027] v and ω are feature vectors of objects from consecutive frames, and l is the dimension of the vector;
[0028]
[0029] When the cosine distance loss of multiple consecutive frames of input objects exceeds a certain threshold, the current prediction is filtered out, and the next prediction is performed.
[0030] A 3D target detection device for unmanned delivery vehicles, optionally further comprising a first neural network model based on a two-dimensional convolutional neural network model, specifically including: a multi-layer residual network unit, a first convolutional feature layer, a second convolutional feature layer, a third convolutional feature layer, and a fourth convolutional feature layer, wherein the multi-layer residual network unit is used to process the input image data and extract features of the input size and number of channels of the image data according to the model parameters of the first convolutional feature layer;
[0031] From the first convolutional feature layer to the fourth convolutional feature layer, each time a convolutional layer is passed, the size is reduced to twice its original size, and the number of channels is increased to twice its original size.
[0032] A 3D target detection device for unmanned delivery vehicles, further optionally, the first neural network model further includes: a first prediction layer, a second prediction layer, a third prediction layer, a fourth prediction layer, and a merging layer;
[0033] From top to bottom, starting from the fourth convolutional feature layer to the first convolutional feature layer, the output of the lower layer and the output of the corresponding upper layer are upsampled by 2 times through the deconvolution layer and then added to form the third prediction layer, the second prediction layer, and the first prediction layer in sequence.
[0034] The fourth prediction layer is the same as the fourth convolutional feature layer;
[0035] The first to fourth prediction layers are merged to form a merged layer;
[0036] After the merged layer passes through a 1*1 convolutional layer, regression is performed to obtain the 2D target location information and the corresponding category of the 2D target.
[0037] A 3D target detection device for unmanned delivery vehicles, wherein, optionally, the second neural network comprises: a first branch global feature extraction network, a center offset feature extraction network, a feature center change network, and a second branch global feature extraction network;
[0038] In this process, the first branch global feature extraction network extracts the first global feature and then inputs it into the center offset feature extraction network and the feature center change network for corresponding feature extraction.
[0039] Then, the output of the center offset feature extraction network is concatenated with the output of the feature center change network, and then input into the second branch global feature extraction network to extract the second global feature.
[0040] A 3D target detection device for unmanned delivery vehicles, further optionally, includes a loss function for the first neural network comprising: localization loss, the calculation formula for the Loc_loss localization loss function is as follows:
[0041] h = |h1-h2|, w = |w1-w2|
[0042]
[0043] Loc_loss=2*In2-2*In(1+gloss)
[0044] Where h1, w1, h2, and w2 are the height and width of the target bounding box and the predicted bounding box, respectively;
[0045] IoU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box;
[0046] The Euclidean distance r between the center points of the predicted bounding box and the target bounding box;
[0047] Ch and Cw are the length and width of the minimum bounding rectangle that covers the prediction box and the target box, respectively.
[0048] A 3D target detection device for unmanned delivery vehicles, further optionally, the first branch global feature extraction network includes a sequence branch network, through which an orthogonal matrix is trained to rotate the input point cloud, so that the entire network model can match point clouds with various rotation angles;
[0049] A 3D target detection device for unmanned delivery vehicles, further optionally, the first branch global feature extraction network includes at least: 3 one-dimensional convolutional layers, 1 max pooling layer, 4 fully connected layers, and then forms a U matrix of [batchsize,k,k] by reshaping, where batchsize is the number of targets and k is the dimension;
[0050] A 3D target detection device for unmanned delivery vehicles, which optionally further obtains U by training through the construction of an orthogonal loss function in order to make U an orthogonal matrix;
[0051] Orthogonal loss function: loss(U) = min(E – U * U) T )
[0052] Where loss(U) is the normal loss function, E is the unit vector, and U is the orthogonal matrix;
[0053] By training, the loss(U) approaches 0, and U can be obtained as a normal matrix.
[0054] The result obtained by multiplying an orthogonal matrix with the output of the corresponding convolutional layer is output to the next convolutional layer for computation.
[0055] Beneficial effects:
[0056] In the technical solution of the present invention, image data and point cloud data are acquired by an on-board Ethernet camera and LiDAR installed in an unmanned delivery vehicle. The 2D target position coordinates in the image are obtained by a first neural network. Then, the 3D data of the corresponding target are selected based on the 2D target position coordinates in the 3D LiDAR point cloud data. The 3D target position and classification are then regressed by a second neural network and a third neural network.
[0057] This invention synchronizes and matches the data of the vehicle-mounted Ethernet LiDAR and the vehicle-mounted Ethernet, solving the technical problem of data mismatch in the prior art;
[0058] In addition, in the third neural network, an LSTM network that outputs global feature data from multiple consecutive frames is used to temporarily enhance the extraction of global features, which can accurately regress the position of the 3D target. Attached Figure Description
[0059] The following figures are for illustrative purposes only and do not limit the scope of the invention.
[0060] Figure 1a This is a schematic diagram of a 3D target detection device for an unmanned delivery vehicle according to an embodiment of the present invention.
[0061] Figure 1b This is a schematic diagram of a 3D target detection method for unmanned delivery vehicles in one embodiment of the present invention.
[0062] Figure 2 This is a schematic diagram of the model structure of the first neural network in one embodiment of the present invention.
[0063] Figure 3 This is a schematic diagram of the geometric positions of the prediction box and the target box in one embodiment of the present invention.
[0064] Figure 4 This is a schematic diagram of the model structure of the second neural network in one embodiment of the present invention.
[0065] Figure 5 This is the first branch of the global feature extraction network in the model structure of the second neural network in one embodiment of the present invention.
[0066] Figure 6 This is the second branch of the global feature extraction network in the model structure of the second neural network in one embodiment of the present invention.
[0067] Figure 7 This is a center offset feature extraction network in the model structure of the second neural network in one embodiment of the present invention.
[0068] Figure 8 This is a feature center transformation network in the model structure of the second neural network in one embodiment of the present invention.
[0069] Figure 9 This is a schematic diagram of the model structure of the third neural network in one embodiment of the present invention. Detailed Implementation
[0070] To provide a clearer understanding of the technical features, objectives, and effects of this invention, specific embodiments are now described with reference to the accompanying drawings, in which the same reference numerals denote the same parts. For the sake of simplicity, the parts related to this invention are shown schematically in each drawing and do not represent their actual structure as a product. Furthermore, for the sake of clarity and ease of understanding, in some drawings, components with the same structure or function are shown only schematically, or only one is labeled.
[0071] Regarding control systems, as is well known to those skilled in the art, functional modules and application programs (APPs) can take any suitable form, whether hardware or software, and can be multiple discrete functional modules or multiple functional units integrated onto a single hardware device. In its simplest form, the control system can be a controller, such as a combinational logic controller or a microprogrammed controller, as long as it can implement the operations described in this application. Of course, the control system can also be integrated as different modules onto a single physical device, without departing from the basic principles and scope of protection of this invention.
[0072] In this invention, "connection" can include direct connection, indirect connection, communication connection, and electrical connection, unless otherwise specified.
[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly specifies otherwise. It will also be understood that, when used in the specification, the terms “comprising” and / or “including” mean the presence of the stated features, values, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, values, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items.
[0074] It should be understood that the term "vehicle" or "of a vehicle" or other similar terms as used herein generally includes motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, boats including various vessels, aircraft, etc., and includes hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other vehicles powered by alternative fuels (e.g., fuels derived from non-petroleum energy sources). As mentioned herein, a hybrid vehicle is a vehicle with two or more power sources, such as both gasoline and electric power.
[0075] Furthermore, the controller disclosed herein can be embodied as a non-transient computer-readable medium on a computer-readable medium containing executable program instructions that are executed by a processor, controller, or the like. Examples of computer-readable media include, but are not limited to, ROM, RAM, optical disc (CD)-ROM, magnetic tape, floppy disk, flash memory drive, smart card, and optical data storage device. Computer-readable recording media can also be distributed across a network-coupled computer system, such that the computer-readable media are stored and executed in a distributed manner, for example, via a telematics server or a controller area network (CAN).
[0076] This invention provides a 3D target detection device for unmanned delivery vehicles, specifically including: (see details) Figure 1a The vehicle is equipped with an onboard Ethernet camera and LiDAR.
[0077] An autonomous driving host that receives image data from an in-vehicle Ethernet camera and point cloud data from a LiDAR and performs 3D target detection;
[0078] The autonomous driving host includes: a field of view correction module, a synchronous data matching module, and a 3D target detection module. The field of view correction module is used to match the field of view of the LiDAR and the vehicle camera based on their installation positions and imaging relationship.
[0079] The synchronous data matching module is used to synchronize and match the frame data of the LiDAR with the frame data of the camera, so that the frame data of the LiDAR and the frame data of the camera are captured at the same time.
[0080] The 3D target detection module is used to perform 3D recognition of targets ahead during driving.
[0081] Specifically, the camera uses an in-vehicle Ethernet to transmit data, and the lidar uses an in-vehicle Ethernet to transmit data. Usually, only one lidar needs to be installed, and the lidar is installed on the roof of the vehicle.
[0082] The vehicle-mounted Ethernet cameras are installed on the left and right sides of the front of the vehicle.
[0083] The data from the LiDAR and camera is transmitted to the TSN gateway, which then parses the data and sends it to the autonomous driving host for processing.
[0084] The autonomous driving host has a variety of functions, such as path planning and issuing control commands to the underlying sensors. This embodiment only considers the 3D target detection function of the autonomous driving host. Therefore, other functions are not described in this embodiment.
[0085] The synchronous data matching module is used to register the LiDAR and vehicle camera acquisitions. Specifically, it includes: calculating the time delay and frequency offset of the vehicle Ethernet camera and LiDAR to the autonomous driving host according to the gPTP protocol.
[0086] The vehicle-mounted Ethernet camera and LiDAR are time-synchronized and calibrated according to their respective time delays and frequency offsets.
[0087] Using the time point of the point cloud data sent by the LiDAR received by the autonomous driving host as a reference benchmark, the image data of the previous frame and the image data of the next frame corresponding to the point cloud data of the current LiDAR are obtained. The registered image data is obtained by linear interpolation and the current reference timestamp is marked on the registered image data.
[0088] The reference timestamp is matched with the LiDAR timestamp. Subsequently, based on the reference timestamp, it is determined whether the LiDAR point cloud data and the camera image data were captured at the same time.
[0089] Specifically, such as Figure 1b As shown, the specific methods for implementing 3D object detection by the 3D object detection module include:
[0090] The images captured by the vehicle-mounted camera are processed by the first neural network model to obtain the location and category of the two-dimensional targets in the images;
[0091] Based on the acquired position of the two-dimensional target, the target position is mapped to the three-dimensional point cloud image of the lidar imaging and the point cloud data corresponding to the target is selected;
[0092] After inputting the category and point cloud data into the second neural network for processing, the global features of the 3D target are segmented from the point cloud data;
[0093] The global features of the 3D target in multiple consecutive frames are input into the third neural network, and the position of the 3D target is regressed through the network.
[0094] Specifically, the unmanned delivery vehicle is equipped with a lidar on its roof and two vehicle-mounted Ethernet cameras at the front of the vehicle.
[0095] During driving, based on the inherent properties of LiDAR and vehicle-mounted Ethernet camera sensors, each acquires images at its own frequency. However, since the acquisition frequencies of the vehicle-mounted Ethernet camera and LiDAR are different, the environmental images acquired by the two may not be at the same moment, that is, the environmental images obtained are different, resulting in a large error after the two are fused.
[0096] In existing technologies, the common approach is to find two similar moments to obtain the emission point cloud image from the LiDAR and the image captured by the vehicle-mounted camera. However, this method still suffers from a lack of synchronization between the point cloud image formed by the LiDAR and the image captured by the vehicle-mounted camera; they are only relatively close. Furthermore, due to the delay in data transmission from the sensors to the autonomous driving host, and the differences in clock oscillators and transmission routes between different sensors, time delays and time offsets vary. This results in existing LVDS interface cameras and FlexRay or traditional Ethernet bus-based transmissions lacking time synchronization capabilities. Therefore, even if images from the same moment can be obtained while searching for adjacent moments, the lack of time synchronization will still result in the images captured by the LiDAR and the camera not being at the same instant. Therefore, to address this issue, this embodiment employs the following method for real-time fusion to maintain the synchronization and consistency of data acquired by the vehicle-mounted Ethernet LiDAR and the vehicle-mounted Ethernet camera, specifically including:
[0097] The vehicle-mounted camera and lidar use vehicle-mounted Ethernet interfaces. Non-control signals are transmitted using the IEEE 1722AVTP protocol, while control signals are transmitted based on the SOME / IP protocol.
[0098] Calculate the time delay and frequency offset of the vehicle-mounted Ethernet camera and vehicle-mounted Ethernet LiDAR to the autonomous driving host according to the gPTP protocol.
[0099] The vehicle-mounted Ethernet camera and LiDAR are time-synchronized and calibrated according to their respective time delays and frequency offsets.
[0100] Using the time point of the point cloud data sent by the LiDAR received by the autonomous driving host as a reference benchmark, the image data of the previous frame and the image data of the next frame corresponding to the point cloud data of the current LiDAR are obtained. The registered image data is obtained by linear interpolation and the current reference timestamp is marked on the registered image data.
[0101] Specifically, in this embodiment, since both LiDAR and camera sensors are fused, and the location predicted by the 2D image is used to extract the corresponding point cloud data from the LiDAR which has depth information, the geometric positions of the vehicle-mounted Ethernet camera and the LiDAR on the vehicle body are not the same, resulting in their images not being in the same coordinate system. Therefore, it is necessary to calibrate the LiDAR and camera to make their pixel coordinate systems correspond. The field-of-view correction module is used to achieve field-of-view matching between the LiDAR and the vehicle-mounted camera, and the specific implementation includes: Specifically:
[0102] A checkerboard method is used to obtain the calibration matrix, which is then used to crop the input point cloud. Each point p(x,y,z) is projected onto the image plane using the calibration matrix, retaining only points within the image plane. This ensures that the data from both the LiDAR and camera sensors have the same field of view. The projection formula is as follows:
[0103]
[0104] Where K is the intrinsic parameter calibration matrix of the camera, R and t are the rotation matrix and translation vector describing the position between the lidar and the camera, and (u,v) are the image coordinates corresponding to point p(x,y,z);
[0105] Specifically, see Figure 2 The first neural network model is a two-dimensional convolutional neural network model, specifically including: multi-layer residual network units, a first convolutional feature layer, a second convolutional feature layer, a third convolutional feature layer, and a fourth convolutional feature layer. Among them, the multi-layer residual network units are used to process the input image data and extract features from the input size and number of channels of the image data according to the model parameters of the first convolutional feature layer.
[0106] From the first convolutional feature layer to the fourth convolutional feature layer, each time a convolutional layer is passed, the size is reduced to twice its original size, and the number of channels is increased to twice its original size.
[0107] The multi-layer residual network unit includes multiple convolutional 3D convolutional layers. Based on the feature extraction of higher-level semantics, different numbers of convolutional layers can be added, and the width, height and number of channels of the input of each layer can be adjusted to match the input of the first convolutional feature layer.
[0108] From the first convolutional feature layer to the fourth convolutional feature layer, one possible input shape control is: [Batchsize, 256, 256, 128], [Batchsize, 128, 128, 256], [Batchsize, 64, 64, 256], [Batchsize, 32, 32, 1024], where batchsize is the number of input images per batch.
[0109] The first neural network model also includes: a first prediction layer, a second prediction layer, a third prediction layer, a fourth prediction layer, and a merging layer;
[0110] From top to bottom, starting from the fourth convolutional feature layer to the first convolutional feature layer, the output of the lower layer and the corresponding output of the upper layer are upsampled by 2 times through the deconvolution layer and the output with the number of channels matched with the lower layer is added to form the third prediction layer, the second prediction layer, and the first prediction layer in sequence.
[0111] The fourth prediction layer is the same as the fourth convolutional feature layer;
[0112] Specifically, in order for the first to fourth prediction layers to be merged, since the number of channels is different, it is necessary to use a (1*1) convolutional kernel to match the number of channels.
[0113] The first to fourth prediction layers are merged to form a merged layer;
[0114] After the merged layer passes through a 1*1 convolutional layer, regression is performed to obtain the 2D target location information and the corresponding category of the 2D target.
[0115] Specifically, in the first neural network model, the loss function is defined as:
[0116] Loss function = Classification loss + Localization loss + Confidence loss;
[0117] The classification loss function and confidence loss were calculated using BCELogits and the binary cross-entropy loss function BCEloss, respectively.
[0118] The localization loss function is calculated as follows: Traditional IOU loss has significant limitations and is difficult to converge when dealing with two non-intersecting objects. In addition, when two objects have an inclusion relationship, traditional IOU cannot measure the distance between the two objects, which slows down or prevents convergence.
[0119] Specifically, see Figure 3 Since A∩B=B=C, let c be the diagonal distance between the minimum bounding rectangle C of A and B, and r be the Euclidean distance between the center points of the predicted box B and the target box A. We can then calculate the differences in length and width between the target box and the predicted box, h and w, where Ch and Cw are the length and width of the minimum bounding rectangles covering the predicted and target boxes, respectively. h1, w1, h2, and w2 are the height and width of the target and predicted boxes, respectively. The localization loss (Loc_loss) is then defined as follows:
[0120] h = |h1-h2|, w = |w1-w2|
[0121]
[0122] Loc_loss=2*In2-2*In(1+gloss)
[0123] Specifically, after obtaining the predicted 2D target location and classification through the first neural network model, the 2D target location is mapped to the 3D laser point cloud to obtain the point cloud cone at the corresponding location. Specifically, it can be obtained by constructing geometric equations based on geometric relationships, such as plane equations in different directions. IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. IoU calculates the overlap rate between the "predicted bounding box" and the "ground truth bounding box", that is, the ratio of their intersection to their union.
[0124] Once the cone-shaped point cloud data of the corresponding 2D target is obtained, inputting all the point clouds into the model would result in a massive computational burden. Therefore, for each target's 3D cone, a preset number of points are selected and input into the network, such as 256 or 512 points. If the number of points is insufficient, zeros are used for padding. The point cloud of the 3D cone is then input into a second neural network to obtain the global features of the segmented 3D target.
[0125] Specifically, a point cloud is represented as [batchsize, num_points, features], where batchsize is the number of targets input in each batch, num_points is the number of points, and features is the number of features, such as the three-dimensional spatial features of the points.
[0126] Specifically, see Figures 4 to 8 For details, see Figure 4 The second neural network includes: a first branch global feature extraction network, a center offset feature extraction network, a feature center change network, and a second branch global feature extraction network;
[0127] In this process, the first branch global feature extraction network extracts the first global feature and then inputs it into the center offset feature extraction network and the feature center change network for corresponding feature extraction.
[0128] Then, the output of the center offset extraction network is concatenated and transformed with the output of the feature center transformation network, and then input into the second branch global feature extraction network to extract the second global features;
[0129] Specifically, see Figure 5 The first branch of the global feature extraction network includes at least one-dimensional convolution, specifically including: multiple convolutional layers, max pooling layers, and fully connected layers;
[0130] With each convolutional layer, the number of output channels may increase or remain the same, while the number of point clouds remains constant.
[0131] Specifically, an optimal model example includes: three 64-channel one-dimensional convolutional layers, one 128-channel one-dimensional convolutional layer, one 512-channel one-dimensional convolutional layer, and one 1024-channel one-dimensional convolutional layer.
[0132] Compared to the initial feature which is 9-dimensional or 7-dimensional, after being upgraded to 1024-dimensional and undergoing multiple high-level semantic feature extractions, it can completely express the information of each point of the target.
[0133] Then, through a max pooling layer, the 3D target is represented by the feature value with the largest value among all target points.
[0134] Semantic features are further extracted after passing through two fully connected layers;
[0135] 3D target feature data is obtained by concatenating the output of a preset convolutional layer with the output of a preset fully connected layer;
[0136] Specifically, for example, the 3D target feature data is obtained by concatenating the output of the third convolutional layer with the output after two fully connected layers. Specifically, through this design, the initial low-level features and high-level features can be preserved. After merging the two, it is beneficial to the subsequent point cloud classification, ensuring that each point contains specific initial features and high-level semantic features.
[0137] By stitching together the data, each feature point of the 3D target contains the same global feature point confidence, which is beneficial for subsequent object classification and segmentation recognition points.
[0138] After concatenation, the output conforms to the input of the feature extraction network and the input of the feature center transformation network after three convolutions.
[0139] The input point cloud is batchsize*n*4. Because the point cloud is unordered, when the same point cloud is input with point cloud sequences of different rotation angles, the neural network model may have difficulty recognizing them. To solve this problem, in this embodiment, a sequence branching network is introduced into the neural network. An orthogonal matrix is trained through the sequence branching network to rotate the input point cloud, so that the entire network model can adapt to point clouds with various rotation angles.
[0140] Specifically, the first branch global feature extraction network includes a sequence branch network, which includes at least three one-dimensional convolutional layers, one max pooling layer, and four fully connected layers. Then, it is reshaped to form a U matrix of [batchsize, k, k], where batchsize is the number of targets and k is the dimension.
[0141] To ensure that U is an orthogonal matrix, it is obtained through training by constructing an orthogonal loss function;
[0142] The specific orthogonal loss function is: loss(U) = min(E – U * U) T )
[0143] Where loss(U) is the normal loss function, E is the unit vector, and U is the orthogonal matrix;
[0144] By training, the loss(U) approaches 0, and U can be obtained as a normal matrix.
[0145] The result obtained by multiplying an orthogonal matrix with the output of the corresponding convolutional layer is output to the next convolutional layer for computation.
[0146] See Figure 7 The center offset feature extraction network predicts the center offset of the 3D target object. Since the center of the 3D target may be accurate after the 3D target features are extracted by the first branch global feature extraction network, it is necessary to regress the center of the 3D target again to obtain the prepared center offset. Through the center offset, the center of the 3D target can be accurately obtained.
[0147] The center offset feature extraction network specifically includes: multiple one-dimensional convolutional layers, pooling layers, and fully connected layers;
[0148] Specifically, the optimal center offset feature extraction network model includes: three one-dimensional convolutional layers and one pooling layer. After the first one-dimensional convolutional layer, the number of channels becomes 128; after the second one-dimensional convolutional layer, the number of channels becomes 256; and after the third one-dimensional convolutional layer, the number of channels becomes 512.
[0149] After the max pooling layer, and then the fully connected layer, the result is regressed to the center offset (Δx, Δy, Δz) of the 3D target object.
[0150] Specifically, see Figure 8 The feature center transformation network processes the output of the center offset feature extraction network and the features of the first branch global feature extraction network, and then after three corresponding convolutional layers, the input feature vector conforms to the input feature vector of the second branch global feature extraction network.
[0151] See Figure 6 The second branch of the global feature extraction network further extracts more precise semantic features. After passing through the first branch of the global feature extraction network, it is necessary to regress a more accurate 3D target location.
[0152] The second branch of global feature extraction, after obtaining the center offset of the 3D target object, combines the global features obtained from the first branch of global feature extraction and extracts them again.
[0153] Specifically, the second branch of the feature extraction network includes: multiple one-dimensional convolutional layers, max pooling layers, and fully connected layers;
[0154] Specifically, the number of convolutional layers and fully connected layers can be set according to the actual situation;
[0155] For fully connected layers, when there are multiple fully connected layers, in order to avoid overfitting, a preset percentage of nodes are randomly selected for calculation, and the corresponding percentage of nodes are discarded.
[0156] Specifically, one of the best instances in the second branch global feature extraction network is: after the first one-dimensional convolution, the number of channels becomes 128; after the first one-dimensional convolution, the number of channels becomes 128.
[0157] After the second one-dimensional convolution, the number of channels becomes 256;
[0158] After the third one-dimensional convolution, the number of channels becomes 512;
[0159] After the third one-dimensional convolution, the number of channels becomes 1024;
[0160] Then, after the max pooling layer, the final output is an accurate global feature vector after passing through a fully connected layer.
[0161] Specifically, after obtaining the global graph feature vector, it is input into the third neural network for prediction.
[0162] Compared to the first and second neural networks, the third neural network does not use just one frame when regressing 3D targets, but uses multiple frames of continuous data for regression.
[0163] Specifically, in the current technology for unmanned delivery vehicles, which uses single-frame data, if a 3D target is occluded, the target cannot be identified in that frame. However, because the target is real, the unmanned delivery vehicle may not detect the target in time and continue driving normally. But when the target suddenly appears, there may not be enough time to react, leading to an accident. To address this problem, this embodiment makes the following improvements:
[0164] See Figure 9 The third neural network includes: LSTM network and fully connected layers;
[0165] The LSTM network is used to process the global features of the 3D target in multiple consecutive frames to extract temporary enhanced features of the target of interest. The temporary enhanced features are then input into a fully connected layer to regress the precise location of the 3D target.
[0166] Therefore, when performing calculations on the third neural network, regression is performed using LiDAR data and image data over a period of time. Through regression of multiple frames of data, objects that are obscured during the autonomous delivery vehicle's operation can be identified. This avoids situations where moving pedestrians, such as children, are obscured by ground structures or obstacles, causing the unmanned delivery vehicle to fail to recognize them.
[0167] Specifically, in an LSTM network, for example, five consecutive frames of image data are fed into the LSTM network, enabling the LSTM network to automatically complete the features of occluded objects based on the five frames of data.
[0168] LSTM networks can include multiple ones;
[0169] The specific structure of LSTM is as follows:
[0170] First, the forget gate determines the cell state C from the previous moment. t-1 The first step discards information from the old cell state; the second step determines the information that needs to be updated in the cell state. This process is further divided into two sub-steps: the first sub-step uses the input gate to determine the information in the old cell state that needs to be updated, and the second sub-step uses the tanh layer to generate candidate information for updating. The third step is to update the cell state based on the preparations made in the first two steps, and obtain a new cell state C. t Finally, the output gate filters and scales the new cell state to obtain the output information h(t) of the hidden layer.
[0171] All hidden layers use multi-layer LSTM units, and the appropriate number of hidden layers and the appropriate number of neurons hyperparameter values can be determined by grid search method.
[0172] Features are extracted using LSTM units and then input into a fully connected layer to regress detailed parameters of the 3D target bounding box.
[0173] Specifically, a loss function is calculated to measure the similarity of global features of 3D targets across multiple consecutive frames:
[0174] The loss functions for the 3D target in the second and third neural networks are as follows:
[0175] Loss_3d=Lseg+β(Lc+L box-reg +L h-reg +L h-cls +L s-reg +L s-cls )
[0176] Where Lseg represents the loss of the second neural network for 3D instance segmentation, and Lc represents the offset loss of the center offset feature extraction network.
[0177] L h-reg L h-cls These represent the regression loss and classification loss for the heading angle, respectively.
[0178] L s-reg L s-cls Let size represent the classification and regression losses for table size, respectively. Then size = (h, w, l). If we calculate the centroid, it represents (Cx, Cy, Cz), etc.
[0179] Cross-entropy loss is used for classification and point segmentation;
[0180] The loss function for the regression of center, frame size, and orientation angle is calculated as follows:
[0181]
[0182] Where Y represents the true value, Y* represents the predicted value, and δ represents the preset threshold.
[0183] Similarity is calculated using multi-frame data by constructing a loss function to provide information about the similarity of feature embeddings of the same object across consecutive frames. The loss function is calculated by taking the similarity difference between two feature vectors of the same object in consecutive frames, as follows: where v and ω are feature vectors of the object from consecutive frames, and l is the dimension of the vector.
[0184]
[0185] When the similarity loss of multiple consecutive input frames exceeds a certain threshold, the current prediction is filtered out, and the next prediction is performed.
[0186] By introducing similarity loss, large fluctuations in the input vector across multiple consecutive frames can be avoided, thereby improving the accuracy for partially occluded targets.
[0187] The above description represents only preferred embodiments of the present invention, and the present invention is not limited to the above embodiments. Those skilled in the art will understand that the forms in these embodiments are not limited thereto, nor are the adjustments possible. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the basic concept of the present invention should be considered to be included within the scope of protection of the present invention.
Claims
1. A 3D target detection device for unmanned delivery vehicles, characterized in that, At least including: Vehicle-mounted Ethernet cameras and LiDAR installed on the vehicle body; An autonomous driving host that receives image data from an in-vehicle Ethernet camera and point cloud data from a LiDAR sensor and performs 3D target detection; The autonomous driving host includes: a field of view correction module, a synchronous data matching module, and a 3D target detection module. The field of view correction module is used to match the field of view of the LiDAR and the vehicle camera based on their installation positions and imaging relationship. The synchronous data matching module is used to synchronize and match the frame data of the LiDAR with the frame data of the camera, so that the frame data of the LiDAR and the frame data of the camera are captured at the same time. The 3D target detection module is used to perform 3D recognition of targets ahead during driving; Specifically, the process includes: processing images captured by a vehicle-mounted camera using a first neural network model to obtain the location and category of a two-dimensional target in the image; mapping the location of the two-dimensional target to the target location in a three-dimensional point cloud image obtained by LiDAR imaging and selecting the corresponding point cloud data; inputting the category and point cloud data into a second neural network for processing to extract global features of the 3D target from the point cloud data; inputting the global features of the 3D target from multiple consecutive frames into a third neural network to regress the location of the 3D target; wherein the third neural network includes: at least one LSTM network and at least one fully connected layer; using the LSTM network to process the global feature vectors of the 3D target from multiple consecutive frames to extract a temporary enhanced feature vector of the target of interest, and inputting the temporary enhanced feature vector into the fully connected layer to regress the precise location of the 3D target; the second neural network includes: a first branch global feature extraction network, a center offset feature extraction network, a feature center change network, and a second branch global feature extraction network; wherein the first branch global feature extraction network extracts the first global features and then inputs them into the center offset feature extraction network. The first branch global feature extraction network and the feature center transformation network perform corresponding feature extraction. Then, the output of the center offset feature extraction network and the output of the feature center transformation network are concatenated and then input into the second branch global feature extraction network for the extraction of the second global features. The first branch global feature extraction network includes a sequence branch network, which trains an orthogonal matrix to rotate the input point cloud so that the entire network model can match point clouds with various rotation angles. The sequence branch network includes at least 3 one-dimensional convolutional layers, 1 max pooling layer, and 4 fully connected layers. Then, it is reshaped to form a U matrix of [batchsize, k, k], where batchsize is the number of targets and k is the dimension. To make U an orthogonal matrix, an orthogonal loss function is constructed and trained. The orthogonal loss function is: loss(U) = min(E – U*UT), where E is a unit vector and U is an orthogonal matrix. The loss(U) is trained to approach 0. The result obtained by multiplying the orthogonal matrix with the output of the corresponding convolutional layer is output to the next convolutional layer for computation.
2. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, The synchronous data matching module is used to register the data acquired by the LiDAR and the vehicle camera. Specifically, it includes: calculating the time delay and frequency offset of the vehicle Ethernet camera and the LiDAR to the autonomous driving host according to the gPTP protocol. The vehicle-mounted Ethernet camera and LiDAR are time-synchronized and calibrated according to their respective time delays and frequency offsets. Using the time point of the point cloud data sent by the LiDAR received by the autonomous driving host as a reference benchmark, the image data of the previous frame and the image data of the next frame corresponding to the point cloud data of the current LiDAR are obtained. The registered image data is obtained by linear interpolation and the current reference timestamp is marked on the registered image data.
3. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, The field-of-view correction module is used to achieve field-of-view matching between the LiDAR and the vehicle-mounted camera. Specifically, it includes: A checkerboard method is used to obtain the calibration matrix, which is then used to crop the input point cloud. Each point p(x, y, z) is projected onto the image plane using the calibration matrix, retaining only points within the image plane. This ensures that the data from both the LiDAR and camera sensors have the same field of view. The projection formula is: Where K is the intrinsic parameter calibration matrix of the camera, R and t are the rotation matrix and translation vector describing the position between the lidar and the camera, and (u, v) are the image coordinates corresponding to point p(x, y, z).
4. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, All hidden layers use multi-layer LSTM units, and the optimal values of the number of hidden layers and the number of neurons are determined by a grid search method. Features are extracted using LSTM units and then input into a fully connected layer to regress detailed parameters of the 3D target bounding box.
5. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, A loss function is applied to the global feature vectors of 3D targets in consecutive frames to control the similarity, so as to provide that the feature embeddings of the same object are similar in consecutive frames; The similarity difference between two feature vectors of the same object in consecutive frames is used for calculation. v and ω are feature vectors of objects from consecutive frames, and l is the dimension of the vector; When the cosine distance loss of multiple consecutive frames of input objects exceeds a certain threshold, the current prediction is filtered out, and the next prediction is performed.
6. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, The first neural network model is a two-dimensional convolutional neural network model, specifically including: multi-layer residual network units, a first convolutional feature layer, a second convolutional feature layer, a third convolutional feature layer, and a fourth convolutional feature layer. Among them, the multi-layer residual network units are used to process the input image data and extract features from the input size and number of channels of the image data according to the model parameters of the first convolutional feature layer. From the first convolutional feature layer to the fourth convolutional feature layer, each time a convolutional layer is passed, the size is reduced to twice its original size, and the number of channels is increased to twice its original size.
7. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, The first neural network model also includes: a first prediction layer, a second prediction layer, a third prediction layer, a fourth prediction layer, and a merging layer; From top to bottom, starting from the fourth convolutional feature layer to the first convolutional feature layer, the output of the lower layer and the output of the corresponding upper layer are upsampled by 2 times through the deconvolution layer and then added to form the third prediction layer, the second prediction layer, and the first prediction layer in sequence. The fourth prediction layer is the same as the fourth convolutional feature layer; The first to fourth prediction layers are merged to form a merged layer; After the merged layer passes through a 1*1 convolutional layer, regression is performed to obtain the 2D target location information and the corresponding category of the 2D target.
8. The 3D target detection device for unmanned delivery vehicles as described in claim 1, characterized in that, The loss function of the first neural network includes: localization loss, The formula for calculating the localization loss function is as follows: h = |h1 - h2 | , w= |w1 -w2 | Where h1, w1, h2, and w2 are the height and width of the target bounding box and the predicted bounding box, respectively; IoU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box; The Euclidean distance r between the center points of the predicted bounding box and the target bounding box; Ch and Cw are the length and width of the minimum bounding rectangle that covers the prediction box and the target box, respectively.
Citation Information
Patent Citations
Three-dimensional target detection method, system and device based on RGB-D
CN111723721A
Vehicle-mounted time sequence synchronization system
CN111917709A
Apple picking method based on improved deep neural network
CN112734727A
Unmanned aerial vehicle obstacle detection and avoidance method and system based on image and point cloud
CN114581831A
