3D vehicle target detection method fusing image texture features and prior information

Abstract

The invention provides a 3D vehicle target detection method fusing image texture features and prior information, and the method comprises the steps: obtaining a data set, carrying out the point cloud and image fusion based on a pixel level based on the point cloud data and image data in the data set, carrying out the detection of the image data based on a preset image target detector, and generating an image two-dimensional target frame. And detecting the fused point cloud data based on a preset three-dimensional vehicle target detection model to generate a point cloud three-dimensional target frame, and fusing the image two-dimensional target frame and the point cloud three-dimensional target frame based on a decision-level fusion scheme of feature distance and channel attention to obtain a final three-dimensional vehicle target detection result. According to the method, the precision and the stability of three-dimensional vehicle detection are remarkably improved by fusing the image texture features and the prior geometric information, the method is particularly excellent in performance in a complex environment, the real-time performance of the system is ensured by adopting an efficient geometric optimization and depth information combination technology, and the actual application requirements of automatic driving and intelligent traffic are met.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional vehicle target detection, and in particular to a 3D vehicle target detection method that integrates image texture features and prior information. Background Art

[0002] With the rapid development of autonomous driving and intelligent transportation systems, accurate and real-time three-dimensional vehicle target detection technology is particularly important. At present, common target detection methods mainly rely on deep learning technology, detect based on two-dimensional images, and then obtain three-dimensional information through post-processing steps. However, such methods usually do not handle the posture, angle, and occlusion of the target object well. In addition, existing methods rarely make full use of the vehicle's prior information and image texture features for detection, resulting in insufficient detection accuracy and robustness. Therefore, it is very necessary to design a 3D vehicle target detection method that integrates image texture features and prior information. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, an object of the present invention is to provide a 3D vehicle target detection method that integrates image texture features and prior information.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] The present invention provides a 3D vehicle target detection method integrating image texture features and prior information, comprising:

[0006] Obtain a data set, and perform pixel-level point cloud and image fusion based on the point cloud data and image data in the data set;

[0007] Detect the image data based on a preset image target detector to generate a two-dimensional target frame of the image;

[0008] Detect the fused point cloud data based on the preset 3D vehicle target detection model to generate a point cloud 3D target frame;

[0009] A decision-level fusion scheme based on feature distance and channel attention is used to fuse the image 2D target box and the point cloud 3D target box to obtain the final 3D vehicle target detection result.

[0010] Preferably, the data set is a KITTI data set.

[0011] Preferably, pixel-level point cloud and image fusion is performed based on the point cloud data and image data in the data set, specifically:

[0012] The image target prior information is obtained from the two-dimensional target real box label file of the KITTI dataset. The target category and target box range in the image are obtained based on the image target prior information. After mapping the point cloud to the image, the point cloud located in the image target box is screened out according to the target box range. The category information assignment method based on Gaussian distance encoding is used to determine the probability that the point cloud in the viewing cone belongs to the foreground target point through the Gaussian distance feature information. The probability is assigned to the point cloud data as the category score, and the target category information is assigned to these point clouds. On this basis, the corresponding pixel R, G, B color values ​​are assigned to each point cloud to generate point cloud data with rich feature information.

[0013] Preferably, the image data is detected based on a preset image target detector to generate an image two-dimensional target frame, specifically:

[0014] Build an image object detector based on the CascadeRCNN algorithm;

[0015] Train the image object detector based on a preset training dataset;

[0016] The image is input into the trained image object detector for detection to generate a two-dimensional object box for the image.

[0017] Preferably, the fused point cloud data is detected based on a preset three-dimensional vehicle target detection model to generate a point cloud three-dimensional target frame, specifically:

[0018] Construct a 3D vehicle target detection model based on the improved PointPillars network model;

[0019] The three-dimensional vehicle target detection model is trained based on the KITTI data set to obtain a trained three-dimensional vehicle target detection model;

[0020] The fused point cloud data is input into the trained 3D vehicle target detection model to generate a point cloud 3D target box.

[0021] Preferably, a three-dimensional vehicle target detection model is constructed based on the improved PointPillars network model, specifically:

[0022] The PointPillars network model is improved by adding a point cloud local attention mechanism module to the body column feature network of the traditional PointPillars network model to capture the complex features in a specific area of ​​the input point cloud. The SE attention mechanism module is integrated into the two-dimensional pseudo image network of the traditional PointPillars network model to enhance the network's ability to obtain global features and information.

[0023] A three-dimensional vehicle target detection model is constructed based on the improved PointPillars network model.

[0024] Preferably, the decision-level fusion scheme based on feature distance and channel attention is used to fuse the image 2D target frame and the point cloud 3D target frame to obtain the final 3D vehicle target detection result, specifically:

[0025] Encode the image 2D target box and point cloud 3D target box respectively, fuse the category confidence with the target box position, and generate their respective encoding tensors;

[0026] Based on semantic consistency and geometric consistency, the encoding tensors of the image 2D target box and the point cloud 3D target box are fused to obtain a fused feature tensor.

[0027] The fused feature tensor is input into the convolutional neural network structure based on channel attention encoding, and finally the fused confidence score is obtained;

[0028] The fusion confidence scores are redistributed to the point cloud 3D target boxes, and the non-maximum suppression (NMS) algorithm is used to filter redundant and falsely detected target boxes to generate the final 3D vehicle target detection results.

[0029] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0030] The present invention provides a 3D vehicle target detection method that fuses image texture features and prior information, the method comprising acquiring a data set, performing pixel-level point cloud and image fusion based on point cloud data and image data in the data set, detecting the image data based on a preset image target detector to generate an image two-dimensional target frame, detecting the fused point cloud data based on a preset three-dimensional vehicle target detection model to generate a point cloud three-dimensional target frame, and fusing the image two-dimensional target frame and the point cloud three-dimensional target frame based on a decision-level fusion scheme of feature distance and channel attention to obtain a final three-dimensional vehicle target detection result. The present invention significantly improves the accuracy and stability of three-dimensional vehicle detection by fusing image texture features with prior geometric information, especially in complex environments. In addition, the use of efficient geometric optimization and depth information combination technology ensures the real-time performance of the system and meets the practical application requirements of autonomous driving and intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of KITTI laser radar coordinate system and camera coordinate system;

[0034] Figure 3 This is the result of fusion of point cloud and image data;

[0035] Figure 4 It is the point cloud image of the vehicle target frustum;

[0036] Figure 5 This is a point cloud visualization scene diagram from a forward perspective (red represents the target point cloud);

[0037] Figure 6 It is a point cloud visualization image fused with RGB information at the target box level;

[0038] Figure 7 It is a point cloud visualization diagram based on the RGB information fusion of the scene;

[0039] Figure 8 This is a schematic diagram of the PointPillars network algorithm framework;

[0040] Fig. 9 This is a schematic diagram of the improved PointPillars network algorithm framework;

[0041] Fig.10 It is a geometric consistency relationship diagram under the same goal;

[0042] Fig.11 Schematic diagram of the convolutional network structure based on channel attention encoding;

[0043] Fig.12a Visualize result graphs for some scenarios of other algorithms;

[0044] Figure 12b This is a visualization result diagram of some scenes of the algorithm of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The purpose of the present invention is to provide a 3D vehicle target detection method that integrates image texture features and prior information, which can realize three-dimensional vehicle target detection based on the integrated image texture features and prior information and is easy to use.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a 3D vehicle target detection method that integrates image texture features and prior information, including:

[0049] Step 100: Acquire a data set, and perform pixel-level point cloud and image fusion based on point cloud data and image data in the data set;

[0050] Step 200: Detect the image data based on a preset image target detector to generate an image two-dimensional target frame;

[0051] Step 300: Detect the fused point cloud data based on a preset three-dimensional vehicle target detection model to generate a point cloud three-dimensional target frame;

[0052] Step 400: A decision-level fusion scheme based on feature distance and channel attention is used to fuse the image 2D target frame and the point cloud 3D target frame to obtain a final 3D vehicle target detection result.

[0053] The dataset is a KITTI dataset. KITTI uses a vehicle-mounted laser radar to collect point cloud data and image data. Since the collected image data does not have depth information, it is difficult to map the image to the point cloud scene using traditional methods. For the fusion of point cloud and image data, most of the current methods are to map the point cloud to the image, reduce the dimensionality of the three-dimensional data to two-dimensional data and retain the depth information. In order to achieve coordinate conversion, the KITTI dataset provides the camera and laser radar calibration parameters for each frame of data. According to the calibration parameters, the conversion matrix from the three-dimensional laser radar coordinate system to the two-dimensional image coordinate system can be calculated to achieve point cloud mapping. Since the laser radar point cloud sampling range is wide and the field of view of the entire point cloud scene is 360°, considering the field of view of the camera sampling image and the point cloud distribution in the real traffic scene, the point cloud outside the image field of view is discarded during the point cloud projection process, and only the point cloud under the forward perspective is retained. Figure 2 The coordinate axis directions of the lidar coordinate system and the camera coordinate system in the KITTI dataset are shown. The X-axis of the lidar coordinate system represents forward, the Y-axis represents left, and the Z-axis represents upward; the X-axis of the camera coordinate system represents right, the Y-axis represents downward, and the Z-axis represents forward.

[0054] (1) Converting LiDAR point cloud coordinates to camera point cloud coordinates

[0055] To convert the point cloud coordinates from the LiDAR coordinate system to the camera coordinate system, a calibration extrinsic matrix, namely the Tr_velo_to_cam conversion matrix, is required, which contains the rotation matrix R and the translation matrix t. This matrix is ​​a 3×4 matrix, and after removing the reflection intensity from the input LiDAR point cloud coordinates, it is an N×3 matrix containing the number of point clouds N and the original three-dimensional coordinate points in the LiDAR coordinate system. To ensure that the output point cloud coordinates in the camera coordinate system are also an N×3 matrix, the original point cloud needs to be dimensionally expanded, filled with a column of 1, and then transformed into an N×4 matrix. After multiplying it with the transposed extrinsic matrix, the point cloud coordinates in the camera coordinate system are obtained. Since KITTI provides a total of 4 cameras, and the Tr_velo_to_cam matrix only converts the LiDAR point cloud coordinates into the point cloud coordinates in the camera coordinate system No. 0, in order to ensure that other cameras can use the converted point cloud coordinates, it is necessary to multiply it with the correction rotation matrix R0_rect of the camera No. 0. The function of this matrix is ​​to make the imaging of all cameras coplanar while ensuring that the optical centers of the four cameras are located on the same xoy plane. The overall calculation process is shown as follows:

[0056]

[0057] Among them, rect0 represents the point cloud coordinate matrix in the uncorrected camera coordinate system 0, lidar represents the point cloud matrix in the radar coordinate system, R represents the rotation matrix with a size of 3×3, t represents the translation matrix with a size of 3×1, and rect_R0 represents the point cloud coordinate matrix in the corrected camera coordinate system 0. During the calculation process, the [R|t] matrix and the R0_rect matrix need to be transposed.

[0058] (2) Converting camera point cloud coordinates to image point cloud coordinates

[0059] In order to convert the point cloud coordinates from the camera coordinate system to the image coordinate system, the camera intrinsic calibration matrix is ​​needed, that is, P i Matrix, P i The matrix formula is as follows:

[0060]

[0061] Among them, f u and f v is the focal length of the camera, c u and c v It refers to the offset from the optical center to the CMOS zero point, that is, the coordinate of the camera optical center in the image coordinate system, b x It refers to the distance offset from the current camera to camera No. 0 in the x-axis direction.

[0062] Using color camera No. 2 as the conversion standard, the calculated point cloud coordinates in the camera coordinate system are multiplied by the internal reference P2 of camera No. 2 to generate the point cloud coordinates in the image coordinate system. Since P2 is a 3×4 matrix, and the output point cloud coordinates in the camera coordinate system are an N×3 matrix, it is necessary to fill the point cloud coordinates with 1 in the columns, generate an N×4 matrix and multiply it with the transposed matrix of P2 to obtain the point cloud coordinate results in the image coordinate system. In order to obtain homogeneous coordinates in the image coordinate system, it is necessary to normalize in the depth direction of the z-axis. Divide the point cloud coordinates in each image coordinate system by the depth information to obtain the final coordinate result, as shown in the following formula:

[0063]

[0064] Among them, cam represents the point cloud coordinate matrix in the image coordinate system, and cam / z represents the normalized point cloud matrix. The image depth map is generated according to the calculated point cloud coordinates in the image coordinate system. First, the point clouds that are not in the image perspective are filtered according to the length and width of the image. At the same time, since the z-axis represents forward in the camera coordinate system, it is necessary to filter out the point clouds with z-axis depth values ​​less than 0. These point clouds are not in the forward perspective. In order to make each point cloud a valid value and since the coordinates of each pixel point in the image are integers, the point cloud coordinates need to be integerated to ensure that each point cloud can be correctly mapped to the image pixel point. The calibration file provided by KITTI is shown in Table 1, and the image and point cloud data fusion diagram is shown in Figure 3 As shown;

[0065] Table 1 KITTI dataset calibration file format example table

[0066]

[0067]

[0068] In step 100, pixel-level point cloud and image fusion is performed based on the point cloud data and image data in the data set, specifically:

[0069] The image target prior information is obtained from the two-dimensional target real box label file of the KITTI dataset. The target category and target box range in the image are obtained based on the image target prior information. After the point cloud is mapped to the image, the point cloud located in the image target box is screened out according to the target box range. Based on the Gaussian distance encoding category information assignment method, the probability of the point cloud in the frustum belonging to the foreground target point is determined by the Gaussian distance feature information. The probability is assigned to the point cloud data as a category score, and the target category information is assigned to these point clouds. On this basis, the corresponding pixel R, G, B color values ​​are assigned to each point cloud to generate point cloud data with rich feature information. The specific process is introduced as follows:

[0070] First, the pixel-level fusion based on target prior information and Gaussian coding is introduced. PointPainting first uses a semantic segmentation network to perform semantic segmentation on the original image, obtains the category and category score of each pixel, maps the point cloud to the image, and then adds the pixel category score to the corresponding point cloud according to the correspondence between the pixel and the mapped point cloud, generating point cloud data with 8 dimensions (x, y, z, r, s1, s2, s3, s4), and putting the generated new point cloud data into a three-dimensional target detector for three-dimensional target detection. Since the algorithm requires that the semantic segmentation model and the three-dimensional target detection model have high coupling, the scope of application of the algorithm is limited. To solve this problem, the present invention proposes a point cloud and image fusion method based on image target prior information, which uses the prior results obtained by image target detection to expand the information of the original point cloud and generate point cloud data containing rich information. Since both two-dimensional target detection and three-dimensional target detection belong to target detection tasks, the two are combined in a low-coupling manner. In addition, during the training process, the KITTI dataset provides a two-dimensional target real box label file, and training data can be generated directly based on the image target prior information given in the file. There is no need to re-use the two-dimensional detection model to perform target detection on the training images and then generate point cloud fusion results, which enhances the practicality of the algorithm and reduces the difficulty of data fusion.

[0071] The image target prior information provides the target category and target frame range in the current image. After mapping the point cloud to the image, the point cloud within the image target frame is filtered out according to the target frame range, and the target category information is assigned to these point clouds, which can provide more detailed information in the subsequent three-dimensional target detection process. Since the point cloud data collected by the lidar is in three-dimensional space, and the image data collected by the camera is in two-dimensional space, the depth information is lost when the point cloud is mapped to the image, which is equivalent to compressing the three-dimensional plane to the two-dimensional plane, resulting in the point cloud in the image target frame being mapped back to the three-dimensional space and presenting a three-dimensional cone shape, such as Figure 4 As shown in the figure, it can be seen that the point cloud in the viewing cone can fully present the shape of the current target, but due to the compression in the depth direction, a large number of background points are in the two-dimensional target box. Therefore, assigning the same numerical category score to all point clouds in the viewing cone will produce a large number of background noise point clouds. To address the above problems, this section designs a category information assignment method based on Gaussian distance encoding. The probability of the point cloud in the viewing cone belonging to the foreground target point is determined through the Gaussian distance feature information, and the probability is assigned to the point cloud data as a category score to enrich the original point cloud information.

[0072] After observation, it is found that the coordinate center point of the two-dimensional target frame is located on the detection target, and the point cloud within a certain range of the center point is most likely the target point cloud. Therefore, this paper uses Gaussian distance as the standard to measure the position distance between the point cloud in the target frame and the center point of the two-dimensional target frame. The larger the Gaussian distance, the closer the point cloud is to the center of the target frame on the two-dimensional image, and the more likely it is the target point cloud. The smaller the distance, the farther the point cloud is from the center of the target frame on the two-dimensional image, and the more likely it is the background point cloud. The Gaussian distance calculation formula is shown as follows:

[0073]

[0074] Among them, (x, y) represents the point cloud coordinates mapped to the image target box. is the center coordinate of the two-dimensional target box, w and h represent the width and height of the two-dimensional target box respectively. The present invention only detects vehicle targets, so non-vehicle targets are regarded as background points in the process of generating category information. The calculated Gaussian distance information is assigned to the point cloud as the category information of the current target, and a total of 6-dimensional point cloud data (x, y, z, r, s1, s2) is generated, where s1 represents the probability that the current point cloud is a vehicle target point cloud, and s2 represents the probability that the current point cloud is a background point cloud. The expanded point cloud data is mapped back to the three-dimensional space to obtain the following: Figure 5 The visualized scene graph shown in the figure only visualizes the point cloud under the forward perspective for easy observation. The red part in the figure represents the point cloud under the viewing cone, that is, the point cloud data within the image target detection frame, and the black part represents the background point cloud data.

[0075] Secondly, the pixel-level fusion based on image RGB texture features is introduced:

[0076] Image data, as a two-dimensional data containing rich texture and color information, is mainly represented by a two-dimensional matrix composed of several pixels, and each pixel contains three color channels R, G, and B. Point cloud data, as a data set containing several point clouds, only provides the three-dimensional coordinates x, y, z and reflection intensity r of the original point cloud. In the two-dimensional target detection task, RGB information provides sufficient image information for the convolutional neural network, ensuring the accuracy of target detection. Based on this, the present invention designs a fusion method based on the image RGB texture features on the basis of fusing the image target prior information, assigns the corresponding pixel R, G, B color values ​​to each point cloud, and combines the image target prior result information to generate point cloud data with rich feature information, including (x, y, z, r, s1, s2, R, G, B) a total of 9 dimensions of data;

[0077] The present invention provides two image RGB texture information and point cloud fusion strategies, which are divided into target frame-level RGB information fusion strategy (method 1) and scene-based RGB information fusion strategy (method 2) according to different fusion areas:

[0078] Method 1: Assign RGB information to several target frustum point clouds obtained from the image target prior information, so that the subsequent 3D target detector can fully focus on the target area point cloud and realize the target classification and bounding box regression tasks, such as Figure 6 As shown;

[0079] Method 2: Assign RGB information to the point cloud from the perspective of the entire scene. Assign values ​​to all point clouds within the image viewing angle to ensure that the point cloud has sufficient context structure. The subsequent 3D object detector can extract the entire scene when performing feature extraction, preventing the design of method 1 from losing some key point cloud information, such as Figure 7 shown.

[0080] In step 200, the image data is detected based on a preset image target detector to generate an image two-dimensional target frame, specifically:

[0081] Build an image object detector based on the CascadeRCNN algorithm;

[0082] Train the image object detector based on a preset training dataset;

[0083] The image is input into the trained image object detector for detection to generate a two-dimensional object box for the image.

[0084] In step 300, the fused point cloud data is detected based on a preset three-dimensional vehicle target detection model to generate a point cloud three-dimensional target frame, specifically:

[0085] Construct a 3D vehicle target detection model based on the improved PointPillars network model;

[0086] The three-dimensional vehicle target detection model is trained based on the KITTI data set to obtain a trained three-dimensional vehicle target detection model;

[0087] The fused point cloud data is input into the trained 3D vehicle target detection model to generate a point cloud 3D target box.

[0088] A three-dimensional vehicle target detection model is constructed based on the improved PointPillars network model, specifically:

[0089] The PointPillars network model is improved by adding a point cloud local attention mechanism module to the body column feature network of the traditional PointPillars network model to capture the complex features in a specific area of ​​the input point cloud. The SE attention mechanism module is integrated into the two-dimensional pseudo image network of the traditional PointPillars network model to enhance the network's ability to obtain global features and information.

[0090] Construct a 3D vehicle target detection model based on the improved PointPillars network model;

[0091] A detailed description of this process:

[0092] like Figure 8 As shown in the figure, the main components of the traditional PointPillars network model include the Pillar Feature Net, the Backbone, and the SSD Detection Head. First, the point cloud data is preprocessed and feature extracted, and then the data is projected into a bird's-eye view through the PFN module, and a feature representation of each voxel is generated. PSE is used to encode the location information of the voxel to improve the network's ability to understand the spatial structure. Next, the target proposal generator generates candidate target detection boxes based on the feature map, and finally the target detection head predicts the existence, category, and location information of the target based on these boxes.

[0093] The present invention improves the traditional PointPillars network model as follows: First, a point cloud local attention mechanism network is added to the point cloud pillar feature extraction module to extract rich features for a single point cloud. This attention mechanism allows the feature information of the surrounding point clouds to be more fully integrated during the feature extraction stage. Secondly, a channel attention mechanism module is added to the two-dimensional pseudo image backbone network. The final features are input into the SSD algorithm for target detection, and the overall network structure is shown in 9.

[0094] The main function of the point cloud local attention module is to enhance the representation ability of point cloud data by introducing an attention mechanism. It enables the network to learn important information from the features of the point cloud while reducing the acquisition of irrelevant information. The input point cloud data is P, and each point cloud contains its own position coordinates (x, y, z). The point cloud to be tested undergoes a series of convolution operations to convert the input data into query, key, and value tensors with different channel dimensions. The similarity between the query and the key is then calculated, and the values ​​are weighted summed using the similarity to obtain the final output representation. The code divides the input data into local regions and adjusts the values ​​of the query, key, and value tensors accordingly. Then the attention score is calculated to determine the importance of the elements in each local region. After further convolution and reshaping, the final output features are generated. This local attention module captures the local neighbor point information of a specific point cloud in the body column and outputs the features of the key information of the local point cloud sampling points around the point cloud.

[0095] The present invention adopts the idea of ​​SE attention mechanism and adds the channel attention mechanism module to the two-dimensional pseudo image backbone network. Specifically, the BaseBackbone model class is defined in Pytorch, and the channel attention mechanism sub-network is added. First, the input feature map is reduced in dimension, and the input three-dimensional tensor (C, H, W) is reduced in dimension to (1, 1, C) using the global average pooling layer, so that the tensor can be regarded as a channel dimension descriptor. Next, through two linear layers and the ReLU activation function, the nonlinear interaction between channels is learned. Finally, the channel weight is mapped to between 0 and 1 through the Sigmoid activation function to form a channel attention weight.

[0096] The detection module uses Single Shot MultiBox Detector (SSD) for object detection, which is a widely used single-stage detection algorithm known for its fast detection speed and high accuracy. In the SSD network, the concept of anchors is introduced to adapt to the multi-scale object detection task, making it more suitable for capturing large-scale transformations in point cloud data. The SSD architecture consists of six main modules. The first module includes the initial five layers (Conv1-Conv5) of the VGG16 convolutional network. Subsequently, the FC6 layer of VGG16 is merged in and the FC7 fully connected layer is converted to a Conv6 layer. The Conv7 layer forms the second module. In order to extract object information at different scales, four additional modules are introduced, namely Conv8, Conv9, Conv10 and Conv11 convolutional layers. The final stage involves object classification, detection and non-maximum suppression of the regressed positions. Non-maximum suppression uses the two-dimensional Intersection over Union (IoU) technique to match the prior bounding box with the actual target, ensuring that only the most relevant bounding boxes are retained and eliminating redundant detections.

[0097] The decision-level fusion scheme based on feature distance and channel attention fuses the image 2D target box and the point cloud 3D target box to obtain the final 3D vehicle target detection result, which is as follows:

[0098] Encode the image 2D target box and point cloud 3D target box respectively, fuse the category confidence with the target box position, and generate their respective encoding tensors;

[0099] Based on semantic consistency and geometric consistency, the encoding tensors of the image 2D target box and the point cloud 3D target box are fused to obtain a fused feature tensor.

[0100] The fused feature tensor is input into the convolutional neural network structure based on channel attention encoding, and finally the fused confidence score is obtained;

[0101] The fusion confidence score is redistributed to the point cloud 3D target frame, and the non-maximum suppression (NMS) algorithm is used to filter out redundant and falsely detected target frames to generate the final 3D vehicle target detection result.

[0102] The above process is introduced in detail. First, the relationship between semantic consistency and geometric consistency is introduced as follows:

[0103] The decision-level fusion scheme independently detects the data collected by each sensor, and fuses the detection results of each part through the fusion module. In this fusion scheme, the data of different sensors are processed independently, and the sensors do not interfere with each other. That is, when one of the sensors fails, it does not affect the overall detection process, thereby improving the robustness of the detection. The scheme includes three modules in the process of point cloud and image fusion: image target detection, point cloud target detection, and target frame fusion module. The present invention uses a method of fusing image RGB texture features and image target prior information to generate a fused point cloud, and uses the improved PointPillars network model to perform point cloud vehicle target detection on the fused point cloud. At the same time, the existing CascadeRCNN algorithm is used as the image target detector for image vehicle target detection. The two different detection results are fused at the decision level before the non-maximum suppression algorithm. In order to ensure the effectiveness of the fusion module and improve the target detection accuracy, the present invention designs a fusion module based on feature distance and channel attention coding on the basis of the CLOCs algorithm;

[0104] The detection results obtained by different sensors have no correlation in category confidence scores. From the perspective of geometric consistency and semantic consistency, the present invention encodes the image target detection results and the point cloud target detection results into feature tensors with consistent associations, performs feature extraction and pooling on the feature tensors, and remaps the generated probability scores back to the three-dimensional target detection results.

[0105] (1) Geometric consistency

[0106] The 3D target detection result box is essentially a bounding box generated by drawing lines from 8 angle data found in a 3D scene. Therefore, similar to point cloud data, the 8 corner point data of the 3D target detection result box can be projected onto a 2D plane, and the intersection over union (IoU) with the 2D target box is calculated to quantify the geometric consistency of the two different detection result boxes. In addition, in view of the problem that the CLOCs method only uses the intersection over union (IoU) as a representation method to describe the geometric relationship between the 2D target box and the 3D target box, and the representation relationship is relatively simple, a method based on feature distance encoding is designed to expand the geometric consistency. Fig.10 As shown in the figure, after projecting the 3D target frame onto the 2D plane, if the 3D target frame and the 2D target frame in the image represent the same target, they have a larger intersection over union (IoU). Correspondingly, the feature distance between the center points of the two target frames is smaller, indicating that the two target frames belong to the same target. This section uses the Euclidean distance as the feature distance to measure the center point of the target frame. The calculation method is shown in the following formula:

[0107]

[0108] Among them, (x 3D ,y3D ) represents the coordinates of the center point of the 3D target detection box, (x 2D ,y 2D ) represents the coordinates of the center point of the two-dimensional object detection box.

[0109] (2) Semantic consistency

[0110] In the target detection task, one frame of data may contain multiple targets, so the generated target detection results may include target detection boxes under multiple categories and corresponding category confidence scores. In order to achieve category confidence association of target detection results of different sensor data, only candidate detection is performed on target detection boxes of the same category to achieve semantic consistency association.

[0111] Target box encoding method:

[0112] Before generating a fused feature tensor based on semantic consistency and geometric consistency, it is necessary to encode the image detection result box and the point cloud detection result box separately, fuse the category confidence with the target box position, and generate their respective encoding tensors.

[0113] The image target detection result box contains the coordinate points of the two-dimensional target box and the category confidence score. All detection results of the same category in a frame of data are encoded. This article only focuses on vehicle targets. Assuming that there are m two-dimensional vehicle target detection result boxes, the encoding process is as follows:

[0114]

[0115] Among them, P 2D Represents the set of m target detection result boxes in a frame of image data, P i 2D is the target detection result box of the i-th image, s i 2D represents the category confidence score of the i-th result box, (x i1 2D ,y i1 2D ) and (x i2 2D ,y i2 2D ) represent the coordinate values ​​of the upper left corner and lower right corner of the two-dimensional target box respectively;

[0116] The point cloud target detection result box contains the center point coordinates of the 3D target box, the size of the 3D target box, and the category confidence score. Assuming that a frame of point cloud data contains n vehicle target detection result boxes, the same encoding method as the image target box is used. The encoding process is as follows:

[0117]

[0118] Among them, P 3D Represents the set of n target detection box encoding results in a frame of point cloud data, P i 3D is the i-th point cloud target detection result box, (x i ,y i , x i ,h i , w i , l i ,θ i ) represent the center point coordinates, height, width, length and orientation angle of the 3D target detection box, s i 3D Represents the category confidence score of the i-th target detection box;

[0119] Based on the encoding results of the integrated image target frame and the point cloud target frame, for a frame of data with m two-dimensional detection result frames and n three-dimensional detection result frames, a fusion tensor T of size m×n×4 is constructed, as shown in the following formula:

[0120]

[0121] Among them, T i,j Represents the fusion tensor generated by the i-th two-dimensional target result box and the j-th three-dimensional target result box, IoU i,j represents the intersection-over-union ratio of the j-th 3D target result box projected onto the image and the i-th 2D target box, s i 2D and i 3D are the category confidence scores of the i-th two-dimensional target result box and the j-th three-dimensional target result box, respectively, and d j Represents the normalized distance from the jth 3D target result box to the lidar plane;

[0122] At the same time, according to the encoding results of the image target frame and the point cloud target frame, the center point feature distance is calculated to construct a feature distance tensor of size m×n×1, as shown in the following formula:

[0123] D i,j ={dlc i,j}

[0124] Among them, dlc i,j Represents the Euclidean distance between the projection of the center point of the i-th 2D target box and the center point of the j-th 3D target box.

[0125] CLOCs designed a convolutional neural network based on a fully connected structure. By inputting the fused target box encoding tensor into the network, it performs feature dimensionality upgrade and maximum pooling to generate a new probability score map. However, since the network only extracts features on the same target pair (an element in the fused tensor set) during the feature dimensionality upgrade process, the correlation between different target pairs is ignored, resulting in the importance of different features being neglected. In response to the above problems, the present invention designs a convolutional neural network structure based on channel attention encoding, which uses a two-dimensional convolution kernel of size 1×1 for feature dimensionality upgrade, and introduces channel attention to strengthen the correlation between different target pairs. The network structure is as follows Fig.11 As shown;

[0126] The resulting fused tensor T i,j and feature distance tensor D i,j There are a lot of empty elements in , because the number of two-dimensional target detection frames is much smaller than that of three-dimensional target detection frames and the fusion tensor and feature tensor can only be constructed when two target detection frames have an intersection. Therefore, before feature extraction, it is necessary to find the non-empty elements p of the two tensors and perform convolution operations on the non-empty elements p to reduce computing memory and improve computing efficiency. i,j First, a 4-layer convolutional neural network as shown in Table 2 is used to expand the feature vector to 96 dimensions. After each layer of convolution, the ReLU activation function is used to filter out feature values ​​less than or equal to 0, so as to ensure that the generated fusion confidence scores are all greater than 0. In order to strengthen the association between different target pairs, the channel attention module is introduced, and the maximum pooling and average pooling are performed on the non-empty element dimension respectively to obtain the association scores of different target pairs and multiply them with the fusion tensor after dimension increase to generate a fusion tensor with rich semantic information. The maximum pooling method is used to obtain the probability distribution map based on the fusion vector. For the feature distance tensor D i,j , a small 4-layer convolutional neural network and maximum pooling as shown in Table 3 are used to obtain the feature distance weight score, which is multiplied and corrected with the probability distribution map obtained by the fusion tensor branch to obtain the final fusion confidence score. The fusion confidence score is redistributed to the 3D detection target, and the non-maximum suppression NMS algorithm is used to filter redundant and falsely detected target boxes to generate the final 3D vehicle target detection result.

[0127] Table 2 Fusion tensor branch convolution network structure

[0128]

[0129]

[0130] Table 3 Feature distance tensor branch convolution network structure

[0131]

[0132] The present invention uses the Focal Loss function as the loss function of the decision-level fusion module to solve the problem of imbalance between foreground and background, as shown in the following formula:

[0133]

[0134] Among them, L total Represents the total loss value, N pos Represents the number of positive samples, N a Represents the total number of generated 3D target boxes; c i represents the true value of the i-th target box, c i 1 represents the vehicle target, c i 0 represents background target; p i is the category confidence prediction value of the i-th 3D detection box. γ is the hard and easy sample factor hyperparameter, which aims to enhance the model's learning of difficult samples; α is the balance factor hyperparameter, which aims to adjust the ratio of positive and negative samples. The present invention sets α=0.25 and γ=2.

[0135] The experimental results of the present invention are analyzed as follows:

[0136] (1) Performance comparison between point cloud and image fusion 3D object detector and single point cloud 3D object detector

[0137] The present invention adopts AP 3D _R11 and AP 3D _R40 two average detection accuracy indicators are used to evaluate the performance of the multimodal fusion algorithm proposed in this invention on the KITTI 3D Object dataset. When the R11 method is used to calculate the average detection accuracy, the proposed method achieves 90.35%, 80.39%, and 78.43% of the three-dimensional vehicle target detection accuracy AP at the three difficulty levels of easy, medium, and difficult, respectively. 3D ; When the R40 method is used to calculate the average detection accuracy, the proposed method achieves 92.94%, 82.70%, and 78.21% of the three-dimensional vehicle target detection accuracy AP3D at the three difficulty levels of simple, medium, and difficult, respectively, proving that the target detection algorithm of the point cloud and image fusion proposed in the present invention can effectively realize three-dimensional vehicle target detection. In order to reduce the time overhead of adding image information to the pixel-level fusion method and the decision-level fusion method, it is verified that the three-dimensional target detector of the point cloud and image fusion has certain advantages over the three-dimensional target detector of the single point cloud.

[0138] (2) Comparison of two image RGB information fusion methods

[0139] Table 4 shows the detection accuracy comparison results of two different image RGB information fusion methods. Method 1 is based on the fusion of image RGB texture information at the target box level. The target point cloud is screened according to the prior two-dimensional target box to generate a point cloud cone. The RGB information is only fused for the target point cloud cone in the scene to ensure that the algorithm pays full attention to the target point cloud during the feature extraction process; Method 2 is based on the scene-based image RGB texture information fusion. The RGB information of the point cloud scene under the forward perspective of the entire image is fused to ensure that the point cloud information has sufficient context structure and prevent the loss of key point cloud information. The experimental results show that the method based on the fusion of image RGB information of the entire scene (Method 2) can provide more point cloud information and scene structure information for the three-dimensional target detector. In the feature extraction process, the algorithm can extract more complete and richer high-dimensional semantic features of the point cloud. Therefore, the average detection accuracy AP of Method 2 is 3D Slightly higher than method 1, with the best performance.

[0140] Table 4 Comparison of average detection accuracy of two image RGB information fusion methods

[0141]

[0142] (3) Comparison table of the method of the present invention and other methods

[0143] Comparison of the average detection accuracy AP of the fusion method proposed in this invention and other 3D target detection methods 3D ,As shown in Table 5, 11 interpolation points (R11) are used for measurement, and the accuracy of other 3D object detection methods are derived from their respective literature.

[0144] Table 5 Comparison of the average detection accuracy of the proposed method and other methods on the KITTI 3D Object validation set

[0145]

[0146] According to Table 5, compared with Part-A2, SECOND-L, IA-SSD and other 3D target detection methods based only on lidar point clouds, the proposed method has achieved better vehicle target detection results on simple, medium and difficult samples. For example, the average detection accuracy AP of 0.88%, 1.46% and 2.01% were achieved on simple samples. 3D Enhancement; and MV 3DCompared with other multimodal feature fusion methods based on view projection, the proposed method has achieved leading vehicle target detection performance; the detection results of the multimodal fusion method proposed in this paper are also better than other multimodal fusion methods such as FPointNet, ContFusion, PointPainting, MANet, UberATG-MMF, etc., with an average detection accuracy AP of 1.13 on medium-difficulty samples. 3D The increases were 9.47%, 7.14%, 2.65%, 2.76% and 2.96% respectively.

[0147] (4) Qualitative analysis of target detection performance

[0148] Use Open3D and OpenCV and other tools to draw the original point cloud, three-dimensional target detection results, original image and image detection results, and the visualization results are shown in Figure 12. In the three-dimensional scene, the green three-dimensional target box represents the three-dimensional real vehicle target box, the red three-dimensional target box represents the vehicle target prediction box of the method of the present invention, and the blue three-dimensional target box represents the vehicle target prediction box of other methods, and the white is the original point cloud; in the two-dimensional image, the green three-dimensional target box represents the real vehicle target box, the red three-dimensional target box represents the vehicle target prediction box of the method of the present invention, and the blue three-dimensional target box represents the vehicle target prediction box of other methods. In the two visualization results, the orange circle represents the missed detection target, and the pink circle represents the misdetected target.

[0149] It can be seen from the visualization results that when other algorithms are used to detect vehicle targets on LiDAR point clouds, the algorithms can better locate vehicle targets with different postures and different truncation levels. However, since this method is only based on point clouds for detection, the single point cloud data collected by the LiDAR sensor does not contain sufficient information, resulting in missed detection and false detection in some traffic scenes, such as Fig.12a As shown, they are marked with orange circles and pink circles respectively. Compared with the detection method based only on point cloud, the present invention incorporates the image information branch, adds a pixel-level fusion method in the data processing stage, and adds a decision-level fusion method before non-maximum suppression, which greatly improves the detection performance. Figure 12b It can be found that compared with the point cloud 3D target detection method without fusion of image information, the algorithm of the present invention compensates for the problem of insufficient semantic information contained in the point cloud data collected by a single lidar sensor to a certain extent after fusing the image information branch. In the process of 3D vehicle target detection feature extraction, the algorithm can extract denser and richer semantic features from the multimodal fusion point cloud, effectively improving the accuracy of 3D vehicle target detection and enhancing the detection performance of the above-mentioned missed and misdetected targets. At the same time, effective detection can be achieved for vehicle targets with different degrees of occlusion, different postures and sparse point clouds.

[0150] The present invention also evaluates the pixel-level fusion method based on image RGB texture features and image target prior results;

[0151] By applying the fusion scheme to different existing 3D target detectors for comparative experiments, the R11 method is used to measure the average detection accuracy, and the results are shown in Table 6. According to the experimental results, the method proposed in this paper has a good effect on improving the detection performance of different 3D target detectors. Comprehensive comparison shows that the fusion scheme has the best effect on the PointRCNN 3D target detection algorithm based on the original point cloud. Compared with the original method PointRCNN, it has improved by 0.94%, 1.03% and 1.12% respectively in the three difficulty samples of simple, medium and difficult. Since the pixel-level fusion method in this paper directly expands the dimension and enriches the information at the level of the original point cloud data, the feature extraction network based on the original point cloud can directly obtain better semantic features and achieve high-precision detection. The above results show that the pixel-level fusion method proposed in this paper can effectively enhance the point cloud information and improve the accuracy of target detection.

[0152] Table 6 Comparison of average detection accuracy of the pixel-level fusion scheme of the present invention on different 3D detectors

[0153]

[0154] The present invention also evaluates the decision-level fusion method based on channel attention and feature distance. The decision-level fusion module mentioned in this paper can also be regarded as an independent module, which realizes the target frame fusion by analyzing the geometric consistency and semantic consistency of the detection results of different sensors. The present invention analyzes the experimental results from two aspects: ablation experiment and performance comparison on different three-dimensional target detectors. The image two-dimensional target detectors used in the experiment are all Cascade-RCNN.

[0155] (1) Ablation experiment

[0156] Based on the CLOCs method, this paper introduces channel attention encoding and feature distance encoding to improve the problems existing in the CLOCs method. To verify the effectiveness of the introduced module, the ablation comparison experiment shown in Table 7 was conducted. The R11 and R40 methods were used for measurement. When the channel attention module and the feature distance module are not added, the fusion module degenerates into the CLOCs method; when the channel attention module is added, the different target pairs between the fusion tensors generated by the two target detection result boxes interact in the convolutional network, generating a more reliable category confidence score. Under the R11 method measurement, AP 3DThe performance of the simple, medium and difficult samples was improved by 0.12%, 0.03% and 0.06% respectively. The feature distance module improves the problem of the single method of quantifying geometric consistency in the CLOCs method and enhances the geometric consistency relationship of the detection results of different sensors. Under the R11 method, AP3D is improved by 0.21%, 0.07% and 0.02% on the simple, medium and difficult samples respectively. When the channel attention and feature distance are introduced into the fusion module at the same time, when the R11 metric is used, AP3D is improved by 0.21%, 0.07% and 0.02% on the simple, medium and difficult samples compared with the original CLOCs method. 3D The improvements were 0.71%, 0.44% and 0.36% respectively. The above results show that the proposed method can effectively improve the problems of CLOCs method.

[0157] Table 7 Channel attention and feature distance coding effectiveness verification experiment table

[0158]

[0159] (2) Comparison of decision-level fusion results on different 3D object detectors

[0160] This experiment adds decision-level fusion to a variety of 3D target detectors for experimental comparison, all of which are measured using R11. The results in Table 8 show that the detection accuracy of different 3D target detectors is improved by adding the decision-level fusion method proposed in this paper. For methods with low baseline accuracy, such as the SECOND and PointPillars algorithms, the decision-level fusion scheme significantly improves the vehicle target detection accuracy. When measured using R11, for the SECOND algorithm, the average detection accuracy AP of the decision-level fusion scheme in this paper is 0.04477 W / m under difficult conditions. 3D The PointPillars algorithm achieved an average detection accuracy of 3.87%, 2.02% and 3.19% AP on three difficulty levels: easy, medium and difficult. 3D Improvement. For methods with higher baseline accuracy, such as the PointRCNN algorithm and the PV-RCNN algorithm, the decision-level fusion scheme has a certain improvement on the detection accuracy. When using the R11 metric, for the PointRCNN algorithm and the PV-RCNN algorithm, the decision-level fusion scheme in this paper achieved an average detection accuracy AP of 1.41% and 1.58% on medium-difficulty samples, respectively. 3D The above results show that the decision-level fusion scheme proposed in this paper has a good effect on improving the detection performance of the 3D target detection method based on a single point cloud.

[0161] Table 8 Comparison of average detection accuracy of the decision-level fusion scheme in this paper on different 3D object detectors

[0162]

[0163] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

Claims

1. A 3D vehicle target detection method integrating image texture features and prior information, characterized in that: include: Obtain a data set, and perform pixel-level point cloud and image fusion based on the point cloud data and image data in the data set; Detect the image data based on a preset image target detector to generate a two-dimensional target frame of the image; Detect the fused point cloud data based on the preset 3D vehicle target detection model to generate a point cloud 3D target frame; A decision-level fusion scheme based on feature distance and channel attention is used to fuse the image 2D target box and the point cloud 3D target box to obtain the final 3D vehicle target detection result.

2. The method according to claim 1, characterized in that The dataset is the KITTI dataset.

3. The method according to claim 2, characterized in that Based on the point cloud data and image data in the data set, pixel-level point cloud and image fusion is performed, specifically: The image target prior information is obtained from the two-dimensional target real box label file of the KITTI dataset. The target category and target box range in the image are obtained based on the image target prior information. After mapping the point cloud to the image, the point cloud located in the image target box is screened out according to the target box range. The category information assignment method based on Gaussian distance encoding is used to determine the probability that the point cloud in the viewing cone belongs to the foreground target point through the Gaussian distance feature information. The probability is assigned to the point cloud data as the category score, and the target category information is assigned to these point clouds. On this basis, the corresponding pixel R, G, B color values ​​are assigned to each point cloud to generate point cloud data with rich feature information.

4. The method according to claim 1, characterized in that The image data is detected based on the preset image target detector to generate the image two-dimensional target frame, specifically: Build an image object detector based on the CascadeRCNN algorithm; Train the image object detector based on a preset training dataset; The image is input into the trained image object detector for detection to generate a two-dimensional object box for the image.

5. The method according to claim 2, characterized in that: The fused point cloud data is detected based on the preset 3D vehicle target detection model to generate a point cloud 3D target frame, specifically: Construct a 3D vehicle target detection model based on the improved PointPillars network model; The three-dimensional vehicle target detection model is trained based on the KITTI data set to obtain a trained three-dimensional vehicle target detection model; The fused point cloud data is input into the trained 3D vehicle target detection model to generate a point cloud 3D target box.

6. The method according to claim 5, characterized in that A three-dimensional vehicle target detection model is constructed based on the improved PointPillars network model, specifically: The PointPillars network model is improved by adding a point cloud local attention mechanism module to the body column feature network of the traditional PointPillars network model to capture the complex features in a specific area of ​​the input point cloud. The SE attention mechanism module is integrated into the two-dimensional pseudo image network of the traditional PointPillars network model to enhance the network's ability to obtain global features and information. A three-dimensional vehicle target detection model is constructed based on the improved PointPillars network model.

7. The method according to claim 1, characterized in that The decision-level fusion scheme based on feature distance and channel attention fuses the image 2D target box and the point cloud 3D target box to obtain the final 3D vehicle target detection result, which is as follows: Encode the image 2D target box and point cloud 3D target box respectively, fuse the category confidence with the target box position, and generate their respective encoding tensors; Based on semantic consistency and geometric consistency, the encoding tensors of the image 2D target box and the point cloud 3D target box are fused to obtain a fused feature tensor. The fused feature tensor is input into the convolutional neural network structure based on channel attention encoding, and finally the fused confidence score is obtained; The fusion confidence scores are redistributed to the point cloud 3D target boxes, and the non-maximum suppression (NMS) algorithm is used to filter redundant and falsely detected target boxes to generate the final 3D vehicle target detection results.

Images