A three-dimensional target detection method and related apparatus

Abstract

A three-dimensional target detection method and related device, comprising: obtaining a training data set; expanding the pre-processed image data and 3D point cloud data; dividing the entire three-dimensional space into equal three-dimensional voxel grids; constructing a three-dimensional target detection network for the features of the voxel grid and the image data; building a missing finding module and a missing prediction module; using the trained model for 3D target detection. The present application realizes the missing target detection in three-dimensional target detection, can find out the undetected targets in three-dimensional target detection according to the advanced accuracy of two-dimensional target detection and make two-stage detection, improve the accuracy of three-dimensional target detection, and be more comprehensive and accurate when detecting long-distance targets. The special training of virtual point cloud of missing targets overcomes the problem of failing to capture part of the three-dimensional features of the target, and can detect the occluded or truncated target.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a three-dimensional target detection method and related apparatus. Background Technology

[0002] 3D object detection is the foundation for autonomous driving and robotics to perform environmental perception and behavioral decision-making. It can sense the information of the surrounding scene: relevant information of the surrounding environment (roads, vehicles, pedestrians, etc.) (position, direction, size, etc.); the main sources of perception information are cameras and LiDAR (Light Detection and Ranging).

[0003] Currently, there are two main approaches to 3D object detection: point-based and voxel-based methods. Voxel-based methods overcome the technical problems of sampling feature loss and low computational efficiency inherent in point-based methods. However, some objects that are clearly visible in RGB images may have almost no point cloud representation in their labels due to distance or occlusion. In such cases, both point-based and voxel-based methods struggle to predict object locations because it is nearly impossible to extract any features from the point cloud. Considering the inability to extract effective 3D features from the point cloud when detecting occluded or truncated 3D objects, these objects are often missed or predicted with excessively high errors by 3D object detection networks. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional target detection method and related apparatus to solve the problem of detection omissions caused by the inability to effectively extract the 3D features of occluded or truncated 3D objects.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a three-dimensional target detection method, comprising:

[0007] Obtain the training dataset and preprocess it.

[0008] The preprocessed image data and 3D point cloud data are augmented;

[0009] The entire three-dimensional space is divided into equal parts of a three-dimensional voxel grid. The 3D point cloud is placed in the voxel grid according to its coordinate position. The feature of each voxel grid is encoded as the average value of the point cloud features within that grid.

[0010] A 3D target detection network was constructed based on the features of voxel meshes and image data.

[0011] Build a module for finding omissions and a module for predicting omissions;

[0012] After training the omission detection module and obtaining the trained parameters, the point cloud and images of the 3D object detection dataset are used as inputs to the various channels of the neural network and fed into the entire network model for training.

[0013] The trained model is used for 3D object detection. The trained object detection model is used to predict the objects on each sample on the validation set and the test set one by one until all samples have been predicted, thus completing the 3D object detection test.

[0014] Optionally, obtain the training dataset and preprocess it:

[0015] Download the KITTI public dataset's 2D and 3D object detection datasets. The 2D object detection dataset includes a training set of 200 samples. The 3D object detection dataset includes a left RGB image, point cloud, camera parameters, and labels, divided into training and test sets. Preprocess the datasets by removing excess width from the point cloud scene, limiting the entire scene range to a fixed value, randomly flipping along the X-axis, randomly rotating, and randomly scaling the scene.

[0016] Optionally, the preprocessed image data and 3D point cloud data can be augmented:

[0017] Methods for augmenting RGB images include: horizontal flipping, angle transformation, brightness / contrast / color transformation, image blurring and sharpening, adding Gaussian noise, and random curling;

[0018] Methods for expanding 3D point clouds include: random world flipping, random world rotation, random world scaling, and random image sliding.

[0019] Optionally, construct a 3D object detection network:

[0020] The structure of the 3D object detection network consists of a batch normalization layer, an activation layer, a 4-layer 3D sparse convolutional neural network, a feature compression layer, and a region proposal network (RPN). The normalization layer and activation layer are used to normalize and activate voxel features, the 3D sparse convolutional layer is used to extract 3D features, and the RPN network is used to achieve object detection.

[0021] Optionally, build a missing item search module and a missing item prediction module:

[0022] The omission detection module consists of an adaptive average pooling layer, a convolutional layer, a max pooling layer, four residual convolutional groups, an activation layer, an average pooling layer, a feature pyramid network (FPN), and an omission detection algorithm. The adaptive average pooling layer is used to adjust the image size, the convolutional layer, the max pooling layer, and the residual convolutional groups are used to extract 2D features from the RGB image, the activation layer is used to add non-linear factors, the feature pyramid network is used to predict the target from the 2D features, and the omission detection algorithm is used to find objects missed by the 3D target detection network.

[0023] The omission prediction module has two prediction networks:

[0024] a. This is the subtle prediction module, used to extract subtle features for prediction. Its structure includes, in sequence, a point cloud projection module, 3 upsampling layers, 2 interpolation layers, a feature compression layer, 6 2D convolutional layers and 3 2D deconvolutional layers, a feature stitching layer, and 2 2D convolutional layers. The projection module is used to filter out the point cloud of missed objects. The upsampling and interpolation layers are used to add features to the sparse point cloud. The feature compression layer is used to compress 3D features into 2D features. The 6 2D convolutional layers are used to extract 2D features. The 3 2D deconvolutional layers are used to align features. The feature stitching layer is used to stitch features together. The 2 2D convolutional layers are used to achieve object detection.

[0025] b. Virtual point cloud prediction module, whose structure includes, in sequence, a virtual point cloud generation algorithm, an orientation prediction algorithm, and a sliding window-based target detection network; the orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results, the virtual point cloud generation algorithm is used to generate cone-shaped point clouds of missed objects, and the sliding window-based target detection network is used to realize target detection.

[0026] Optionally, the orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results. Its main implementation method is to use the coordinates of the center point of the missing object in the 2D image and the aspect ratio of the detection box for prediction. The 3D scene range of the virtual point cloud generation algorithm is x∈[0,70],y∈[-40,40],z∈[-3,3]. First, 140×160×6=134400 virtual points are generated. All virtual points are further filtered by projection to generate a cone-shaped point cloud of the missing objects. The object detection network based on the sliding window is used to implement object detection. The length, width and height of the sliding window are [3.9,1.6,1.56]. The orientation of the sliding window is determined by the orientation prediction algorithm.

[0027] Optional, train the network model:

[0028] First, the FPN network in the omission detection module needs to be trained by inputting the KITTI 2D object detection dataset into the network. After obtaining the trained parameters, the point cloud and RGB image of the KITTI 3D object detection dataset are used as inputs to the neural network channels and fed into the entire network model for training. The trained network model is then validated using average accuracy to ensure it meets the requirements. The omission prediction module includes a network for detail detection and a network for generating virtual point cloud detection. In detail detection, as the network progresses, the point cloud of the object is continuously upsampled and interpolated to enrich its 3D features. In the virtual point cloud detection network, a fixed-orientation sliding window slides with a fixed step size to accurately locate the target of interest.

[0029] Secondly, the present invention provides a three-dimensional target detection system, comprising:

[0030] The data acquisition module is used to acquire the training dataset and preprocess the acquired training dataset.

[0031] The data augmentation module is used to augment the preprocessed image data and 3D point cloud data;

[0032] The voxel mesh creation module is used to divide the entire three-dimensional space into equal parts of the three-dimensional voxel mesh, and place the 3D point cloud in the voxel mesh according to the coordinate position. The feature of each voxel mesh is encoded as the average value of the point cloud features within that mesh.

[0033] A 3D target detection network construction module is used to construct a 3D target detection network based on the features of voxel meshes and image data.

[0034] The training module is used to build the omission detection module and the omission prediction module. After training the omission detection module and obtaining the trained parameters, the point cloud and images of the 3D object detection dataset are used as the inputs of each channel of the neural network and input into the entire network model for training.

[0035] The detection module is used to perform 3D object detection using the trained model. It predicts the objects on each sample one by one on the validation set and the test set until all samples have been predicted, thus completing the 3D object detection test.

[0036] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a three-dimensional target detection method.

[0037] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a three-dimensional target detection method.

[0038] Compared with the prior art, the present invention has the following technical effects:

[0039] This invention first downloads the KITTI 2D and 3D object detection datasets and preprocesses them. An improved 3D object detection network is obtained by adding a missing object detection module and a missing object prediction module to the 3D object detection network. The processed dataset is then used as input to pre-train the network on training samples, obtaining the initial parameters of the 3D object detection network model. Finally, the trained 3D object detection network model is used to detect objects on test samples, yielding the predicted results. This invention achieves missing object detection in 3D object detection, finding undetected targets in 3D object detection with the advanced accuracy of 2D object detection and performing a two-stage detection, thus improving the accuracy of 3D object detection. It is more comprehensive and accurate in detecting distant targets. Specialized training using virtual point clouds of missing targets overcomes the problem of failing to capture some 3D features of targets, enabling the detection of occluded or truncated targets.

[0040] Experimental results show that the new model has advanced performance efficiency and improves the accuracy of 3D object detection tasks. Compared with other methods, the method of this invention can find some missed objects in 3D object detection, fully extract their features, and make predictions.

[0041] The method of this invention significantly improves the prediction accuracy of medium and hard KITTI samples and can be widely applied in autonomous driving and robotics. Attached Figure Description

[0042] Figure 1 This is an overall flowchart of Embodiment 1 of the present invention.

[0043] Figure 2 This is a schematic diagram of the overall structure of Embodiment 1 of the present invention.

[0044] Figure 3 This is a schematic diagram of the three-dimensional target detection network structure (3) of Embodiment 1 of the present invention.

[0045] Figure 4 This is a schematic diagram of the structure of the omission search module (4) in Embodiment 1 of the present invention.

[0046] Figure 5 This is a schematic diagram of the omission prediction module (5) of Embodiment 1 of the present invention.

[0047] Figure 6This is a schematic diagram of the sample difficulty level in Embodiment 1 of the present invention.

[0048] Figure 7 This is a schematic diagram of upsampling in the fine prediction module of the omission prediction module (5) of Embodiment 1 of the present invention.

[0049] Figure 8 This is a schematic diagram of the projection and sliding window of the virtual point cloud prediction module of the omission prediction module (5) in Embodiment 1 of the present invention.

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. Clear embodiments of this disclosure have been shown in the foregoing drawings, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept of this disclosure in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art by reference to specific embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0052] This invention comprises the following steps: acquiring a training dataset; preprocessing the acquired point cloud and images; voxelizing or sparsely sampling the point cloud; extracting 3D features from the point cloud; compressing the 3D features; outputting the first 3D object detection result; training a 2D object detection model; detecting objects in the 2D image; finding missed targets; extracting the point cloud of the missed targets; performing specialized target detection training on the point cloud of the missed targets; generating a virtual point cloud of the missed targets; training the virtual point cloud for target detection; and outputting the final 3D object detection result. This invention achieves the detection of missed targets in 3D object detection, enabling it to find targets not detected in 3D object detection based on the advanced accuracy of 2D object detection and perform a second-stage detection, thus improving the accuracy of 3D object detection. It is more comprehensive and accurate when detecting distant targets. The use of specialized training with virtual point clouds of missed targets overcomes the problem of not being able to capture some target 3D features, and can detect occluded or truncated targets.

[0053] An embodiment of the three-dimensional target detection method of the present invention is as follows: Figure 1 As shown, the overall framework of the model is as follows: Figure 2 As shown, the steps are as follows:

[0054] (1) Data preprocessing

[0055] Download the 2D and 3D object detection datasets from the public dataset KITTI. The 2D object detection dataset includes a training set of 200 samples. The 3D object detection dataset includes a left RGB image (image_2), point cloud (velodyne), camera parameters (calib), and labels (label_2), divided into a training set (7481 samples) and a test set (7518 samples). Preprocessing operations are performed on the datasets, including: removing redundant widths from the point cloud scene, limiting the entire scene range to a fixed value, random flipping along the X-axis, random rotation, and random scene scaling. Overfitting is prone to occur when training large neural networks with small sample training data. To address this issue, we augment the RGB images for 2D object detection using methods such as horizontal flipping, angle transformation, brightness / contrast / color transformation, image blurring and sharpening, adding Gaussian noise, and random curling. For 3D object detection, we augment the point cloud using methods such as random world flipping, random world rotation, random world scaling, and random sliding.

[0056] (2) Encoding voxel features:

[0057] First, the entire 3D space is divided into equal parts of a 3D voxel grid. Then, the point cloud is placed in the voxel grid according to its coordinate position. The feature of each voxel grid is encoded as the average value of the point cloud features within that grid, i.e.:

[0058] f i point =(x,y,z,r) (1)

[0059]

[0060] Where f i point , Let p represent the features of the i-th point and voxel, respectively, where (x, y, z, r) are the coordinates and intensity value of that point. i ~p n ∈v j This indicates that only features of points within that voxel are extracted.

[0061] (3) Construct a 3D target detection network:

[0062] The structure of a 3D target detection network is as follows: Figure 3 As shown, the encoded voxel features are first processed by batch normalization and ReLU activation functions, and then 3D features are extracted through 4 layers of 3D sparse convolution. The extracted features are then sent to the feature compression layer for dimensionality reduction, that is, the Z-axis of the features is compressed into the channel. Finally, the 3D target is predicted by the RPN network.

[0063] (4) Build an omission detection module:

[0064] The structure of the omission lookup module is as follows: Figure 4 As shown, its structure includes, in sequence, one convolutional layer, a max pooling layer, four residual convolutional groups, an activation layer, an average pooling layer, a Feature Pyramid Network (FPN), and an omission detection algorithm. The RGB image is first input into the adaptive average pooling layer, outputting a 3×224×224 feature map. Then, 2D features are extracted through a convolutional layer with a 7×7 kernel and a stride of 2, a 3×3 max pooling layer with a stride of 2, and four residual convolutional groups with 3×3 kernels and a stride of 2. After adding a non-linear factor through the activation layer, the image is input into the Feature Pyramid Network (FPN) to predict targets from the 2D features. The results of the FPN prediction and the 3D object detection network prediction are input into the omission detection algorithm. They are first sorted by a ranking algorithm to extract key features, that is, sorting the 2D and 3D object detections according to their distance. Then, a feature comparison algorithm compares the similarity between the prediction results of each 2D and 3D object to determine the missing objects.

[0065] (5) Build an omission prediction module:

[0066] The structure of the omission prediction module is as follows: Figure 5 As shown, it contains two prediction modules:

[0067] a. This is the subtle prediction module, used to extract subtle features for prediction. First, the point cloud is projected onto the 2D bounding box of the missing object to obtain the point cloud of the missing object. The sparse point cloud is augmented with features by 3 upsampling layers and 2 interpolation layers. The upsampling number for each point is 3, and the interpolation value for each point in the interpolation layer is 2. Then, the features are reduced in dimensionality by a feature compression layer, which multiplies the feature's channel C by its height H. The dimensionality-reduced features are then fed into a 3×3 2D convolutional layer and a 2D deconvolutional layer to extract features. The deconvolutional layer is used to align the features before inputting them into a feature concatenation layer. The concatenated features are then passed through two 1×1 2D convolutional layers for classification and regression, respectively.

[0068] b. The virtual point cloud prediction module first generates 140×160×6=134400 virtual points in the three-dimensional scene range x∈[0,70],∈[-40,40],∈[-3,3]. All virtual points are further filtered by projection to generate a cone-shaped point cloud of the missing object. Then, the features of the missing object are fed into the orientation prediction algorithm. The orientation is predicted by using the center point coordinates of the missing object in the 2D image and the aspect ratio of the detection box. A sliding window with a size of [3.9,1.6,1.56] and the orientation is the predicted value is generated. The three-dimensional object is predicted in the target detection network based on the sliding window. Finally, the prediction result of the missing object is selected by voting and combined with the prediction result of the three-dimensional target detection network in (3) as the final prediction result.

[0069] (6) Training the network model

[0070] First, the FPN network in the omission detection module needs to be trained. The training set of the KITTI 2D object detection dataset is divided into training and validation sets in a 7:3 ratio and input into the network for training. After obtaining the trained parameters, the training set of the KITTI 3D object detection dataset is divided into a training set of 3712 and a validation set of 3769 for training. The samples in the KITTI 3D object detection dataset can be divided into three difficulty levels: easy, medium, and hard. Figure 6 As shown, (a) is a point cloud example of a simple-level sample, (b) is a point cloud example of a medium-level sample, and (c) is a point cloud example of a difficult-level sample.

[0071] Focal Loss is used to address the imbalance between easy and difficult samples during training.

[0072]

[0073] For simple samples, Pt will be relatively large in the above formula, so the weights will naturally decrease. For difficult samples, Pt is relatively small, so the weights are relatively large, making the network more inclined to use such samples for parameter updates.

[0074] Figure 7 The point cloud after upsampling and interpolation is shown. In the omission prediction module (5), the sliding window size is set to [3.9, 1.6, 1.56], corresponding to the length, width, and height of the car, and the orientation of the sliding window is provided by the orientation prediction algorithm. The virtual point cloud and sliding window are shown below. Figure 8 As shown.

[0075] During training, the amount of data fed into the gap-filling model each time is 2, the number of model iterations is 80, and the learning rate is 0.01. The gap-filling model continuously optimizes the weight parameters of each module in the model by backpropagating the loss function L, thus completing the training of the model.

[0076] The specific process of model setting in this embodiment of the invention is as follows:

[0077] 6.1) Initialization. The initialization function selects the Adam optimizer to alleviate the gradient vanishing or exploding problem during backpropagation, which helps the neural network converge better and faster.

[0078] 6.2) Activation Function. In the extension path, a Corrected Linear Unit (ReLU) is used in each convolutional layer to perform a non-linear transformation on the data after the convolutional layer, solving the gradient vanishing problem in the network. Define the input as x, then ReLU is defined as:

[0079] f(X)=max(0,x) (4)

[0080] 6.3) Pooling Layer. The model uses a max-pooling function to select the maximum value of feature points in the neighborhood. This reduces the number of parameters the network needs to learn, prevents overfitting, and expands the receptive field. The model uses bilinear interpolation for upsampling, which works by selecting one of two linear directions for interpolation, helping to increase the feature density of the target. The interpolation layer uses nearest-neighbor interpolation, which increases the density of the point cloud relatively quickly.

[0081] 6.4) Loss Function. The loss function for object detection includes classification loss and regression loss, and the model's loss function L is determined by the following formula:

[0082]

[0083] Where N fg Indicates the number of foreground anchors. and These represent the outputs of classification and regression, respectively. and These are the classification labels and the regression labels, respectively. This represents the regression loss calculated using only the foreground points.

[0084] (7) Use the trained model for 3D object detection

[0085] The trained object detection model is used to predict objects one by one on each sample on the validation and test sets until all samples have been predicted, thus completing the 3D object detection test. To evaluate the performance of the proposed method, the most commonly used standard performance metric for 3D object detection is used: average precision (AP), with a threshold of 0.7 (IoU). That is, a prediction is considered correct if the overlap between the predicted value and the label is greater than 0.7. The higher the AP, the more accurate the prediction.

[0086] To verify the beneficial effects of the present invention, the inventors conducted comparative experiments on the validation set of the KITTI dataset using the method of the embodiments of the present invention, compared with SECOND, PointRCNN and Voxel-RCNN (hereinafter referred to as the comparative experiments).

[0087] In the comparative experiments, the evaluation metric was average precision (AP), with a threshold of 0.7 (IoU), and 11 items were recalled per iteration. For each category, the detector was evaluated according to three difficulty levels: easy, medium, and hard. The difficulty evaluation criteria were based on the height, occlusion, and truncation of objects in the 2D results.

[0088] Table 1. Experimental results of the method in Example 1 and the comparative experiment.

[0089]

[0090]

[0091] Experimental results show that the present invention performs well in the detection of medium and difficult samples, which indicates that the present invention can effectively detect missing objects in 3D target detection and improve the overall accuracy of 3D target detection.

[0092] In another embodiment of the present invention, a three-dimensional target detection system is provided, which can be used to implement the above-described three-dimensional target detection method. Specifically, the system includes:

[0093] The data acquisition module is used to acquire the training dataset and preprocess the acquired training dataset.

[0094] The data augmentation module is used to augment the preprocessed image data and 3D point cloud data;

[0095] The voxel mesh creation module is used to divide the entire three-dimensional space into equal parts of the three-dimensional voxel mesh, and place the 3D point cloud in the voxel mesh according to the coordinate position. The feature of each voxel mesh is encoded as the average value of the point cloud features within that mesh.

[0096] A 3D target detection network construction module is used to construct a 3D target detection network based on the features of voxel meshes and image data.

[0097] The training module is used to build the omission detection module and the omission prediction module. After training the omission detection module and obtaining the trained parameters, the point cloud and images of the 3D object detection dataset are used as the inputs of each channel of the neural network and input into the entire network model for training.

[0098] The detection module is used to perform 3D object detection using the trained model. It predicts the objects on each sample one by one on the validation set and the test set until all samples have been predicted, thus completing the 3D object detection test.

[0099] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0100] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a three-dimensional target detection method.

[0101] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the three-dimensional target detection method in the above embodiments.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional target detection method, characterized in that, include: Obtain the training dataset and preprocess it. The preprocessed image data and 3D point cloud data are augmented; The entire three-dimensional space is divided into equal parts of a three-dimensional voxel grid. The 3D point cloud is placed in the voxel grid according to its coordinate position. The feature of each voxel grid is encoded as the average value of the point cloud features within that grid. A 3D target detection network was constructed based on the features of voxel meshes and image data. Build a module for finding omissions and a module for predicting omissions; After training the omission detection module and obtaining the trained parameters, the point cloud and images of the 3D object detection dataset are used as inputs to the neural network channels in the 3D object detection network model for training. 3D object detection is performed using a trained 3D object detection network model. The trained 3D object detection network model predicts the objects on each sample one by one on the validation set and the test set until all samples have been predicted, thus completing the 3D object detection test. Build the omission detection module and the omission prediction module: The structure of the omission lookup module includes, in sequence, an adaptive average pooling layer, a convolutional layer, a max pooling layer, four residual convolutional groups, an activation layer, an average pooling layer, a feature pyramid network (FPN), and an omission lookup algorithm. Adaptive average pooling layers are used to adjust the image size, convolutional layers, max pooling layers and residual convolutional groups are used to extract 2D features from RGB images, activation layers are used to add non-linear factors, feature pyramid networks are used to predict targets from 2D features, and omission lookup algorithms are used to find objects missed by the 3D target detection network. The omission prediction module has two prediction networks: a. This is the subtle prediction module, used to extract subtle features for prediction. Its structure includes, in sequence, a point cloud projection module, 3 upsampling layers, 2 interpolation layers, a feature compression layer, 6 2D convolutional layers and 3 2D deconvolutional layers, a feature stitching layer, and 2 2D convolutional layers. The projection module is used to filter out the point cloud of missed objects. The upsampling and interpolation layers are used to add features to the sparse point cloud. The feature compression layer is used to compress 3D features into 2D features. The 6 2D convolutional layers are used to extract 2D features. The 3 2D deconvolutional layers are used to align features. The feature stitching layer is used to stitch features together. The 2 2D convolutional layers are used to achieve object detection. b. Virtual point cloud prediction module, whose structure includes, in sequence, a virtual point cloud generation algorithm, an orientation prediction algorithm, and a sliding window-based target detection network; the orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results, the virtual point cloud generation algorithm is used to generate cone-shaped point clouds of missed objects, and the sliding window-based target detection network is used to realize target detection. The orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results. Its main implementation method is to use the coordinates of the center point of the missing object in the 2D image and the aspect ratio of the detection box for prediction. The 3D scene range of the virtual point cloud generation algorithm is x∈[0,70],y∈[-40,40],z∈[-3,3]. First, 140×160×6=134400 virtual points are generated. All virtual points are further filtered by projection to generate a cone-shaped point cloud of the missing objects. The object detection network based on the sliding window is used to implement object detection. The length, width and height of the sliding window are [3.9, 1.6, 1.56]. The orientation of the sliding window is determined by the orientation prediction algorithm.

2. The three-dimensional target detection method according to claim 1, characterized in that, Obtain the training dataset and preprocess it: Download the KITTI public dataset's 2D and 3D object detection datasets. The 2D object detection dataset includes a training set of 200 samples. The 3D object detection dataset includes a left RGB image, point cloud, camera parameters, and labels, divided into training and test sets. Preprocess the datasets by removing excess width from the point cloud scene, limiting the entire scene range to a fixed value, randomly flipping along the X-axis, randomly rotating, and randomly scaling the scene.

3. The three-dimensional target detection method according to claim 1, characterized in that, The preprocessed image data and 3D point cloud data are augmented: Methods for augmenting RGB images include: horizontal flipping, angle transformation, brightness / contrast / color transformation, image blurring and sharpening, adding Gaussian noise, and random curling; Methods for expanding 3D point clouds include: random world flipping, random world rotation, random world scaling, and random image sliding.

4. The three-dimensional target detection method according to claim 1, characterized in that, Constructing a 3D object detection network: The structure of the 3D object detection network consists of a batch normalization layer, an activation layer, a 4-layer 3D sparse convolutional neural network, a feature compression layer, and a region proposal network (RPN). The normalization layer and activation layer are used to normalize and activate voxel features, the 3D sparse convolutional layer is used to extract 3D features, and the RPN network is used to achieve object detection.

5. The three-dimensional target detection method according to claim 1, characterized in that, Training the network model: First, the FPN network in the omission detection module needs to be trained by inputting the KITTI 2D object detection dataset into the network. After obtaining the trained parameters, the point cloud and RGB image of the KITTI 3D object detection dataset are used as inputs to the neural network channels and fed into the entire network model for training. The trained network model is then validated using average accuracy to ensure it meets the requirements. The omission prediction module includes a network for detail detection and a network for generating virtual point cloud detection. In detail detection, as the network progresses, the point cloud of the object is continuously upsampled and interpolated to enrich its 3D features. In the virtual point cloud detection network, a fixed-orientation sliding window slides with a fixed step size to accurately locate the target of interest.

6. A three-dimensional target detection system, characterized in that, include: The data acquisition module is used to acquire the training dataset and preprocess the acquired training dataset. The data augmentation module is used to augment the preprocessed image data and 3D point cloud data; The voxel mesh creation module is used to divide the entire three-dimensional space into equal parts of the three-dimensional voxel mesh, and place the 3D point cloud in the voxel mesh according to the coordinate position. The feature of each voxel mesh is encoded as the average value of the point cloud features within that mesh. A 3D target detection network construction module is used to construct a 3D target detection network based on the features of voxel meshes and image data. The training module is used to build the omission detection module and the omission prediction module; After training the omission detection module and obtaining the trained parameters, the point cloud and images of the 3D object detection dataset are used as inputs to the neural network channels in the 3D object detection network model for training. The detection module is used to perform 3D object detection using the trained 3D object detection network model. It predicts the objects on each sample one by one on the validation set and the test set until all samples have been predicted, thus completing the 3D object detection test. Build the omission detection module and the omission prediction module: The structure of the omission lookup module includes, in sequence, an adaptive average pooling layer, a convolutional layer, a max pooling layer, four residual convolutional groups, an activation layer, an average pooling layer, a feature pyramid network (FPN), and an omission lookup algorithm. Adaptive average pooling layers are used to adjust the image size, convolutional layers, max pooling layers and residual convolutional groups are used to extract 2D features from RGB images, activation layers are used to add non-linear factors, feature pyramid networks are used to predict targets from 2D features, and omission lookup algorithms are used to find objects missed by the 3D target detection network. The omission prediction module has two prediction networks: a. This is the subtle prediction module, used to extract subtle features for prediction. Its structure includes, in sequence, a point cloud projection module, 3 upsampling layers, 2 interpolation layers, a feature compression layer, 6 2D convolutional layers and 3 2D deconvolutional layers, a feature stitching layer, and 2 2D convolutional layers. The projection module is used to filter out the point cloud of missed objects. The upsampling and interpolation layers are used to add features to the sparse point cloud. The feature compression layer is used to compress 3D features into 2D features. The 6 2D convolutional layers are used to extract 2D features. The 3 2D deconvolutional layers are used to align features. The feature stitching layer is used to stitch features together. The 2 2D convolutional layers are used to achieve object detection. b. Virtual point cloud prediction module, whose structure includes, in sequence, a virtual point cloud generation algorithm, an orientation prediction algorithm, and a sliding window-based target detection network; the orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results, the virtual point cloud generation algorithm is used to generate cone-shaped point clouds of missed objects, and the sliding window-based target detection network is used to realize target detection. The orientation prediction algorithm is used to predict the orientation of 3D objects from 2D detection results. Its main implementation method is to use the coordinates of the center point of the missing object in the 2D image and the aspect ratio of the detection box for prediction. The 3D scene range of the virtual point cloud generation algorithm is x∈[0,70],y∈[-40,40],z∈[-3,3]. First, 140×160×6=134400 virtual points are generated. All virtual points are further filtered by projection to generate a cone-shaped point cloud of the missing objects. The object detection network based on the sliding window is used to implement object detection. The length, width and height of the sliding window are [3.9, 1.6, 1.56]. The orientation of the sliding window is determined by the orientation prediction algorithm.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the three-dimensional target detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional target detection method as described in any one of claims 1 to 5.

Images