Object detection methods and devices, computer-readable storage media and unmanned vehicles

Abstract

This disclosure relates to object detection methods and apparatus, computer-readable storage media, and unmanned vehicles, relating to the fields of artificial intelligence and intelligent driving. The object detection method includes: acquiring an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels for pixels in the image; generating candidate detection boxes for the object to be detected based on the original point cloud; generating a virtual point cloud based on the candidate detection boxes; fusing the original point cloud and the virtual point cloud to obtain a fused point cloud; determining the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto an image coordinate system; and generating a 3D detection box for the object to be detected using a 3D detection model based on the fused point cloud and the semantic labels corresponding to each point in the fused point cloud. According to this disclosure, the accuracy of object detection is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to the field of autonomous driving, and especially to object detection methods and apparatus, computer-readable storage media and unmanned vehicles. Background Technology

[0002] With the introduction of convolutional neural networks into the field of object detection, 2D object detection has become a research hotspot, with various new methods constantly emerging. However, in applications such as autonomous driving, robotics, and augmented reality, ordinary 2D object detection cannot provide all the information needed to perceive the environment; it can only provide the position and corresponding category of the target object in a 2D image.

[0003] However, in the real three-dimensional world, objects have three-dimensional shapes. For example, in autonomous driving scenarios, autonomous vehicles must detect and identify obstacles that may hinder their movement, requiring information such as the length, width, and height of the target object in order to make appropriate avoidance actions based on different obstacle types and states. Therefore, three-dimensional object detection plays a crucial role in path planning and control.

[0004] Currently, the main method for detecting 3D objects in the environment is to use imaging results from devices such as monocular cameras, binocular cameras, and LiDAR. Summary of the Invention

[0005] According to a first aspect of this disclosure, an object detection method is provided, comprising:

[0006] Obtain the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image;

[0007] Based on the original point cloud, generate candidate detection boxes for the objects to be detected;

[0008] Based on the candidate detection boxes, generate a virtual point cloud;

[0009] The original point cloud is fused with the virtual point cloud to obtain the fused point cloud;

[0010] By projecting the fused point cloud onto the image coordinate system, the semantic label corresponding to each point in the fused point cloud is determined;

[0011] Using a 3D detection model, a 3D detection bounding box for the object to be detected is generated based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud.

[0012] In some embodiments, generating a virtual point cloud based on candidate detection boxes includes:

[0013] Generate a grid for candidate detection boxes;

[0014] A virtual point cloud is generated based on the grid of the candidate detection boxes.

[0015] In some embodiments, each point in the virtual point cloud corresponds to a grid in the candidate detection box, and the density of the virtual point cloud is greater than that of the original point cloud.

[0016] In some embodiments, determining the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto an image coordinate system includes:

[0017] By projecting the fused point cloud onto the image coordinate system, the correspondence between each point in the fused point cloud and each pixel in the image is determined;

[0018] Based on the correspondence between each point in the fused point cloud and each pixel in the image, and the semantic label of each pixel in the image, the semantic label corresponding to each point in the fused point cloud is determined.

[0019] In some embodiments, fusing the original point cloud with the virtual point cloud to obtain a fused point cloud includes:

[0020] The original point cloud is superimposed with the virtual point cloud to obtain a fused point cloud.

[0021] In some embodiments, generating a 3D detection bounding box for the object to be detected using a 3D detection model, based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud, includes:

[0022] The coordinates of each point in the fused point cloud are concatenated with the semantic label of that point to obtain the fusion information of the point cloud and the image;

[0023] Using a 3D detection model, a 3D detection bounding box for the object to be detected is generated based on the fusion information of point cloud and image.

[0024] In some embodiments, concatenating the coordinates of each point in the fused point cloud with the semantic label of that point to obtain the fusion information of the point cloud and the image includes: concatenating the coordinates of each point in the fused point cloud with the semantic label of that point into an array, which serves as the fusion information of the point cloud and the image.

[0025] In some embodiments, the 3D detection model includes a first feature extraction network and a detection network. The step of generating a 3D detection bounding box for the object to be detected based on the fusion information of point cloud and image using the 3D detection model includes:

[0026] The first feature extraction network is used to extract features from the fusion information of point cloud and image;

[0027] Using the first detection network, a 3D detection bounding box for the object to be detected is generated based on the features of the fusion information of point cloud and image.

[0028] In some embodiments, the semantic labels of pixels in the image are the categories of each pixel generated by semantic segmentation of the image.

[0029] In some embodiments, the original point cloud of the object to be detected is obtained by scanning the object with a lidar, and the image of the object to be detected is obtained by capturing the object with a camera.

[0030] In some embodiments, generating candidate detection boxes for the object to be detected based on the original point cloud includes:

[0031] The second feature extraction network is used to extract features from the original point cloud.

[0032] Using a second detection network, candidate detection boxes for the objects to be detected are generated based on the features of the original point cloud.

[0033] According to a second aspect of this disclosure, an object detection apparatus is provided, comprising:

[0034] The acquisition module is configured to acquire the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image;

[0035] The candidate detection box generation module is configured to generate candidate detection boxes for the object to be detected based on the original point cloud.

[0036] The virtual point cloud generation module is configured to generate a virtual point cloud based on the candidate detection boxes;

[0037] The point cloud fusion module is configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud;

[0038] The determination module is configured to determine the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto the image coordinate system;

[0039] The 3D detection box generation module is configured to use a 3D detection model to generate a 3D detection box for the object to be detected based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud.

[0040] According to a third aspect of this disclosure, an object detection apparatus is provided, comprising:

[0041] Memory; and

[0042] A processor coupled to the memory is configured to execute an object detection method as described in any embodiment of the present disclosure based on instructions stored in the memory.

[0043] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the object detection method as described in any embodiment of this disclosure.

[0044] According to a fifth aspect of this disclosure, an unmanned vehicle is provided, equipped with an object detection device as described in any embodiment of this disclosure. Attached Figure Description

[0045] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0046] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0047] Figure 1 A flowchart of an object detection method according to some embodiments of the present disclosure is shown;

[0048] Figure 2A and 2B A schematic diagram of a virtual point cloud generation method according to some embodiments of the present disclosure is shown;

[0049] Figure 3 A schematic diagram illustrating the determination of semantic labels for points in a fused point cloud according to some embodiments of the present disclosure is shown;

[0050] Figure 4 A flowchart illustrating the generation of a 3D detection box according to other embodiments of this disclosure is shown;

[0051] Figure 5 A block diagram of an object detection apparatus according to some embodiments of the present disclosure is shown;

[0052] Figure 6 A block diagram of an object detection apparatus according to other embodiments of the present disclosure is shown;

[0053] Figure 7 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown. Detailed Implementation

[0054] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0055] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0056] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0057] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0058] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0060] In related technologies, there are two main types of algorithms for 3D object detection. The first algorithm fuses the original 3D point cloud data and the original 2D image data together, resulting in fused data containing both RGB and 3D information. A detector then processes this fused data to output the detection result. However, this algorithm has several problems. First, because point cloud imaging and image imaging are performed from different perspectives, it's difficult to align the point cloud and image data during fusion. Second, this algorithm requires two separate models to extract features from data from different sensors before fusion detection, increasing the algorithm's complexity. Finally, the sparsity of radar data and the density of image data lead to unsatisfactory fusion results, making it difficult to use for effective feature learning and reducing the accuracy of object detection.

[0061] To address the aforementioned problems, this disclosure proposes an object detection method and apparatus, a computer-readable storage medium, and an unmanned vehicle. This disclosure solves the alignment problem between 3D point clouds and 2D images by projecting a point cloud onto an image coordinate system and determining the correspondence between points in the point cloud and the semantic labels of each point in the 2D image coordinate system.

[0062] Furthermore, this disclosure utilizes a 3D detection model to generate a 3D detection bounding box for the object to be detected based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud. On the one hand, it uses only a single 3D detection model, reducing model complexity and facilitating deployment on vehicles. On the other hand, the final data used for object detection retains both the depth information of the 3D point cloud and the semantic label information of the 2D image, providing more information for the 3D detection model and improving the accuracy of 3D object detection.

[0063] Figure 1A flowchart illustrating an object detection method according to some embodiments of the present disclosure is shown. In some embodiments, the following image inpainting methods are performed by an object detection apparatus.

[0064] like Figure 1 As shown, the object detection method includes steps S1-S6. In step S1, the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image are obtained.

[0065] For example, the raw point cloud and the image can be obtained by scanning the same object using different sensors. These sensors can be LiDAR, monocular cameras, and binocular cameras, etc.

[0066] In some embodiments, the original point cloud of the object to be detected is obtained by scanning the object with a lidar, and the image of the object to be detected is obtained by capturing the object with a camera.

[0067] In some embodiments, the semantic labels of pixels in an image are the categories of each pixel generated by semantic segmentation of the image.

[0068] For example, a two-dimensional semantic segmentation model can be used to perform semantic segmentation on a two-dimensional image at the pixel level. The model's input is the color information (RGB) of the image's three channels (red, green, and blue), and the output is the semantic category of each pixel. This allows pixels belonging to the same category to be grouped together, obtaining a label for each pixel's category. For instance, pixels belonging to people can be grouped into one category, and pixels belonging to cars into another. Semantic labels could be, for example, "obstacle," "non-obstacle bicycle," "pedestrian," and "background," etc.

[0069] In step S2, candidate detection boxes for the objects to be detected are generated based on the original point cloud.

[0070] In some embodiments, generating candidate detection boxes for an object to be detected based on the original point cloud includes: extracting features from the original point cloud using a second feature extraction network; and generating candidate detection boxes for the object to be detected based on the features of the original point cloud using a second detection network.

[0071] For example, candidate detection boxes are first generated from the point cloud. Specifically, methods such as PointPillar and VoxelNet are used to voxelize the original point cloud, and then the results of the voxelization are fed into a 3D backbone network for feature extraction. Finally, a detection head is used to generate proposals (3D detection boxes) based on the extracted features.

[0072] For example, a 3D target detection method based on bird's-eye view (BEV) can be used to project the LiDAR point cloud onto the XY coordinate plane and obtain a BEV feature map after discretization. The BEV image presents the point cloud in image form while preserving the spatial relationships of obstacles in the 3D world. Then, based on this feature map, a detector is used to generate candidate detection boxes.

[0073] In addition, the non-maximum suppression (NMS) method can be used to remove redundant 3D detection boxes and obtain candidate detection boxes.

[0074] This disclosure uses point cloud data to extract foreground points of potential candidate boxes. By using this method, after the information of the candidate boxes is fused with the image data, it can be used for 3D object detection, which can reduce the detection error caused by inaccurate calibration parameters between different sensors.

[0075] In step S3, a virtual point cloud is generated based on the candidate detection boxes.

[0076] In some embodiments, generating a virtual point cloud based on candidate detection boxes includes: generating a grid of candidate detection boxes; and generating a virtual point cloud based on the grid of candidate detection boxes.

[0077] For example, a virtual point cloud can be obtained by rasterizing the candidate detection boxes.

[0078] In some embodiments, each point in the virtual point cloud corresponds to a grid in the candidate detection box, and the density of the virtual point cloud is greater than that of the original point cloud.

[0079] Figure 2A and 2B A schematic diagram illustrating a virtual point cloud generation method according to some embodiments of the present disclosure is shown.

[0080] After obtaining the candidate detection boxes, each candidate detection box is rasterized, such as... Figure 2A As shown, the candidate detection box is divided into grids of equal size, and the coordinates of each grid are used as a virtual point. The resulting virtual points are as follows: Figure 2B The dots in the graph are shown. Furthermore, the density of the virtual point cloud can be increased compared to the density of the original point cloud by adjusting the size of the grid.

[0081] Because the density of the virtual point cloud is greater than that of the original point cloud, the virtual point cloud also becomes denser, which solves the problem of excessive density difference between the image and the point cloud and improves the accuracy of 3D target detection.

[0082] In step S4, the original point cloud and the virtual point cloud are fused to obtain a fused point cloud.

[0083] In some embodiments, fusing the original point cloud with the virtual point cloud to obtain a fused point cloud includes: superimposing the original point cloud with the virtual point cloud to obtain a fused point cloud.

[0084] For example, since point clouds are represented by coordinates, merging the coordinate points in the original point cloud and the virtual point cloud results in a fused point cloud, which has a higher density than either the original or virtual point cloud. Furthermore, there are no overlapping points in the generated fused point cloud. The fused point cloud is essentially "original point cloud + virtual point cloud". By fusing the original and virtual point clouds, the resulting fused point cloud has a higher density than the original point cloud, solving the problem of a large density difference between the image and the point cloud. In addition, the virtual point cloud includes information about candidate detection boxes, which essentially provides supervision information in subsequent 3D detection processes, improving the accuracy of 3D object detection.

[0085] In step S5, the semantic label corresponding to each point in the fused point cloud is determined by projecting the fused point cloud onto the image coordinate system.

[0086] In some embodiments, determining the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto an image coordinate system includes: determining the correspondence between each point in the fused point cloud and each pixel in the image by projecting the fused point cloud onto the image coordinate system; and determining the semantic label corresponding to each point in the fused point cloud based on the correspondence between each point in the fused point cloud and each pixel in the image, and the semantic label of each pixel in the image.

[0087] Figure 3 A schematic diagram illustrating the determination of semantic labels for points in a fused point cloud according to some embodiments of the present disclosure is shown.

[0088] like Figure 3 As shown, the fused point cloud is first projected onto the image coordinate system using the camera's calibration information. The projection formula is as follows:

[0089]

[0090] Among them, P x P y P z Z represents the coordinates of points in the fused point cloud in the image coordinate system, K represents the camera's intrinsic parameter matrix, and R and T represent the translation and rotation transformation matrices from the point cloud coordinate system to the image coordinate system, respectively. These two transformation matrices are determined based on the calibration extrinsic parameters of the camera and point cloud. c The image depth information is represented by u and v, which represent the horizontal and vertical coordinates of the pixels in the image coordinate system after the points in the fused point cloud are projected onto the image coordinate system.

[0091] Based on the coordinates (u, v) obtained by projecting points from the fused point cloud onto the image coordinate system, the correspondence between the 3D fused point cloud and the 2D image is obtained. By associating the points in the fused point cloud with the pixels in the image, and considering the semantic labels of the pixels in the image already obtained in the image semantic segmentation step, the correspondence between the points in the fused point cloud and the semantic labels C of the pixels in the image can be derived, ensuring that each point in the fused point cloud has a corresponding semantic label.

[0092] This disclosure utilizes calibration equations to project the fused point cloud onto an image. By projecting the fused point cloud into the image coordinate system, the semantic label of each point in the fused point cloud is obtained, thereby aligning the points in the 3D point cloud with the semantic segmentation results of the 2D image. This avoids the data alignment problem caused by directly fusing the original point cloud data and the original RGB information of the 2D image.

[0093] Furthermore, compared to methods that simply plot images onto point clouds, this invention projects the point cloud into an image coordinate system, avoiding projection errors caused by missing image depth information. The purpose of projecting the point cloud onto the image in this disclosure is to obtain the correspondence between the point cloud and semantic labels. The semantic labels and the point cloud coordinates in 3D space are then fed into the 3D detection model, rather than the projected coordinates of the point cloud in the image coordinate system. Therefore, the final data used for 3D detection includes the depth information of the point cloud, improving detection accuracy.

[0094] In step S6, a 3D detection bounding box of the object to be detected is generated using the 3D detection model based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud.

[0095] In some embodiments, a 3D detection model is used to generate a 3D detection bounding box of the object to be detected based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud. This includes: concatenating the coordinates of each point in the fused point cloud with the semantic label of that point to obtain the fusion information of the point cloud and the image; and using the 3D detection model to generate a 3D detection bounding box of the object to be detected based on the fusion information of the point cloud and the image.

[0096] In some embodiments, the coordinates of each point in the fused point cloud are concatenated with the semantic label of that point to obtain the fusion information of the point cloud and the image, including: concatenating the coordinates of each point in the fused point cloud with the semantic label of that point into an array as the fusion information of the point cloud and the image.

[0097] For example, fusion information can be obtained by concatenating (cat) the coordinates of points in the fused point cloud in the spatial coordinate system with the semantic labels obtained from image semantic segmentation. Here, concatenation refers to concatenating along a certain dimension of the existing data; the total dimensions of the data remain unchanged after the operation. For example, the coordinates of each point in the fused point cloud are concatenated with its semantic label according to the following formula:

[0098] P i =(P x ,P y ,P z C)

[0099] Among them, P i Fusion information for each point, (P x ,P y ,P z ) represents the coordinates of the point in the spatial coordinate system, and C is the semantic label of the point.

[0100] In some embodiments, the three-dimensional detection model includes a first feature extraction network and a detection network. The three-dimensional detection model generates a three-dimensional detection box of the object to be detected based on the fusion information of the point cloud and the image, including: extracting features from the fusion information of the point cloud and the image using the first feature extraction network; and generating a three-dimensional detection box of the object to be detected based on the features from the fusion information of the point cloud and the image using the first detection network.

[0101] The 3D detection model consists of a backbone network for feature extraction and a detection network for generating bounding boxes. Using common point cloud representation methods (such as PointPillar and VoxelNet), the fused information of the point cloud and image is processed into a data structure that the network can learn. Then, a feature extraction network (backbone network) learns the fused information of the point cloud and image from the two sensors. Finally, a detection head is connected to generate 3D bounding boxes, resulting in the final 3D detection result.

[0102] Figure 4 A flowchart illustrating the generation of a 3D detection frame according to other embodiments of this disclosure is shown.

[0103] like Figure 4As shown, the steps for generating 3D detection boxes are as follows: First, the original point cloud is voxelized. The result of the voxelization operation is fed into a 3D backbone network for feature extraction. Using a detection head, proposals (3D detection boxes) are generated based on the extracted features. Non-maximum suppression is used to remove redundant 3D detection boxes, resulting in candidate detection boxes. The candidate detection boxes are rasterized to obtain a virtual point cloud. The original point cloud and the virtual point cloud are then fused to obtain a fused point cloud. Semantic segmentation is performed on the 2D image to obtain semantic labels. The fused point cloud is projected onto the image coordinate system to determine the semantic label corresponding to each point in the fused point cloud. The coordinates of each point in the fused point cloud are concatenated with its semantic label to obtain the fusion information of the point cloud and the image. Common point cloud representation methods are used to process the fusion information of the point cloud and the image into a data structure that the network can learn. A feature extraction network (backbone network) is used to learn the fusion information of the point clouds and images from the two sensors. Finally, a detection head for a detection task is connected. The head is used to generate a 3D detection bounding box, and the final 3D detection result is obtained.

[0104] This disclosure combines the semantic information (semantic labels) of point clouds and images, performs basic encoding operations, and feeds the data into a 3D object detector to obtain the final detection result. In the final detection stage, only one 3D detection model is used to obtain the 3D detection result. Using only one model reduces model complexity and facilitates model deployment on the vehicle.

[0105] In addition, the data used for object detection in this disclosure retains both the depth information of the 3D point cloud and the semantic label information of the 2D image, providing more information for the 3D detection model and improving the accuracy of object detection.

[0106] Figure 5 A block diagram of an object detection apparatus according to some embodiments of the present disclosure is shown.

[0107] like Figure 5 As shown, the object detection device 5 includes an acquisition module 51, a candidate detection box generation module 52, a virtual point cloud generation module 53, a point cloud fusion module 54, a determination module 55, and a three-dimensional detection box generation module 56.

[0108] The acquisition module 51 is configured to acquire the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image, for example, by performing... Figure 1 Step S1 is shown.

[0109] Candidate detection box generation module 52 is configured to generate candidate detection boxes for the object to be detected based on the original point cloud, for example, by performing the following... Figure 1Step S2 is shown.

[0110] The virtual point cloud generation module 53 is configured to generate a virtual point cloud based on candidate detection boxes, for example, by performing the following... Figure 1 Step S3 is shown.

[0111] Point cloud fusion module 54 is configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud, for example, by performing the following... Figure 1 Step S4 is shown.

[0112] Module 55 is configured to determine the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto the image coordinate system, for example, by performing... Figure 1 Step S5 is shown.

[0113] The 3D detection bounding box generation module 56 is configured to use the 3D detection model to generate a 3D detection bounding box for the object to be detected based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud. For example, it performs the following... Figure 1 Step S6 is shown.

[0114] According to the object detection device disclosed herein, by projecting a point cloud onto an image coordinate system, the correspondence between points in the point cloud and the semantic labels of each point in the two-dimensional image coordinate system is determined, thereby solving the alignment problem between the three-dimensional point cloud and the two-dimensional image.

[0115] Furthermore, the object detection device disclosed herein utilizes only a single 3D detection model, reducing model complexity and facilitating deployment on vehicles. On the other hand, the final data used for object detection retains both the depth information of the 3D point cloud and the semantic label information of the 2D image, providing more information for the 3D detection model and improving the accuracy of object detection.

[0116] Figure 6 A block diagram of an object detection apparatus according to other embodiments of the present disclosure is shown.

[0117] like Figure 6 As shown, the object detection device 6 includes a memory 61 and a processor 62 coupled to the memory 61. The memory 61 stores instructions for executing embodiments of the object detection method. The processor 62 is configured to execute the object detection method in any of the embodiments of this disclosure based on the instructions stored in the memory 61.

[0118] This disclosure provides an unmanned vehicle equipped with an object detection device 5 or an object detection device 6. According to this disclosure, the accuracy of object detection can be improved, thereby enabling the unmanned vehicle to avoid obstacles based on detected objects, thus improving the safety of unmanned vehicle operation.

[0119] Figure 7A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0120] like Figure 7 As shown, the computer system 70 can be represented in the form of a general computing device. The computer system 70 includes a memory 710, a processor 720, and a bus 700 connecting different system components.

[0121] The memory 710 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for performing at least one embodiment of the object detection method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0122] The processor 720 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by the central processing unit (CPU) running instructions in memory to execute the corresponding steps, or by dedicated circuitry to execute the corresponding steps.

[0123] Bus 700 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0124] The computer system 70 may also include an input / output interface 730, a network interface 740, and a storage interface 750. These interfaces 730, 740, and 750, as well as the memory 710 and processor 720, can be connected via a bus 700. The input / output interface 730 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0125] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0127] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to operate in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0128] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0129] The object detection method and apparatus, computer-readable storage medium, and unmanned vehicle described in the above embodiments improve the accuracy of object detection.

[0130] This concludes the detailed description of the object detection method and apparatus, computer-readable storage medium, and unmanned vehicle according to the present disclosure. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. An object detection method, comprising: Obtain the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image; Based on the original point cloud, generate candidate detection boxes for the objects to be detected; Based on the candidate detection boxes, a virtual point cloud is generated, including: generating a grid of candidate detection boxes, and generating a virtual point cloud based on the grid of candidate detection boxes, wherein each point in the virtual point cloud corresponds to a grid in the grid of the candidate detection boxes. The original point cloud is fused with the virtual point cloud to obtain the fused point cloud; By projecting the fused point cloud onto the image coordinate system, the semantic label corresponding to each point in the fused point cloud is determined; Using a 3D detection model, a 3D detection bounding box for the object to be detected is generated based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud.

2. The object detection method according to claim 1, wherein, The density of the virtual point cloud is greater than that of the original point cloud.

3. The object detection method according to claim 1, wherein, The step of determining the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto the image coordinate system includes: By projecting the fused point cloud onto the image coordinate system, the correspondence between each point in the fused point cloud and each pixel in the image is determined; Based on the correspondence between each point in the fused point cloud and each pixel in the image, and the semantic label of each pixel in the image, the semantic label corresponding to each point in the fused point cloud is determined.

4. The object detection method according to claim 1, wherein, The process of fusing the original point cloud with the virtual point cloud to obtain a fused point cloud includes: The original point cloud is superimposed with the virtual point cloud to obtain a fused point cloud.

5. The object detection method according to claim 1, wherein, The step of generating a 3D detection bounding box for the object to be detected using a 3D detection model, based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud, includes: The coordinates of each point in the fused point cloud are concatenated with the semantic label of that point to obtain the fusion information of the point cloud and the image; Using a 3D detection model, a 3D detection bounding box for the object to be detected is generated based on the fusion information of point cloud and image.

6. The object detection method according to claim 5, wherein, The step of concatenating the coordinates of each point in the fused point cloud with its semantic label to obtain the fusion information of the point cloud and the image includes: The coordinates of each point in the fused point cloud are concatenated with the semantic label of that point into an array, which serves as the fusion information between the point cloud and the image.

7. The object detection method according to claim 5, wherein, The 3D detection model includes a first feature extraction network and a detection network. The step of generating a 3D detection bounding box for the object to be detected based on the fusion information of point clouds and images using the 3D detection model includes: The first feature extraction network is used to extract features from the fusion information of point cloud and image; Using the first detection network, a 3D detection bounding box for the object to be detected is generated based on the features of the fusion information of point cloud and image.

8. The object detection method according to claim 1, wherein, The semantic labels of the pixels in the image are the categories of each pixel generated by semantic segmentation of the image.

9. The object detection method according to claim 1, wherein, The original point cloud of the object to be detected is obtained by scanning the object with a lidar, and the image of the object to be detected is obtained by capturing the object with a camera.

10. The object detection method according to claim 1, wherein, The step of generating candidate detection boxes for the object to be detected based on the original point cloud includes: The second feature extraction network is used to extract features from the original point cloud. Using a second detection network, candidate detection boxes for the objects to be detected are generated based on the features of the original point cloud.

11. An object detection device, comprising: The acquisition module is configured to acquire the original point cloud of the object to be detected, the image of the object to be detected, and the semantic labels of the pixels in the image; The candidate detection box generation module is configured to generate candidate detection boxes for the object to be detected based on the original point cloud. The virtual point cloud generation module is configured to generate a virtual point cloud based on candidate detection boxes, including: generating a grid of candidate detection boxes, and generating a virtual point cloud based on the grid of candidate detection boxes, wherein each point in the virtual point cloud corresponds to a grid in the grid of the candidate detection box. The point cloud fusion module is configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud; The determination module is configured to determine the semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud onto the image coordinate system; The 3D detection box generation module is configured to use a 3D detection model to generate a 3D detection box for the object to be detected based on the fused point cloud and the semantic label corresponding to each point in the fused point cloud.

12. An object detection device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the object detection method as described in any one of claims 1 to 10 based on instructions stored in the memory.

13. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the object detection method as described in any one of claims 1 to 10.

14. An unmanned vehicle equipped with the object detection device as described in claim 11 or 12.

Images