Object detection method and apparatus, computer-readable storage medium and unmanned vehicle

Abstract

The present disclosure relates to an object detection method and device, as well as a computer-readable storage medium and an unmanned vehicle, and relates to the field of artificial intelligence and intelligent driving. The object detection method includes the steps of obtaining an original point cloud of an object to be detected, an image of the object, and a semantic label of pixel points in the image, generating a candidate detection box of the object according to the original point cloud, generating a virtual point cloud according to the candidate detection box, fusing the original point cloud with the virtual point cloud to obtain a fused point cloud, determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system, and generating a 3D detection box of the object according to the fused point cloud and the semantic label corresponding to each point in the fused point cloud by using a 3D detection model. According to the present disclosure, the accuracy of object detection is improved.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from Chinese Application No. 202210339110.5, filed on April 1, 2022, which is hereby incorporated by reference in its entirety into this application.

[0002] The present disclosure relates to the field of artificial intelligence, in particular to the field of autonomous driving, and in particular to an object detection method and apparatus, a computer-readable storage medium, and an unmanned vehicle. [Background technology]

[0003] With the introduction of convolutional neural networks into the field of target detection, 2D target detection has become a research hotspot, and various new methods are constantly emerging. However, in the applications of autonomous driving, robotics and augmented reality, ordinary 2D target detection cannot provide all the information required to recognize the environment, and 2D target detection can only provide the location and corresponding category of the target object in the 2D image.

[0004] However, in the real 3D world, all objects have 3D shapes. For example, in the scenario of autonomous driving, an autonomous vehicle must detect and identify obstacles that may hinder driving, and needs information such as the length, width and height of the target object to take reasonable avoidance actions according to different types and conditions of the obstacles. Therefore, 3D target detection plays an important role in path planning and control.

[0005] Currently, imaging results from monocular cameras, binocular cameras and lidars are mainly used to detect 3D objects in the environment. Summary of the Invention [Means for solving the problem]

[0006] According to a first aspect of the present disclosure, there is provided an object detection method, comprising: obtaining an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image; generating candidate detection boxes for objects to be detected according to the original point cloud; generating a virtual point cloud according to the candidate detection box; fusing the original point cloud and the virtual point cloud to obtain a fused point cloud; determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system; generating a 3D detection box for the object to be detected by using the 3D detection model according to the fused point cloud and a semantic label corresponding to each point in the fused point cloud; Includes.

[0007] In some embodiments, generating the virtual point cloud according to the candidate detection box comprises: generating a grid of candidate detection boxes; generating a virtual point cloud according to a grid of candidate detection boxes; Includes.

[0008] In some embodiments, each point in the virtual point cloud corresponds to one grid of the candidate detection box, and the virtual point cloud has a greater density than the original point cloud.

[0009] In some embodiments, determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system comprises: determining correspondences between each point in the fused point cloud and each pixel in the image by projecting the fused point cloud into an image coordinate system; determining a semantic label corresponding to each point in the fused point cloud according to the correspondence between each point in the fused point cloud and each pixel in the image and the semantic label of each pixel in the image; Includes.

[0010] In some embodiments, the step of fusing the original point cloud and the virtual point cloud to obtain a fused point cloud comprises: A step of superimposing the original point cloud with the virtual point cloud to obtain a fused point cloud. Includes.

[0011] In some embodiments, the step of generating a 3D detection box for an object to be detected by using a 3D detection model according to the fused point cloud and a semantic label corresponding to each point in the fused point cloud comprises: Concatenating the coordinates of each point in the fused point cloud with the semantic label of the point to obtain point cloud and image fusion information; generating a 3D detection box by using the 3D detection model according to the fusion information of the point cloud and the image; Includes.

[0012] In some embodiments, the step of concatenating coordinates of each point in the fused point cloud with a semantic label of the point to obtain point cloud and image fusion information comprises: Serially connecting the coordinates of each point in the fused point cloud and the semantic labels of the points into an array as the fusion information of the point cloud and the image. Includes.

[0013] In some embodiments, the 3D detection model comprises a first feature extraction network and a first detection network, and the step of generating a 3D detection box for an object to be detected by using the 3D detection model according to the fusion information of the point cloud and the image includes: Extracting features of the point cloud and image fusion information by using a first feature extraction network; and generating a 3D detection box for the object to be detected by using a first detection network according to the features of the point cloud and image fusion information. Includes.

[0014] In some embodiments, the semantic labels of pixels in an image are categories of each pixel generated by semantic segmentation of the image.

[0015] In some embodiments, the original point cloud of the object to be detected is obtained by scanning the object to be detected with a lidar, and the image of the object to be detected is obtained by photographing the object to be detected with a camera.

[0016] In some embodiments, the step of generating candidate detection boxes for the object to be detected according to the original point cloud comprises: extracting features of the original point cloud by using a second feature extraction network; generating candidate detection boxes of objects to be detected by using a second detection network according to the features of the original point cloud; Includes.

[0017] According to a second aspect of the present disclosure, there is provided an object detection apparatus, comprising: an acquisition module configured to acquire an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image; a candidate detection box generation module configured to generate a candidate detection box of an object to be detected according to an original point cloud; a virtual point cloud generation module configured to generate a virtual point cloud according to the candidate detection box; a point cloud fusion module configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud; a determination module configured to determine a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system; a 3D detection box generation module configured to generate a 3D detection box for an object to be detected by using a 3D detection model according to the fused point cloud and a semantic label corresponding to each point in the fused point cloud; Equipped with.

[0018] According to a third aspect of the present disclosure, there is provided an object detection apparatus, comprising: Memory, a processor coupled to the memory, the processor configured to implement an object detection method according to any embodiment of the present disclosure based on instructions stored in the memory; Equipped with.

[0019] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having stored thereon computer program instructions which, when executed by a processor, implement an object detection method according to any embodiment of the present disclosure.

[0020] According to a fifth aspect of the present disclosure, there is provided an unmanned vehicle equipped with an object detection device according to any one of the embodiments of the present disclosure.

[0021] The accompanying drawings form a part of the description and illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0022] The present disclosure will become more clearly understood from the following detailed description when taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0023] [Figure 1] 1 illustrates a flowchart of an object detection method according to some embodiments of the present disclosure. [Figure 2A] FIG. 1 shows a schematic diagram of a virtual point cloud generation method according to some embodiments of the present disclosure. [Figure 2B] FIG. 1 shows a schematic diagram of a virtual point cloud generation method according to some embodiments of the present disclosure. [Diagram 3] FIG. 1 illustrates a schematic diagram of determining semantic labels for points in a fused point cloud according to some embodiments of the present disclosure. [Figure 4] 13 illustrates a flowchart for generating a 3D detection box in accordance with some other embodiments of the present disclosure. [Diagram 5] FIG. 1 illustrates a block diagram of an object detection device according to some embodiments of the present disclosure. [Figure 6]1 shows a block diagram of an object detection device according to some other embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a block diagram of a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Various illustrative embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specified, the elements of the steps recited in these embodiments and their arrangement relative to each other, numerical expressions and values ​​do not limit the scope of the present disclosure.

[0025] Additionally, it should be understood that for ease of illustration, dimensions of elements shown in the figures have not been drawn to scale.

[0026] The following description of at least one illustrative embodiment is merely exemplary in nature and is in no way to be taken as limitation of the present disclosure and its application or uses.

[0027] Detailed descriptions may not be provided with respect to techniques, processes and equipment known to those skilled in the art; however, where appropriate, such techniques, processes and equipment should be considered part of the description.

[0028] In any example shown or described herein, any specific values ​​should be construed as merely illustrative rather than limiting, as other examples of illustrative embodiments may have different values.

[0029] It should be noted that like reference numerals in the following figures represent like elements, so that once an element is defined in a figure, no further description thereof is necessary in subsequent figures.

[0030] Among the related art, there are mainly two kinds of algorithms for 3D target detection. The first algorithm is to fuse the original 3D point cloud data with the original 2D image data, so that the fusion data has both RGB and 3D information, and then use a detector to detect the fusion data and output the detection result. But there are some problems with this algorithm. First, since the imaging mode of the point cloud and the imaging mode of the image are carried out from different viewpoints, it is difficult to align the point cloud data and the image data when they are fused. Second, this algorithm needs to use two models to extract features from the data of different sensors and then carry out fusion detection, which increases the complexity of the algorithm. Finally, due to the sparsity of the radar data and the density of the image data, the two fusion results are not ideal and are difficult to be used for effective feature learning, which reduces the accuracy of target detection.

[0031] To solve the above problems, the present disclosure proposes an object detection method and apparatus, a computer-readable storage medium, and an unmanned vehicle. The present disclosure can solve the registration problem between a 3D point cloud and a 2D image by projecting the point cloud into an image coordinate system and determining the correspondence between the points in the point cloud and the semantic label of each point in the 2D image coordinate system.

[0032] In addition, the present disclosure uses a 3D detection model to generate a 3D detection box for the object to be detected according to the fusion point cloud and the semantic label corresponding to each point in the fusion point cloud. On the one hand, only one 3D detection model is used, which reduces the model complexity and facilitates deployment at the vehicle edge. On the other hand, in the final data used for object detection, both the depth information of the 3D point cloud and the semantic label information of the 2D image are retained, which can provide more information for the 3D detection model and improve the accuracy of 3D object detection.

[0033] 1 shows a flowchart of an object detection method according to some embodiments of the present disclosure. In some embodiments, the following image restoration method is performed by an object detection device.

[0034] 1, the object detection method includes steps S1 to S6. Step S1 includes obtaining an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image.

[0035] For example, the original point cloud and the image can each be scanned by different sensors for the same object. The sensors can be lidar, monocular camera, binocular camera, etc.

[0036] In some embodiments, the original point cloud of the object to be detected is obtained by scanning the object to be detected with a lidar, and the image of the object to be detected is obtained by photographing the object to be detected with a camera.

[0037] In some embodiments, the semantic labels of pixels in an image are categories of each pixel generated by semantic segmentation of the image.

[0038] For example, a 2D semantic segmentation model is used to semantically segment a 2D image at pixel level. The input of the model is the color information of the red, green and blue channels (RGB) of the image, and its output is the semantic category of each pixel. Thus, pixels that belong to the same category can be grouped to obtain a category label for each pixel. For example, pixels belonging to humans are classified into one category and pixels belonging to cars are classified into one category. The semantic labels may include, for example, "obstacle", "non-obstacle bicycle", "pedestrian" and "background".

[0039] Step S2 includes generating candidate detection boxes of the objects to be detected according to the original point cloud.

[0040] In some embodiments, generating a candidate detection box of the object to be detected according to the original point cloud includes extracting features of the original point cloud by using a second feature extraction network, and generating a candidate detection box of the object to be detected according to the features of the original point cloud by using the second detection network.

[0041] For example, candidate detection boxes are first generated according to the point cloud. In particular, a voxelization operation is performed on the original point cloud by using methods such as PointPillar or VoxelNet, and then the result of the voxelization operation is sent to a 3D backbone network for feature extraction. Finally, a proposal (3D detection box) is generated by using a detection head according to the extracted features.

[0042] For example, a 3D target detection method based on BEV (bird's eye view) can be used to project the lidar point cloud onto the XY coordinate plane, and a BEV feature map can be obtained after discretization. The BEV map presents the point cloud in the form of an image while preserving the spatial relationship of obstacles in the 3D world. Then, based on this feature map, a candidate detection box is generated by using a detector.

[0043] Besides, a non-maximum suppression (NMS) method can be further used to remove redundant 3D detection boxes and obtain candidate detection boxes.

[0044] The present disclosure uses point cloud data to extract foreground spots of possible candidate boxes. Such a method can reduce detection errors caused by inaccurate calibration parameters between different sensors, since the information of the candidate boxes is fused with image data and then used for 3D object detection.

[0045] Step S3 includes generating a virtual point cloud according to the candidate detection box.

[0046] In some embodiments, generating the virtual point cloud according to the candidate detection boxes includes generating a grid of candidate detection boxes; and generating the virtual point cloud according to the grid of candidate detection boxes.

[0047] For example, the virtual point cloud can be obtained by rasterizing a candidate detection box.

[0048] In some embodiments, each point in the virtual point cloud corresponds to one grid of the candidate detection box, and the virtual point cloud has a greater density than the original point cloud.

[0049] 2A and 2B show schematic diagrams of a virtual point cloud generation method according to some embodiments of the present disclosure.

[0050] After obtaining the candidate detection boxes, each candidate detection box is rasterized and divided into grids with equal sizes, as shown in Fig. 2A, and then the coordinates of each grid are taken as virtual points, and the obtained virtual points are shown as dots in Fig. 2B. Moreover, the density of the virtual point cloud can be made larger than that of the original point cloud by adjusting the size of the grid.

[0051] Since the density of the virtual point cloud is greater than that of the original point cloud, the virtual point cloud is also denser, which solves the problem of the density difference between the image and the point cloud being too large, and improves the accuracy of 3D target detection.

[0052] Step S4 includes fusing the original point cloud and the virtual point cloud to obtain a fused point cloud.

[0053] In some embodiments, fusing the original point cloud and the virtual point cloud to obtain a fused point cloud includes convolving the original point cloud with the virtual point cloud to obtain the fused point cloud.

[0054] For example, since the point cloud is represented by coordinates, the coordinate points in the original point cloud and the coordinate points in the virtual point cloud are merged to obtain a fused point cloud, and the density of the fused point cloud is greater than that of the original point cloud or the virtual point cloud. Moreover, the obtained fused point cloud has no overlapping points. The fused point cloud is "original point cloud + virtual point cloud". Fusing the original point cloud with the virtual point cloud to obtain a fused point cloud so that the fused point cloud has a density greater than that of the original point cloud solves the problem of having too large a density difference between the image and the point cloud. In addition, the virtual point cloud is equipped with the information of the candidate detection box, which is equivalent to providing management information in the subsequent 3D detection process and improving the accuracy of 3D target detection of the object.

[0055] Step S5 involves determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into the image coordinate system.

[0056] In some embodiments, determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system includes determining a correspondence between each point in the fused point cloud and each pixel in the image by projecting the fused point cloud into the image coordinate system, and determining a semantic label corresponding to each point in the fused point cloud according to the correspondence between each point in the fused point cloud and each pixel in the image and the semantic label of each pixel in the image.

[0057] FIG. 3 shows a schematic diagram of determining semantic labels for points in a fused point cloud according to some embodiments of the present disclosure.

[0058] As illustrated in Figure 3, the fused point cloud is first projected into the image coordinate system by using the camera calibration information, and the projection formula is:

[0059]

number

[0060] In the formula, P x , P yand P z represent the coordinates of the points in the fused point cloud in the image coordinate system, K represents the intrinsic reference matrix of the camera, R and T represent the translation and rotation transformation matrices from the point cloud coordinate system to the image coordinate system, respectively. These two transformation matrices are determined according to the calibrated extrinsic parameters of the camera and the point cloud, and z c represents the image depth information, and u and v represent the horizontal and vertical coordinates of a pixel in the image coordinate system after a point in the fused point cloud is projected into the image coordinate system.

[0061] According to the coordinates (u,v) obtained after the points in the fused point cloud are projected into the image coordinate system, the correspondence between the 3D fused point cloud and the 2D image is obtained. By matching the points in the fused point cloud with the pixels in the image, the correspondence between the points in the fused point cloud and the semantic labels C of the pixels in the image can be derived since the semantic labels of the pixels in the image have been obtained in the image semantic segmentation step, so that each point in the fused point cloud has a corresponding semantic label.

[0062] According to the present disclosure, the fused point cloud is projected into an image by using a calibration equation, and the semantic label of each point in the fused point cloud is obtained through the projection of the fused point cloud in the image coordinate system, so that the points in the 3D point cloud are aligned with the semantic segmentation results of the 2D image, and the data alignment problem caused by the method of directly fusing the original point cloud data with the original RGB information of the 2D image is avoided.

[0063] In addition, compared with the method of drawing an image to a point cloud, according to the present disclosure, the point cloud is projected to an image coordinate system, so as to avoid the projection error caused by the lack of image depth information. The purpose of projecting the point cloud to an image in the present disclosure is to obtain the correspondence between the point cloud and the semantic label, and then the semantic label and the point cloud coordinate in 3D space are sent to the 3D detection model instead of the projected coordinate of the point cloud in the image coordinate system. Therefore, the final data used for 3D detection is equipped with the depth information of the point cloud, which improves the accuracy of detection.

[0064] Step S6 includes generating a 3D detection box for the object to be detected by using the 3D detection model according to the fused point cloud and the semantic labels corresponding to each point in the fused point cloud.

[0065] In some embodiments, generating a 3D detection box for an object to be detected by using a 3D detection model according to the fused point cloud and a semantic label corresponding to each point in the fused point cloud includes concatenating coordinates of each point in the fused point cloud with the semantic label of the point to obtain point cloud and image fusion information, and generating a 3D detection box for the object to be detected by using the 3D detection model according to the point cloud and image fusion information.

[0066] In some embodiments, concatenating the coordinates of each point in the fused point cloud with the semantic label of the point to obtain the point cloud and image fusion information includes serially connecting the coordinates of each point in the fused point cloud and the semantic label of the point into an array as the point cloud and image fusion information.

[0067] For example, the fusion information can be obtained by cat- ing the coordinates of the points in the fused point cloud in the spatial coordinate system with the semantic labels obtained in image semantic segmentation, where cat- ing refers to cat- ing along a certain dimension of the existing data, and the total dimension of the data remains unchanged after the operation. For example, the coordinates of each point in the fused point cloud are cat- ed with the semantic label of the point according to the following formula: P i =(P x ,P y ,P z ,C)

[0068] In the formula, P i is the fusion information for each point, (P x ,P y ,P z ) are the coordinates of the point in the spatial coordinate system, and C is the semantic label of the point.

[0069] In some embodiments, the 3D detection model comprises a first feature extraction network and a first detection network, and generating a 3D detection box for the object to be detected by using the 3D detection model according to the point cloud and image fusion information includes extracting features of the point cloud and image fusion information by using the first feature extraction network, and generating a 3D detection box for the object to be detected by using the first detection network according to the features of the point cloud and image fusion information.

[0070] The 3D detection model includes a backbone network for feature extraction and a detection network for detection box generation. The method includes using a common point cloud representation method (such as PointPillar, voxelNet, etc.) to process the fusion information of point clouds and images into a data structure that can be learned by the network, then using a feature extraction network (backbone network) to learn the fusion information of the processed point clouds and images of the two sensors, and finally connecting the detection heads of the detection task to generate a 3D detection box and obtain the final 3D detection result.

[0071] FIG. 4 illustrates a flow chart for generating a 3D detection box according to some other embodiments of the present disclosure.

[0072] As illustrated in FIG. 4, the steps of generating a 3D detection box are as follows: first, perform a voxelization operation on the original point cloud, send the result of the voxelization operation to a 3D backbone network for feature extraction, and generate a proposal (3D detection box) by using a detection head according to the extracted features; remove redundant 3D detection boxes by using the non-maximum suppression (NMS) method to obtain candidate detection boxes; obtain a virtual point cloud by rasterizing the candidate detection boxes, and fuse the original point cloud with the virtual point cloud to obtain a fused point cloud; and perform semantic segmentation of the 2D image. to obtain semantic labels; determine the semantic labels corresponding to each point in the fused point cloud by projecting the fused point cloud into the image coordinate system; obtain the point cloud and image fusion information by concatenating the coordinates of each point in the fused point cloud with the point's semantic label; use a common point cloud representation method to process the point cloud and image fusion information into a data structure that can be learned by the network; use a feature extraction network (backbone network) to learn the processed point cloud and image fusion information of the two sensors; finally, connect the detection heads of the detection task to generate a 3D detection box and obtain the final 3D detection result.

[0073] In this disclosure, the fusion point cloud is combined with the semantic information (semantic label) of the image, a basic coding operation is performed, and a 3D target detector is accessed to obtain the final detection result. In the final detection stage, only one 3D detection model is used to obtain the 3D detection result, which can reduce the model complexity and facilitate the deployment of the model in the vehicle.

[0074] In addition, according to the present disclosure, the final data used for object detection retains both the depth information of the 3D point cloud and the semantic label information of the 2D images, providing more information for the 3D detection model and improving the accuracy of object detection.

[0075] FIG. 5 illustrates a block diagram of an object detection device according to some embodiments of the present disclosure.

[0076] As shown in FIG. 5 , the object detection device 5 includes an acquisition module 51, a candidate detection box generation module 52, a virtual point cloud generation module 53, a point cloud fusion module 54, a decision module 55 and a 3D detection box generation module 56.

[0077] The acquisition module 51 is configured to acquire an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image, so as to perform step S1 as illustrated in FIG.

[0078] The candidate detection box generating module 52 is configured to perform step S2 as illustrated in FIG. 1, for generating a candidate detection box of an object to be detected according to the original point cloud.

[0079] The virtual point cloud generation module 53 is configured for generating a virtual point cloud according to the candidate detection box, so as to perform step S3 as illustrated in FIG.

[0080] The point cloud fusion module 54 is configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud, so as to perform step S4 as illustrated in FIG.

[0081] The determination module 55 is configured to determine a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into the image coordinate system, such as performing step S5 as illustrated in FIG.

[0082] The 3D detection box generation module 56 is configured to generate a 3D detection box for an object to be detected by using the 3D detection model according to the fused point cloud and the semantic labels corresponding to each point in the fused point cloud, to perform step S6 as illustrated in FIG. 1.

[0083] The object detection apparatus according to the present disclosure can solve the registration problem between a 3D point cloud and a 2D image by projecting the point cloud into an image coordinate system and determining correspondences between points in the point cloud and the semantic labels of each point in the 2D image coordinate system.

[0084] In addition, the object detection device according to the present disclosure only uses one 3D detection model, which reduces the complexity of the model and facilitates deployment at the edge of the vehicle. Meanwhile, the final data used for object detection retains both the depth information of the 3D point cloud and the semantic label information of the 2D image, which provides more information for the 3D detection model and improves the accuracy of object detection.

[0085] FIG. 6 illustrates a block diagram of an object detection device according to some other embodiments of the present disclosure.

[0086] 6, the object detection device 6 includes a memory 61 and a processor 62 coupled to the memory 61, the memory 61 being configured to store instructions for implementing an embodiment corresponding to the object detection method. The processor 62 is configured to implement the object detection method in any of the embodiments of the present disclosure based on the instructions stored in the memory 61.

[0087] An embodiment of the present disclosure provides an unmanned vehicle, which is equipped with an object detection device 5 or an object detection device 6. According to the present disclosure, the accuracy of object detection can be improved, so that the unmanned vehicle can avoid obstacles based on the detected objects, thereby improving the driving safety of the unmanned vehicle.

[0088] FIG. 7 illustrates a block diagram of a computer system for implementing some embodiments of the present disclosure.

[0089] 7, computer system 70 may be embodied in the form of a general purpose computing device. Computer system 70 includes a memory 710, a processor 720, and a bus 700 connecting various system components.

[0090] The memory 710 comprises, for example, a system memory, a non-volatile storage medium, etc. The system memory stores, for example, an operating system, an application program, a boot loader program, and other programs. The system memory may consist of a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium stores, for example, instructions for implementing an embodiment corresponding to at least one of the object detection methods. The non-volatile storage medium includes, but is not limited to, disk storage, optical storage, flash memory, etc.

[0091] The processor 720 can be implemented in the form of a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or any other programmable logic device, discrete gates or transistors, or other discrete hardware components. Thus, each module, such as the judgment module and the decision module, can be implemented by a central processing unit (CPU) that runs instructions in a memory to perform the corresponding steps, or by a dedicated circuit that performs the corresponding steps.

[0092] Bus 700 may use any of a variety of bus structures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0093] The computer system 70 may further include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750 as well as the memory 710 and the processor 720 may be connected through a bus 700. The input / output interface 730 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for an external storage device such as a floppy disk, a USB flash drive, or an SD card.

[0094] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks, can be implemented by computer readable program instructions.

[0095] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable device such that the instructions, when executed by the processor, cause a machine to generate a device that implements the functions specified in the flowchart and / or block diagram blocks.

[0096] These computer readable program instructions may be stored in a computer readable memory to cause a computer to function in a particular manner to produce an article of manufacture including instructions implementing the functions specified in the flowchart and / or block diagram blocks.

[0097] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.

[0098] Through the object detection method and device, the computer-readable storage medium, and the unmanned vehicle in the above embodiments, the accuracy of object detection is improved.

[0099] Up to now, the object detection method and device, computer-readable storage medium and unmanned vehicle according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details that are generally known are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed in this specification according to the above description. [Explanation of symbols]

[0100] 5. Object detection device 51 Acquisition Module 52 Candidate detection box generation module 53 Virtual point cloud generation module 54 Point Cloud Fusion Module 55 Decision Module 56 3D Detection Box Generation Module 6. Object detection device 61 Memory 62 processors 70 Computer Systems 700 Bus 710 Memory 720 Processor 730 Input / Output Interface 740 Network Interface 750 Storage Interface

Claims

1. obtaining an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image; generating candidate detection boxes for the object to be detected according to the original point cloud; generating a virtual point cloud according to the candidate detection box; fusing the original point cloud and the virtual point cloud to obtain a fused point cloud; determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system; generating a 3D detection box for the object to be detected by using a 3D detection model according to the fused point cloud and the semantic labels corresponding to each point in the fused point cloud; The object detection method includes:

2. generating a virtual point cloud according to the candidate detection box, generating a grid of candidate detection boxes; generating the virtual point cloud according to the grid of the candidate detection box; The object detection method of claim 1 , comprising:

3. The method of claim 2 , wherein each point in the virtual point cloud corresponds to one grid of the candidate detection box, and the virtual point cloud has a density that is greater than a density of the original point cloud.

4. determining a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system, determining correspondences between each point in the fused point cloud and each pixel in the image by projecting the fused point cloud into the image coordinate system; determining the semantic label corresponding to each point in the fused point cloud according to the correspondence between each point in the fused point cloud and each pixel in the image and the semantic label of each pixel in the image; The object detection method of claim 1 , comprising:

5. The step of fusing the original point cloud and the virtual point cloud to obtain a fused point cloud comprises: convolving the original point cloud with the virtual point cloud to obtain the fused point cloud. The object detection method of claim 1 , comprising:

6. generating a 3D detection box for the object to be detected by using a 3D detection model according to the fused point cloud and the semantic labels corresponding to each point in the fused point cloud; concatenating coordinates of each point in the fused point cloud with the semantic label of the point to obtain fusion information of the point cloud and the image; generating the 3D detection box by using the 3D detection model according to the fusion information of the point cloud and the image; The object detection method of claim 1 , comprising:

7. said step of concatenating coordinates of each point in the fused point cloud with the semantic label of the point to obtain fusion information of the point cloud and the image, serially connecting the coordinates of each point in the fused point cloud and the semantic labels of the points into an array as the fusion information of the point cloud and the image. The object detection method of claim 6, comprising:

8. the 3D detection model comprises a first feature extraction network and a first detection network, and the step of generating a 3D detection box for the object to be detected by using the 3D detection model according to the fusion information of the point cloud and the image, extracting features of the fusion information of the point cloud and image by using the first feature extraction network; generating the 3D detection box for the object to be detected by using the first detection network according to the features of the fusion information of the point cloud and images; The object detection method of claim 6, comprising:

9. The method of claim 1 , wherein the semantic labels of pixels in the image are categories of each pixel generated by semantic segmentation of the image.

10. 2. The object detection method of claim 1, wherein the original point cloud of the object to be detected is obtained by scanning the object to be detected with a lidar, and the image of the object to be detected is obtained by photographing the object to be detected with a camera.

11. generating a candidate detection box for the object to be detected according to the original point cloud, extracting features of the original point cloud by using a second feature extraction network; generating the candidate detection boxes of the objects to be detected by using a second detection network according to the features of the original point cloud; The object detection method of claim 1 , comprising:

12. an acquisition module configured to acquire an original point cloud of an object to be detected, an image of the object to be detected, and semantic labels of pixels in the image; a candidate detection box generation module configured to generate a candidate detection box of the object to be detected according to the original point cloud; a virtual point cloud generation module configured to generate a virtual point cloud according to the candidate detection box; a point cloud fusion module configured to fuse the original point cloud with the virtual point cloud to obtain a fused point cloud; a determination module configured to determine a semantic label corresponding to each point in the fused point cloud by projecting the fused point cloud into an image coordinate system; a 3D detection box generation module configured to generate a 3D detection box for the object to be detected by using a 3D detection model according to the fused point cloud and the semantic labels corresponding to each point in the fused point cloud; An object detection device comprising:

13. Memory, A processor coupled to the memory, the processor configured to implement the object detection method according to any one of claims 1 to 11 based on instructions stored in the memory; An object detection device comprising:

14. A computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implements the object detection method of any one of claims 1 to 11.

15. An unmanned vehicle equipped with the object detection device according to claim 12 or 13.

16. 16. The unmanned vehicle of claim 15, further comprising at least one of a lidar or a camera, wherein the lidar is configured to scan an object to be detected to obtain an original point cloud of the object to be detected, and the camera is configured to photograph the object to be detected to obtain an image of the object to be detected.

17. A computer program comprising instructions which, when executed by a processor, cause the processor to implement a method for object detection according to any one of claims 1 to 11.

Images