Method and apparatus for three-dimensional object detection

Abstract

According to embodiments of the present disclosure, a method, apparatus, device, storage medium and program product for three-dimensional object detection are provided. The method proposed herein includes processing point cloud data by using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution; determining a set of fusion features based on the set of point cloud features and a set of image features associated with the first resolution, the set of image features being determined by an image feature extraction model based on image data aligned with the point cloud data; and determining a three-dimensional bounding box associated with a candidate object and classification information based on the set of fusion features, the classification information indicating a confidence that the candidate object belongs to a plurality of predetermined categories. In this way, more accurate three-dimensional object detection can be achieved through the fusion of point cloud features and image features.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of computer technology, and more specifically, to methods, apparatus, motion devices, computer storage media, and computer program products for three-dimensional object detection. Background Technology

[0002] With the development of computer vision technology, object detection technology has been widely applied in many fields, such as intelligent transportation and security. In the field of intelligent transportation, accurately perceiving objects in the surrounding environment is a fundamental yet challenging task. People hope to obtain more accurate object detection results. Summary of the Invention

[0003] According to some embodiments of this disclosure, a scheme for three-dimensional object detection is provided.

[0004] In a first aspect of this disclosure, a method for three-dimensional object detection is provided. The method includes: processing point cloud data using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution; determining a set of fused features based on the set of point cloud features and a set of image features associated with the first resolution, the set of image features being determined by an image feature extraction model based on image data aligned with the point cloud data; and determining, based on the set of fused features, three-dimensional detection boxes and classification information associated with candidate objects, the classification information indicating the confidence level of the candidate objects belonging to multiple predetermined categories.

[0005] In a second aspect of this disclosure, an apparatus for three-dimensional object detection is provided. The apparatus includes: an extraction module configured to process point cloud data using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution; a fusion module configured to determine a set of fused features based on the set of point cloud features and a set of image features associated with the first resolution, the set of image features being determined by an image feature extraction model based on image data aligned with the point cloud data; and a detection module configured to determine, based on the set of fused features, a three-dimensional detection bounding box and classification information associated with a candidate object, the classification information indicating the confidence level of the candidate object belonging to multiple predetermined categories.

[0006] In a third aspect of this disclosure, an electronic device is provided, including one or more processors and a memory, wherein the memory is used to store computer-executable instructions that are executed by the one or more processors to implement the method according to a first aspect of this disclosure.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] In a fifth aspect of this disclosure, a computer program product is provided, comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0009] According to embodiments of this disclosure, more accurate 3D object detection can be achieved by fusing point cloud features and image features. Furthermore, by implementing point-by-point feature fusion, embodiments of this disclosure can also support the detection and recognition of multiple object categories.

[0010] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A block diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0013] Figure 2 A flowchart of a process for three-dimensional object detection according to some embodiments of the present disclosure is shown;

[0014] Figure 3 An example block diagram of a three-dimensional object detection model according to some embodiments of the present disclosure is shown;

[0015] Figure 4 A schematic diagram illustrating feature fusion according to some embodiments of the present disclosure is shown;

[0016] Figure 5 A flowchart illustrating an example process of model training according to some embodiments of this disclosure is shown;

[0017] Figure 6 A block diagram of an apparatus for three-dimensional object detection according to some embodiments of the present disclosure is shown; and

[0018] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0019] Some exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0020] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] As discussed above, there is a desire to improve the accuracy of object detection. In the field of intelligent transportation, some traditional solutions have begun to utilize data from different types of sensors to perform fusion perception in order to improve object accuracy. However, traditional fusion solutions can only handle the recognition of single-type objects and struggle to support the detection of multi-class 3D objects.

[0022] In view of this, embodiments of the present disclosure propose a scheme for 3D object detection. In this scheme, point cloud data is processed using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution. Further, based on the set of point cloud features and a set of image features associated with the first resolution, a set of fused features is determined, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data. Further, based on the set of fused features, 3D detection boxes and classification information associated with candidate objects are determined, wherein the classification information indicates the confidence level of a candidate object belonging to multiple predetermined categories.

[0023] According to this scheme, embodiments of this disclosure can add corresponding semantic information to point cloud features through point-by-point feature fusion under resolution alignment, thereby achieving more accurate 3D object detection. Furthermore, embodiments of this disclosure can also support multi-category 3D object recognition.

[0024] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0025] Example Environment

[0026] Figure 1 A block diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. (See diagram for example.) Figure 1As shown, environment 100 includes detection equipment 130. For example... Figure 1 As shown, the detection device 130 can acquire image data 110 and corresponding point cloud data 120 related to the target scene to detect at least one object in the target scene. In some embodiments, the image data 110 and point cloud data 120 can be aligned, for example, to improve the accuracy of object detection.

[0027] Unlike traditional two-dimensional inspection, the inspection device 130 can input a three-dimensional inspection box 140 related to the object, thereby determining the position of the object in the real scene.

[0028] In some embodiments, the detection device 130 may be a computing device deployed on an intelligent vehicle, for example, which can acquire image data 110 obtained by the image sensor of the intelligent vehicle and point cloud data 120 obtained by a LiDAR. Accordingly, the detection device 130 can determine the location and category of at least one object around the intelligent vehicle for use, for example, in assisting the driving behavior of an intelligent driving tool.

[0029] In some embodiments, the detection device 130 may be an edge computing device, which may acquire image data 110 and point cloud data 120 captured by image sensors and LiDAR mounted on the vehicle via wired or wireless communication. Alternatively, the edge computing device may also acquire image data and point cloud data 120 acquired by sensors deployed on the roadside to perform 3D object detection. Furthermore, information such as the location and category of the detected object may be transmitted to the intelligent vehicle via appropriate wired or wireless communication (e.g., based on vehicle-to-everything (V2X) technology) to assist the intelligent vehicle's operation.

[0030] In some embodiments, the detection device 130 may also be a server device, which may acquire image data 110 and point cloud data 120 captured by appropriate sensors to perform detection of three-dimensional objects.

[0031] It should be understood that although an example implementation of the detection device 130 has been described above, it can also be implemented as any suitable computing device to perform 3D object detection based on the acquired image data 110 and point cloud data 120. Furthermore, the 3D object detection information can also be used for any suitable purpose, including but not limited to: intelligent transportation, security, pedestrian detection, etc.

[0032] The following will be referenced Figures 2 to 4 This section describes the process of 3D object detection in detail.

[0033] Example process

[0034] The following will combine Figure 2 The process for three-dimensional object detection according to embodiments of the present disclosure will be described in detail below. Figure 2 A schematic diagram of a process 200 for three-dimensional object detection according to some embodiments of the present disclosure is shown. For ease of discussion, refer to... Figure 1 Let's discuss the process used for 3D object detection. Process 200, for example, can be... Figure 1 The detection is performed at the illustrated detection device 130. It should be understood that process 200 may also include boxes not shown and / or the boxes shown may be omitted. The scope of this disclosure is not limited in this respect.

[0035] like Figure 2 As shown in box 202, the detection device 130 processes the point cloud data 120 using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution.

[0036] The following will be referenced Figure 3 This section introduces an example architecture for a 3D object detection model. Figure 3 An example block diagram 300 of a three-dimensional object detection model according to some embodiments of the present disclosure is shown.

[0037] like Figure 3 As shown, the detection device 130 can use the point cloud feature extraction model 310 to acquire point cloud features corresponding to different resolutions. In some embodiments, the point cloud feature extraction model 310 may include, for example, multiple set abstraction (SA) layers, each of which can output point cloud feature representations corresponding to different resolutions.

[0038] by Figure 3 As an example, the point cloud feature extraction model 310 may include four SA layers, such as SA1, SA2, SA3, and SA4. The four SA layers can take as input point cloud feature representations corresponding to different resolutions, for example, {S... l}(l∈[1,2,3,4]), where l represents the layer number.

[0039] Continue to refer to Figure 2 In box 204, the detection device 130 determines a set of fused features based on a set of point cloud features and a set of image features associated with a first resolution. The set of image features is determined by an image feature extraction model based on image data 110 aligned with point cloud data 120.

[0040] like Figure 3As shown, the detection device 130 can also utilize the image feature extraction model 320 to extract image feature representations corresponding to different resolutions. In some embodiments, the image feature extraction model 320 may be based on a VGG (Visual Geometry Group) network, for example. Figure 3 As shown, the image feature extraction model 320 may include, for example, five feature extraction layers, referred to as BLOCK 1 to BLOCK 5 in sequence.

[0041] Due to the characteristics of SA and VGG, the point cloud features output by SA1 have the same resolution as the image features output by BLOCK 2, the point cloud features output by SA2 have the same resolution as the image features output by BLOCK 3, the point cloud features output by SA3 have the same resolution as the image features output by BLOCK 4, and the point cloud features output by SA4 have the same resolution as the image features output by BLOCK 5.

[0042] Furthermore, the detection device 130 can perform feature fusion once or multiple times on four pairs of image features and point cloud features with the same resolution. For example, the point cloud feature representations output by SA1 to SA4 can be represented sequentially as {S} l}(l∈[1,2,3,4]), the image feature representations output by BLOCK 2 to BLOCK 5 can be represented as {F l}(l∈[2,3,4,5]).

[0043] In some embodiments, in order to achieve the fusion between point cloud features and corresponding image features, the detection device 130 can determine the image features corresponding to each point in the point cloud feature representation.

[0044] In some embodiments, the detection device 130 may determine the target image features corresponding to the target point cloud features from a set of image features based on the alignment between the point cloud data 120 and the image data 110.

[0045] Specifically, the detection device 130 can determine the projection matrix M from the point cloud data 120 to the image data 110 based on the alignment between the point cloud data 120 and the image data 110. Thus, for a specific location e in a set of point cloud features, the detection device 130 can determine the corresponding location in a set of image features based on the projection matrix M. Therefore, the detection device 130 can detect a set of image features F l+1 The position in the middle Sampling is performed to determine the corresponding target image features. For example, a set of image features F... l+1 After sampling position by position, it can be converted into V l+1 .

[0046] Furthermore, the detection device 130 can determine the target fusion features based on the target point cloud features and the target image features. Figure 3 For example, detection device 130 can detect a set of point cloud features S l and a corresponding set of sampled image features V l+1 Perform point-by-point feature fusion.

[0047] The following will be referenced Figure 4 This section introduces the fusion of a set of point cloud features S l and a corresponding set of sampled image features V l+1 An example process for performing point-by-point feature fusion. Figure 4 A schematic diagram 400 illustrating feature fusion according to some embodiments of the present disclosure is shown.

[0048] Specifically, the detection device 130 can first cascade the target point cloud features and the target image features to determine the cascaded features. For example... Figure 4 As shown, the detection device 130 can detect a set of point cloud features S l With the corresponding set of sampled image features V l+1 Perform position-wise feature concatenation to obtain a set of concatenated features SV. l .

[0049] Furthermore, the detection device 130 can utilize a Multi-Layer Perception (MLP) model to process cascaded features to determine weighted features. For example... Figure 4 As shown, the detection device 130 can determine the weighted feature map W based on the following process:

[0050] W=σ(MLP(SV l (1)

[0051] Here, σ represents the sigmoid activation function. Therefore, the detection device 130 can determine the weights for each location.

[0052] Furthermore, the detection device 130 can calculate a set of point cloud features S based on the weights represented by the weighted feature map W. l Weighted point cloud features And calculate the corresponding set of sampled image features V l+1 Weighted image features Furthermore, the detection device 130 can utilize cascaded weighted point cloud features. and weighted image features A set of fused features is determined and used as input to the next SA layer, or as input for feature passing. The above process can be represented, for example, as follows:

[0053]

[0054]

[0055]

[0056] Continue to refer to Figure 3 ,exist Figure 3 For example, detection device 130 may perform four point-by-point feature fusions. Specifically, detection device 130 may perform point-by-point feature fusion based on the point cloud features output from SA1 layer (which, for example, correspond to a first resolution) and the image feature representation output from BLOCK 2, as feature input to SA2 layer.

[0057] Furthermore, the detection device 130 can utilize the SA2 layer to process the feature input to determine the point cloud features output by the SA2 layer. Additionally, the detection device 130 can perform point-by-point feature fusion using the point cloud features output by the SA2 layer (which, for example, correspond to the second resolution) and the image feature representation output by BLOCK 3, as feature input to the SA3 layer.

[0058] Furthermore, the detection device 130 can utilize the SA3 layer to process the feature input to determine the point cloud features output by the SA3 layer. Additionally, the detection device 130 can perform point-by-point feature fusion using the point cloud features output by the SA3 layer (which, for example, correspond to the third resolution) and the image feature representation output by BLOCK 4, as feature input to the SA4 layer.

[0059] Furthermore, the detection device 130 can utilize the SA4 layer to process the feature input to determine the point cloud features output by the SA4 layer. Additionally, the detection device 130 can perform point-by-point feature fusion using the point cloud features output by the SA4 layer (which, for example, correspond to the second resolution) and the image feature representation output by BLOCK 5, as a feature representation for input to the RPN (Region Proposal Network) model 330.

[0060] Continue to refer to Figure 2 In box 206, the detection device 130 determines a 3D detection box 140 and classification information associated with a candidate object based on a set of fused features, wherein the classification information indicates the confidence level of the candidate object belonging to multiple predetermined categories.

[0061] In some embodiments, the detection device 130 may, for example, use the RPN model 330 to directly input the 3D detection box 140 and the corresponding classification information.

[0062] In some embodiments, to improve detection accuracy, the detection device 130 may also perform region-level feature fusion. Specifically, such as Figure 3As shown, through the calculations of the ROI Pooling module 350, the detection device 130 can, for example, determine at least one 3D candidate bounding box, and can further acquire the region features associated with the at least one 3D candidate bounding box. Exemplarily, the region features can be represented as {r1, r2, ..., r...} t}, where t represents the number of 3D candidate boxes.

[0063] Furthermore, the detection device 130 can also acquire global image features determined by the image feature extraction module 320 based on the image data 110. For example, the detection device 130 can process the features F output by BLOCK 5. 5 Perform a MaxPooling operation to determine global image features f360:

[0064] f = MaxPooling(F) 5 (5)

[0065] Furthermore, the detection device 130 can determine region fusion features for at least one 3D candidate box based on region features and global image features. For example, the detection device 130 can cascade the region features corresponding to each 3D candidate box and the global image features to determine the region fusion features:

[0066]

[0067] Furthermore, the detection device 130 can determine the 3D detection box 140 and classification information associated with the candidate object based on the region fusion features.

[0068] The above describes the process of performing object detection using a 3D object detection model. It can be seen that, based on the above method, the embodiments of this disclosure can fuse the semantic information expressed by image data with point cloud features through at least one point-by-point feature fusion, thereby improving the accuracy of 3D object detection.

[0069] Furthermore, embodiments of this disclosure can also perform region-level secondary feature fusion, thereby further improving the accuracy of 3D object detection.

[0070] The following will be referenced Figure 5 This section will introduce the training process of a 3D object detection model. Figure 5 A flowchart of an example process 500 for model training according to some embodiments of the present disclosure is shown. It should be understood that process 500 can be performed by any suitable training device, which can be... Figure 1 The detection equipment 130 is the same as or different from the electronic equipment.

[0071] like Figure 5As shown in box 510, the training device can utilize the point cloud feature extraction model and the image feature extraction model to determine a set of training fusion features based on the training point cloud data and the training image data.

[0072] Continue to refer to Figure 3 This describes the process of training a 3D object detection model. Specifically, the training device can receive training point cloud data and training image data, and determine a set of training fused features based on the feature fusion process discussed above.

[0073] In box 520, the training device can determine the first classification information by processing a set of training fusion features using a region detection model.

[0074] like Figure 3 As shown, the training device can, for example, utilize the RPN model 330 to determine multiple candidate regions indicated by training point cloud data and training image data, and their corresponding classifications. Specifically, in some embodiments, the training device can utilize a region detection model (e.g., the RPN model) to determine multiple sets of classification probabilities corresponding to the multiple candidate regions, where each set of classification probabilities indicates the probability that the corresponding candidate region belongs to multiple predetermined categories. Exemplarily, the multiple sets of classification probabilities can be represented as follows: Where N represents the number of candidate regions and K represents the number of predetermined categories.

[0075] Furthermore, the training device can aggregate multiple probability distributions to determine the first classification information. In some embodiments, the training device can determine the maximum probability corresponding to each of multiple predetermined categories among the multiple probability distributions to obtain a set of maximum probabilities. Further, the training device can determine the first classification information q based on the set of maximum probabilities. m This process can be represented, for example, as:

[0076]

[0077] Max represents the maximum value operation, which takes an N*K matrix as input and outputs the maximum value in each row to obtain a 1*K vector.

[0078] In box 530, the training device can determine second classification information by processing training image features using an image classification model, wherein the training image features are generated by an image feature extraction model based on the training image data.

[0079] In some embodiments, the training device can process the feature representation F generated by BLOCK5, for example, using K classifier models. 5 Thus, the second classification information p is determined. m , which represents the probability of including objects from each of the K predefined categories.

[0080] In box 540, the training device can collaboratively train the point cloud feature extraction model and the image feature extraction model based on the first difference between the first classification information and the second classification information.

[0081] In some embodiments, the training device may construct a training objective function based, for example, on a first difference between first classification information and second debinding information. Exemplarily, this difference may be expressed as q m With p m The KL divergence between them is used to represent:

[0082] L kl i =D kl (p m ||q m )+D kl (q m ||p m (8)

[0083] It should be understood that KL divergence is merely one exemplary way of representing the first difference, and other appropriate methods can be used to construct a measure to characterize the difference between the first and second classification information.

[0084] Furthermore, the training device can train a 3D object detection model based on the objective function to minimize the divergence represented by formula (8), that is, to make the first classification information and the second classification information as consistent as possible.

[0085] Based on this approach, the embodiments of this disclosure can further support multi-classification tasks, thereby effectively expanding the scope of application of 3D object detection.

[0086] In some embodiments, the training device may determine a second difference between the second classification information and the classification label information associated with the image data, and train the point cloud feature extraction model and the image feature extraction model collaboratively based on the first difference and the second difference.

[0087] In some embodiments, the overall objective function of the 3D object detection model can be constructed as follows:

[0088]

[0089] Where L 3d i For example, it can represent the difference between the 3D detection boxes output during training and the labeled detection boxes; L kl i This refers to the difference between primary and secondary classification information, as discussed above; L mli This represents the difference between the second classification information of the image and the classification label information of the image; λ and μ represent the weight coefficients, and n represents the number of 3D detection boxes.

[0090] In some embodiments, L ml i It can be represented as:

[0091] L ml i =y cT log (p m )+(1-y c ) T log(1-p m (10)

[0092] in This represents the ground truth classification vector determined based on image classification annotation information.

[0093] Based on the above training process, the embodiments of this disclosure train a three-dimensional object model, enabling it to better support multi-classification tasks, thereby expanding the applicable scenarios and detection accuracy of the three-dimensional object model.

[0094] Example devices and equipment

[0095] Figure 6 A schematic structural block diagram of an apparatus 600 for three-dimensional object detection according to certain embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the detection device 130 or other device implementing the process for three-dimensional object detection of the present disclosure.

[0096] like Figure 6 As shown, the apparatus 600 includes an extraction module 610 configured to process point cloud data using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution. The apparatus 600 also includes a fusion module 620 configured to determine a set of fused features based on the set of point cloud features and a set of image features associated with the first resolution, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data. Furthermore, the apparatus 600 includes a detection module 630 configured to determine 3D detection boxes and classification information associated with candidate objects based on the set of fused features, wherein the classification information indicates the confidence level of a candidate object belonging to multiple predetermined categories.

[0097] In some embodiments, the fusion module 620 is further configured to: for a target point cloud feature in a set of point cloud features: determine a target image feature corresponding to the target point cloud feature from a set of image features based on the alignment between the point cloud data and the image data; and determine a target fusion feature based on the target point cloud feature and the target image feature.

[0098] In some embodiments, the fusion module 620 is further configured to: cascade target point cloud features and target image features to determine cascaded features; process the cascaded features using a multilayer perceptron (MLP) model to determine weight coefficients; determine weighted point cloud features and weighted image features based on the weight coefficients; and cascade weighted point cloud features and weighted image features to determine target fusion features.

[0099] In some embodiments, a set of point cloud features is a first set of point cloud features, a set of image features is a first set of image features, and a set of fused features is a first set of fused features. The detection module 630 is further configured to: process the first set of fused features using a point cloud feature extraction model to determine a second set of point cloud features associated with a second resolution; determine a second set of fused features based on the second set of point cloud features and the second set of image features associated with the second resolution, wherein the set of image features is based on the first set of image features determined by the image feature extraction model; and determine a 3D detection box and classification information associated with a candidate object based on the second set of fused features.

[0100] In some embodiments, the detection module 630 is further configured to: determine region features associated with at least one 3D candidate box based on a set of fusion features; acquire global image features determined by the image feature extraction module based on image data; determine region fusion features for at least one 3D candidate box based on the region features and the global image features; and determine 3D detection boxes and classification information associated with the candidate object based on the region fusion features.

[0101] In some embodiments, the apparatus 600 further includes a training module configured to: determine a set of training fusion features using a point cloud feature extraction model and an image feature extraction model, based on training point cloud data and training image data; determine first classification information by processing the set of training fusion features using a region detection model; determine second classification information by processing training image features using an image classification model, wherein the training image features are generated by the image feature extraction model based on the training image data; and collaboratively train the point cloud feature extraction model and the image feature extraction model based on a first difference between the first classification information and the second classification information.

[0102] In some embodiments, the training module is further configured to: determine a second difference between the second classification information and the classification annotation information associated with the image data; and collaboratively train a point cloud feature extraction model and an image feature extraction model based on the first difference and the second difference.

[0103] In some embodiments, the training module is further configured to: utilize a region detection model to determine multiple sets of classification probabilities corresponding to multiple candidate regions, wherein each set of classification probabilities indicates the probability that the corresponding candidate region belongs to multiple predetermined categories; and aggregate multiple sets of probability distributions to determine first classification information.

[0104] In some embodiments, the training module is further configured to: determine the maximum probability corresponding to each of the multiple predetermined categories in the multiple probability distributions to obtain a set of maximum probabilities; and determine first classification information based on the set of maximum probabilities.

[0105] Figure 7 A block diagram is shown illustrating an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The illustrated electronic device 700 can be included in or implemented as Figure 1 The additional testing device 130 or training device or other device for implementing the present disclosure for three-dimensional object detection or training a three-dimensional object detection model.

[0106] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose computing device. Electronic device 700 can also be any type of computing device or server. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 can be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0107] Electronic device 700 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., map data) and can be accessed within electronic device 700.

[0108] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0109] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0110] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0111] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions or programs thereon, wherein the computer-executable instructions or programs are executed by a processor to implement the methods or functions described above. The computer-readable storage medium may include a non-transitory computer-readable medium. According to an exemplary implementation of this disclosure, a computer program product is also provided, including computer-executable instructions or programs, which are executed by a processor to implement the methods or functions described above. This computer program product may be tangibly embodied on a non-transitory computer-readable medium.

[0112] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions or programs.

[0113] These computer-executable instructions or programs can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-executable instructions or programs can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0114] Computer-executable instructions or programs may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

[0117] Example implementation

[0118] TS 1. A method for three-dimensional object detection, comprising:

[0119] Point cloud data is processed using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution.

[0120] Based on the set of point cloud features and a set of image features associated with the first resolution, a set of fused features is determined, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data; and

[0121] Based on the set of fused features, a 3D detection bounding box and classification information associated with the candidate object are determined, wherein the classification information indicates the confidence level of the candidate object belonging to multiple predetermined categories.

[0122] TS 2. The method according to TS 1, wherein determining a set of fusion features includes:

[0123] For the target point cloud features in the aforementioned set of point cloud features:

[0124] Based on the alignment between the point cloud data and the image data, a target image feature corresponding to the target point cloud feature is determined from the set of image features; and

[0125] Based on the target point cloud features and the target image features, the target fusion features are determined.

[0126] TS 3. The method according to TS 2, wherein determining the target fusion features based on the target point cloud features and the target image features includes:

[0127] The target point cloud features and the target image features are cascaded to determine the cascaded features;

[0128] The cascaded features are processed using a multilayer perceptron (MLP) model to determine weight coefficients; based on these weight coefficients, weighted point cloud features and weighted image features are determined; and

[0129] The weighted point cloud features and the weighted image features are cascaded to determine the target fusion features.

[0130] TS 4. According to the method of TS 1, wherein the set of point cloud features is a first set of point cloud features, the set of image features is a first set of image features, the set of fusion features is a first set of fusion features, and determining the 3D detection box and classification information associated with the candidate object includes:

[0131] The first set of fused features is processed using the point cloud feature extraction model to determine a second set of point cloud features associated with the second resolution.

[0132] Based on the second set of point cloud features and the second set of image features associated with the second resolution, a second set of fused features is determined, wherein the set of image features is based on the first set of image features determined by the image feature extraction model; and

[0133] Based on the second set of fusion features, a 3D detection box and classification information associated with the candidate object are determined.

[0134] TS 5. The method according to TS 1, wherein determining the 3D detection bounding box and classification information associated with the candidate object includes:

[0135] Based on the set of fused features, determine the region features associated with at least one 3D candidate box;

[0136] Obtain global image features determined by the image feature extraction module based on the image data;

[0137] Based on the region features and the global image features, determine the region fusion features for the at least one 3D candidate box; and

[0138] Based on the region fusion features, the 3D detection bounding boxes and classification information associated with the candidate objects are determined.

[0139] TS 6. The method according to TS 1, wherein the point cloud feature extraction model and the image feature extraction model are trained based on the following process:

[0140] Using point cloud feature extraction models and image feature extraction models, and based on training point cloud data and training image data, a set of training fusion features is determined;

[0141] The first classification information is determined by processing the set of trained fusion features using a region detection model;

[0142] By processing the training image features using the image classification model, second classification information is determined, wherein the training image features are generated by the image feature extraction model based on the training image data; and

[0143] Based on the first difference between the first classification information and the second classification information, the point cloud feature extraction model and the image feature extraction model are trained collaboratively.

[0144] TS 7. The method according to TS 6, wherein co-training the point cloud feature extraction model and the image feature extraction model comprises:

[0145] Determine the second difference between the second classification information and the classification annotation information associated with the image data; and

[0146] Based on the first difference and the second difference, the point cloud feature extraction model and the image feature extraction model are trained collaboratively.

[0147] TS 8. The method according to TS 6, wherein determining the first classification information includes:

[0148] Using the region detection model, multiple sets of classification probabilities are determined for each candidate region, wherein each set of classification probabilities indicates the probability that the corresponding candidate region belongs to the multiple predetermined categories; and

[0149] The multiple probability distributions are aggregated to determine the first classification information.

[0150] TS 9. The method according to TS 8, wherein aggregating the multiple probability distributions comprises:

[0151] Determine the maximum probability corresponding to each of the multiple predetermined categories in a plurality of probability distributions to obtain a set of maximum probabilities; and

[0152] The first classification information is determined based on the set of maximum probabilities.

[0153] TS 10. An organization for 3D object detection, comprising:

[0154] The extraction module is configured to process point cloud data using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution.

[0155] A fusion module is configured to determine a set of fused features based on the set of point cloud features and a set of image features associated with the first resolution, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data; and

[0156] The detection module is configured to determine a 3D detection bounding box and classification information associated with a candidate object based on the set of fused features, wherein the classification information indicates the confidence level of the candidate object belonging to multiple predetermined categories.

[0157] TS 11. An electronic device comprising:

[0158] Memory and processor;

[0159] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of TS 1 to 9.

[0160] TS 12. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of TS 1 to 9.

[0161] TS 13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of TS 1 to 9.

Claims

1. A method for detecting three-dimensional objects, comprising: Point cloud data is processed using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution. Based on the set of point cloud features and a set of image features associated with the first resolution, a set of fused features is determined, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data; as well as Based on the aforementioned set of fused features, a 3D bounding box and classification information associated with the candidate object are determined. The classification information indicates the confidence level that the candidate object belongs to multiple predetermined categories. The determination of the 3D detection bounding box and classification information associated with the candidate object includes: Based on global image features and region features associated with at least one 3D candidate box, region fusion features are determined for the at least one 3D candidate box, wherein the region features associated with the at least one 3D candidate box are determined based on the set of fusion features, and the global image features are determined by the image feature extraction model based on the image data; and Based on the region fusion features, the 3D detection bounding boxes and classification information associated with the candidate objects are determined.

2. The method of claim 1, wherein determining a set of fusion features comprises: For the target point cloud features in the aforementioned set of point cloud features: Based on the alignment between the point cloud data and the image data, a target image feature corresponding to the target point cloud feature is determined from the set of image features; as well as Based on the target point cloud features and the target image features, the target fusion features are determined.

3. The method according to claim 2, wherein determining the target fusion features based on the target point cloud features and the target image features comprises: The target point cloud features and the target image features are cascaded to determine the cascaded features; The cascaded features are processed using a multilayer perceptron (MLP) model to determine the weighting coefficients. Based on the weighting coefficients, the weighted point cloud features and weighted image features are determined; as well as The weighted point cloud features and the weighted image features are cascaded to determine the target fusion features.

4. The method according to claim 1, wherein the set of point cloud features is a first set of point cloud features, the set of image features is a first set of image features, the set of fusion features is a first set of fusion features, and determining the 3D detection box and classification information associated with the candidate object includes: The first set of fused features is processed using the point cloud feature extraction model to determine a second set of point cloud features associated with the second resolution. Based on the second set of point cloud features and the second set of image features associated with the second resolution, a second set of fused features is determined, wherein the second set of image features is based on the first set of image features determined by the image feature extraction model. as well as Based on the second set of fusion features, a 3D detection box and classification information associated with the candidate object are determined.

5. The method of claim 1, wherein the point cloud feature extraction model and the image feature extraction model are trained based on the following process: Using point cloud feature extraction models and image feature extraction models, and based on training point cloud data and training image data, a set of training fusion features is determined; The first classification information is determined by processing the set of trained fusion features using a region detection model; The second classification information is determined by processing the training image features using an image classification model, wherein the training image features are generated by the image feature extraction model based on the training image data; as well as Based on the first difference between the first classification information and the second classification information, the point cloud feature extraction model and the image feature extraction model are trained collaboratively.

6. The method of claim 5, wherein collaboratively training the point cloud feature extraction model and the image feature extraction model comprises: Determine a second difference between the second classification information and the classification annotation information associated with the image data; as well as Based on the first difference and the second difference, the point cloud feature extraction model and the image feature extraction model are trained collaboratively.

7. The method of claim 5, wherein determining the first classification information comprises: Using the region detection model, multiple sets of classification probabilities corresponding to multiple candidate regions are determined, wherein each set of classification probabilities indicates the probability that the corresponding candidate region belongs to the multiple predetermined categories; as well as The multiple sets of classification probabilities are aggregated to determine the first classification information.

8. The method of claim 7, wherein aggregating the multiple groups of classification probabilities comprises: Determine the maximum probability among multiple sets of classification probabilities corresponding to each of the multiple predetermined categories to obtain a set of maximum probabilities; as well as The first classification information is determined based on the set of maximum probabilities.

9. An apparatus for three-dimensional object detection, comprising: The extraction module is configured to process point cloud data using a point cloud feature extraction model to determine a set of point cloud features associated with a first resolution. The fusion module is configured to determine a set of fused features based on the set of point cloud features and a set of image features associated with the first resolution, wherein the set of image features is determined by an image feature extraction model based on image data aligned with the point cloud data; as well as The detection module is configured to determine, based on the set of fused features, a 3D detection bounding box and classification information associated with a candidate object, wherein the classification information indicates the confidence level that the candidate object belongs to multiple predetermined categories. The detection module is further configured to determine region fusion features for the at least one 3D candidate box based on global image features and region features associated with the at least one 3D candidate box, wherein the region features associated with the at least one 3D candidate box are determined based on the set of fusion features, and the global image features are determined by the image feature extraction model based on the image data; and to determine 3D detection boxes and classification information associated with candidate objects based on the region fusion features.

10. An electronic device, comprising: Memory and processor; The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images