Object detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an object detection method and related equipment. The method comprises: receiving three-dimensional data and image data for object detection; calculating a first feature of the three-dimensional data and a second feature of the image data; calculating a corresponding first object query feature based on the three-dimensional data; calculating a corresponding second object query feature based on the three-dimensional data and a projection relationship between the three-dimensional data and the image data; calculating attention based on the first feature, the second feature, the first object query feature and the second object query feature to obtain attention data; and outputting an object detection result based on the attention data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an object detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] 3D object detection is an important research topic in computer vision, typically using 3D point clouds as input to a conventional setup. A trend in related techniques is to utilize multiple input data sources, such as supplementing 3D point clouds with 2D images, which generally have richer colors and less noise. However, due to the heterogeneous geometry of 2D and 3D representations, it is difficult to apply readily available neural networks to achieve multimodal fusion (fusion of 3D point clouds and 2D images). Summary of the Invention

[0003] In view of this, the purpose of this disclosure is to provide an object detection method, apparatus, electronic device and storage medium to solve or partially solve the above-mentioned technical problems.

[0004] A first aspect of this disclosure provides an object detection method, comprising:

[0005] Receive 3D data and image data for object detection;

[0006] Calculate the first feature of the three-dimensional data and the second feature of the image data;

[0007] Calculate the corresponding first object query feature based on the three-dimensional data;

[0008] Based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data, the second object query feature corresponding to the image data is calculated;

[0009] Based on the first feature, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data; and

[0010] Based on the attention data, the object detection result is output.

[0011] A second aspect of this disclosure provides an object detection device, comprising:

[0012] The data receiving module is configured to receive 3D data and image data for object detection;

[0013] The feature calculation module is configured to: calculate a first feature of the three-dimensional data and a second feature of the image data; calculate a first object query feature based on the three-dimensional data; and calculate a second object query feature corresponding to the image data based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data.

[0014] The object detection module is configured to: calculate attention based on the first feature, the second feature, the first object query feature, and the second object query feature to obtain attention data; and output object detection results based on the attention data.

[0015] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the computer program, implements the method as described in the first aspect.

[0016] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.

[0017] As can be seen from the above, the object detection method, apparatus, electronic device and storage medium provided in this disclosure use object query features to bridge three-dimensional space and two-dimensional space, thereby associating data from different modalities without needing to serialize the data from different modalities and then fuse them using a full attention mechanism. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an exemplary method provided in an embodiment of this disclosure is shown.

[0020] Figure 2A A schematic diagram of an exemplary model provided by an embodiment of this disclosure is shown.

[0021] Figure 2B A comparative test diagram of the object detection method provided in this disclosure with other models is shown in the table.

[0022] Figure 3 A schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure is shown.

[0023] Figure 4 A schematic diagram of the structure of an exemplary device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0025] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] 3D object detection is a core algorithm module in many scenarios, including autonomous driving, industrial robots, and drones. 3D point clouds and 2D images are the two most common and readily available data modalities. A 3D point cloud consists of multiple points in three-dimensional space. While 3D point clouds provide more accurate information than 2D images, they lack features such as color and semantic meaning. Therefore, fusing these two modalities—3D point clouds and 2D images—to perform 3D object detection is a more reasonable approach.

[0027] For 3D learning tasks, point clouds can provide necessary geometrical cues, while information from rich color images can supplement point clouds by filling in missing color information and correcting noise errors. Therefore, the involvement of 2D images may improve the performance of 3D object detection.

[0028] One approach in related technologies is to extract 3D RGB vectors from images to expand point features in point clouds. However, the 3D detection model imVoteNet, based on convolutional neural networks (CNNs), points out that this method struggles to transfer the differences between 2D and 3D features.

[0029] Therefore, imVoteNet replaces the RGB vectors with image features extracted by a pre-trained 2D detector, then uses PointNet++ to extract seed points of the point cloud, finds the approximate projection of 3D points to 2D based on camera parameters, and finally fuses the geometric, semantic, and texture information of 2D points to the corresponding 3D seed points by calculating them.

[0030] However, the assumption of relying on both image voting and point cloud voting in imVoteNet can accumulate inherent grouping errors. Furthermore, since the point cloud learning process is easily influenced by the interaction of mid-layer 2D / 3D features, imVoteNet simply combines image and point features in the first CNN layer. However, this approach prevents the network from fully utilizing the semantic relevance of 2D and 3D features or transferring the differences between them.

[0031] In view of this, embodiments of this disclosure propose an object detection method and related device that uses object query features to bridge three-dimensional space and two-dimensional space, thereby associating data from different modalities without needing to serialize the data from different modalities and then fuse them using a full attention mechanism.

[0032] Figure 1 A flowchart illustrating an exemplary method 100 provided in an embodiment of this disclosure is shown. This method 100 can be used for object detection. In some embodiments, this method 100 can be implemented using a predetermined model. In some embodiments, the predetermined model can be a single Transformer model. Figure 1 As shown, the method 100 may include the following steps.

[0033] In step 102, 3D data and image data for object detection can be received first. The 3D data can be a point cloud, which can be derived from data acquired by a depth camera. The image data can be 2D image data, such as an RGB image.

[0034] In order to extract features from point cloud and RGB images, in some embodiments we can first preprocess the point cloud and RGB images separately.

[0035] Figure 2A A schematic diagram of an exemplary model 200 provided in an embodiment of this disclosure is shown.

[0036] like Figure 2AAs shown, assuming the point cloud consists of N0 points, each point has 3 dimensions (its three-dimensional coordinates), the point cloud can be represented by N0×3. Then, PointNet++ can be used to sample the point cloud and learn the point features to obtain N... pnt There are 3+F seed points, each with a dimension of 3+F, which can be expressed as N. pnt These seed points are represented by ×(3+F), such as Figure 2A As shown. Here, 3 and F represent the dimensions of 3D Euclidean coordinates and point features, respectively.

[0037] Corresponding to a point cloud can be multiple two-dimensional RGB images. For example... Figure 2A As shown, we select the first, second, and third images corresponding to the point cloud. Assume each image consists of an H×W×3 image. Here, H and W are the height and width of the image, respectively, and 3 represents the three color channels of the image—R, G, and B. Then, we can divide each image into patches, for example, dividing an image into N... pat Image patches, such as Figure 2A As shown.

[0038] Then, in step 104, the first feature of the three-dimensional data and the second feature of the image data can be calculated.

[0039] In some embodiments, data N pnt A linear mapping layer 202a with ×(3+F) input sampling points can be used to obtain N. pnt The first feature (point token) is represented as Pnt-Tok#1~#N. pnt ,like Figure 2A As shown. The linear mapping layer 202a for this sampling point can be a multilayer perceptron (MLP). Here, it can be assumed that the MLP will process the data N pnt ×(3+F) is mapped to a dimension of size D, and correspondingly, the first feature can be represented as

[0040] In some embodiments, N pat A linear mapping layer 202b of input image patches can yield N. pat The second feature (patch token) is represented as Pat-Tok#1 to #N. pnt ,like Figure 2A As shown. The linear mapping layer 202b of this image patch can also be a multilayer perceptron (MLP). Here, it can be assumed that the MLP will map N... pat Each image patch is mapped to a dimension of size D; correspondingly, the second feature can be represented as...

[0041] In step 106, the corresponding first object query feature (point object query) can be calculated based on the three-dimensional data.

[0042] In some embodiments, the kNN algorithm (k-nearest neighbor algorithm) can be used first to start from N pnt K points are sampled from a seed point as proposals. Here, we can use... and These represent the 3D coordinates and features of the K suggested points, respectively.

[0043] Then, biases learned based on the features of these K suggestion points can be added to these K suggestion points to obtain multiple corrected suggestion points. The biases learned based on the features of these K suggestion points can be adjusted by applying the features f of these K suggestion points... pnt The following equation (1) illustrates the calculation process: Input an MLP.

[0044] k′ pnt =k pnt +MLP(f pnt (1)

[0045] Next, based on these K revised suggestion points k′ pnt The first object query feature can be calculated as o pnt .

[0046] In some embodiments, these K suggested correction points k′ can be... pnt Input an MLP to get o pnt Meanwhile, in order to achieve the input required by the predetermined model, features can be queried from the first object. pnt Add a positional embedding (PE). Equation (2) below illustrates this calculation process.

[0047] o pnt =MLP(k′) pnt )+PE (2)

[0048] in, It can be embedded at a randomly initialized position.

[0049] In this way, correspondingly, we can obtain K first object query features. pnt As can be seen, the first object query feature o pnt With k′ pnt Therefore, the first object query feature o can be used as a condition. pntThis is called conditional object queries.

[0050] In step 108, the second object query feature (patch object query) corresponding to the image data can be calculated based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data.

[0051] In some embodiments, the second object query feature can be calculated based on the aforementioned K corrected suggestion points and the projection relationship between the point cloud and the RGB image.

[0052] For example, based on the projection relationship between the point cloud and the RGB image, K corrected suggestion points k′ can be selected. pnt Projecting the image onto an RGB image yields the two-dimensional coordinates of K projected points, denoted as...

[0053] Then, based on these K projection points, the query features of the second object are calculated, which can be represented as o. pat .

[0054] In some embodiments, these K projection points proj(k′) can be... pnt Input an MLP to get o pat Meanwhile, in order to achieve the input required by the predetermined model, features can be queried from the second object. pat Add a positional embedding (PE). Equation (3) below illustrates this calculation process.

[0055] o pat =MLP(proj(k′) pnt ))+PE (3)

[0056] in, It can be a randomly initialized position embedding. In some embodiments, in order to align o pnt and o pat The PEs added in the first and second object query features are the same. Shared location embedding intuitively tells the predefined model, o pnt and o pat It is aligned.

[0057] In this way, correspondingly, we can obtain K second object query features. pat As can be seen, the second object query feature o pat With proj(k′) pnt Therefore, the second object query feature can be used as a condition. patAlso known as conditional object queries.

[0058] Given the intrinsic and extrinsic parameters of the camera, each 3D point can be projected onto the camera plane, that is, associating 3D coordinates with 2D image pixels.

[0059] Therefore, in some embodiments, a projection operator can be defined to compute proj in the foregoing embodiments. This represents the coordinates of a 3D point, c = [x, y, z]. T The two-dimensional pixel coordinates projected onto the corresponding image are c' = [u, v]. T And it has:

[0060]

[0061] Here, K and Rt are the intrinsic and extrinsic matrices, corresponding to the intrinsic and extrinsic parameters of a given camera, while Π is the perspective mapping.

[0062] In this way, k′ can be calculated. pnt The projection point proj(k′) onto the RGB image pnt ).

[0063] Thus, N was obtained. pnt Point cloud features and N pat There are 1 image feature and 2K object query features, where the first K are point cloud object queries (denoted as o). pnt The last K are image object queries (denoted as o). pat K is generally more than the number of possible objects.

[0064] As can be seen, the aforementioned embodiments have thus utilized conditional object queries as a bridge connecting 3D and 2D spaces, thereby employing conditional object queries to facilitate the predictive learning process of point- and image-based object query feature alignment.

[0065] After calculating the first feature, the second feature, the first object query feature, and the second object query feature, in step 110, attention can be calculated based on the first feature, the second feature, the first object query feature, and the second object query feature to obtain attention data.

[0066] In some embodiments, first attention data can be calculated based on the first feature and the first object query feature; second attention data can be calculated based on the second feature and the second object query feature; and third and fourth attention data can be calculated based on the first feature, the second feature, the first object query feature, and the second object query feature.

[0067] In some embodiments, attention can be computed using a predefined model (e.g., Transformer model 204).

[0068] like Figure 2A As shown, the first feature p pnt Second feature p pat Position embeddings (PEs) can also be added before Transformer model 204. This is to make the first feature p... pnt Second feature p pat Query features of the first object pnt Second object query feature record pat Alignment: their positions embedded in the PE can be the same.

[0069] Suppose that in this step, these features are fed into the l-th layer of the Transformer model, where l = 1, ..., L. The first feature input to the l-th layer is denoted as... The second feature is denoted as The first object query feature is denoted as The second object query feature is denoted as

[0070] In some embodiments, the Transformer model 204 may further include a first group of model units for processing a first feature, a second group of model units for processing a second feature, a third group of model units for processing a first object query feature, and a fourth group of model units for processing a third object query feature. Figure 2A As shown, the first model unit group may include first model units 2042a to 2042d, the second model unit group may include second model units 2044a to 2044d, the third model unit group may include third model units 2046a and 2046b, and the fourth model unit group may include fourth model units 2048a and 2048b. Each model unit can be a basic unit constituting the Transformer model 204, possessing the structure of an encoder-decoder based on a multi-head attention mechanism of the Transformer model.

[0071] Specifically, the first model units 2042a-2042d in the first model unit group can calculate first attention data based on the first feature and the first object query feature. The second model unit in the second model unit group can calculate second attention data based on the second feature and the second object query feature. The third model unit in the third model unit group can calculate third attention data based on the first feature, the second feature, the first object query feature, and the second object query feature. The fourth model unit in the fourth model unit group can calculate fourth attention data based on the first feature, the second feature, the first object query feature, and the second object query feature.

[0072] like Figure 2A As shown, the aforementioned model units can be divided into two layers. The first layer of model units includes first model units 2042a and 2042b, second model units 2044a and 2044b, third model unit 2046a, and fourth model unit 2048a. The second layer of model units includes first model units 2042c and 2042d, second model units 2044c and 2044d, third model unit 2046b, and fourth model unit 2048b.

[0073] Specifically, during model training, the first feature is prohibited between model units. Second feature Attention between them. Therefore, the first feature Only query features of the first object Computational attention, second feature Only query features of the second object Attention is calculated. The first attention data calculated by the first and third model units, and the second attention data calculated by the second and fourth model units, are respectively fed into the first and second model units of the next layer, which can be represented as follows: and Equation (5) below gives the calculation process of the first attention data and the second attention data.

[0074]

[0075] During model prediction, the above calculations can be achieved by applying a zero mask to the obtained attention.

[0076] Here, although it is prevented in the Transformer module and Attention between them, but and It also has additional features to further bridge the gap between 3D and 2D coordinates. For this purpose, and Attention can be associated with all features; that is, the first object query feature. With the first feature Second feature Second object query features Calculate attention, second object query features With the first feature Second feature and the first object query features Attention is calculated. The third attention data calculated by the first, second, third, and fourth model units, and the fourth attention data calculated by the first, second, third, and fourth model units, are respectively fed into the third and fourth model units of the next layer, which can be represented as follows: and Equation (6) below gives the calculation process of the third attention data and the fourth attention data.

[0077]

[0078] The following is a brief introduction to the mechanism of attention calculation.

[0079] Multi-head attention (MSA), as a fundamental building block of the Transformer architecture, takes three sets of inputs: a query set, a key set, and a value set. Given a query set {q} i} and a common set of elements {p} of a key set and a value set k The MSA output features for each query element are aggregated through linearly projected weighted values, as shown in the formula:

[0080]

[0081] Where h indexes H attention heads; W h and V h These are the weights for the output projection and the value projection, respectively. This is the attention weight, calculated using the following formula:

[0082]

[0083] Q h and U h These represent query projection weight and key projection weight, respectively.

[0084] Thus, based on the above formula, the corresponding attention data can be calculated.

[0085] In addition to the first feature Second feature Query features of the first object Second object query features In addition to correlation, in some embodiments, the first feature is further enhanced by adding an inherent image patch-to-point projection during the calculation of attention data. Second feature The relationship between the two features is described. For example, based on the projection relationship between the 3D data and the image data, the projection features of the second feature are added to the first feature; then, based on the first feature with added projection features, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data.

[0086] Assuming, using N represents pnt The 3D coordinates of n sampling points pnt Projected onto the corresponding camera plane and obtained N pnt 2D pixel coordinates, these coordinates are used express.

[0087] Using the proj defined by formula (4), we can let u n and v n They are proj(n) pnt The x-axis and y-axis values ​​of the nth element of a given array, where n = 1, 2, ..., N. pnt If u n and v n Each satisfies the constraint 1≤u n ≤H and 1≤v n ≤W, such two-dimensional coordinates must reside within an input image of size H×W×3. (The last part, "u", appears to be a typo and can be omitted.) n and v n Rounding to the nearest integer yields the valid coordinates representing a specific image pixel. The index of the corresponding image patch can be obtained using the following formula:

[0088]

[0089] in It is a rounding (approximation) operator, pn∈{1, 2, ..., N} pat} is the block index corresponding to the nth point, and S represents the image patch size. The features of points and image patches are aggregated through the following expression (10) to achieve patch-to-point projection:

[0090]

[0091] Where "=" represents the assignment operator, and the subscripts n and p... nRepresenting features respectively and The above expression indicates that the output of the first model unit in the l-th layer (e.g., first model unit 2042b) is obtained by adding the corresponding index. The value obtained after the projection features is used to reassign the output of the first model unit 2042b in the l-th layer and input to the first model unit (e.g., the first model unit 2042d) in the (l+1)-th layer.

[0092] In this way, 2D information is integrated into the 3D model using a patch-to-point approach. This includes projection features. It can be added to the corresponding MLP after passing through it. This MLP can be used to perform displacement.

[0093] In step 112, the object detection results can be output based on the calculated attention data.

[0094] In some embodiments, step 112 may further include:

[0095] Based on the third attention data, output the 3D object detection result; and

[0096] Based on the fourth attention data, output the two-dimensional object detection result.

[0097] like Figure 2A As shown, model 204 has 2K outputs, corresponding to 2K object query inputs. For example, the K outputs of the third model unit and the K outputs of the fourth model unit can be output to MLP heads 206a and 206b respectively to output object detection results. Specifically, the K outputs of the third model unit can use MLP head 206a to predict the coordinates of 3D bounding boxes and their classification labels (cls). The K outputs of the fourth model unit can use MLP head 206b to predict 2D bounding boxes and their classification labels (cls).

[0098] It should be noted that the model 204 in this embodiment does not require additional 2D bounding box coordinate labels, because they can be obtained by projecting the 3D bounding box coordinate labels onto the 2D camera plane using formula (4) and then taking the axis-aligned 2D bounding box of the projected shape. Therefore, this embodiment can output both 3D and 2D detection boxes (BBoxes) during both training and inference.

[0099] In some embodiments, the SUNRGB-D dataset and the ScanNetV2 dataset can be used when training the model.

[0100] The SUNRGB-D dataset is a single-view RGB-D dataset for 3D scene understanding. It consists of 10,335 RGB-D images annotated with amodal-oriented 3D bounding boxes for 37 object categories and corresponding camera poses. The training and validation sets consist of 5,285 and 5,050 frames, respectively.

[0101] ScanNetV2 is a richly annotated 3D reconstruction mesh dataset for indoor scenes. It contains 1,513 scans covering over 700 unique indoor scenes, with 1,201 scans serving as the training set and the remaining 312 scans forming the validation set. ScanNetV2 includes over 2.5 million images with camera poses and their corresponding reconstructed point clouds, along with 3D semantic annotations for 18 object categories.

[0102] Furthermore, during model optimization, a two-part composite loss function can be designed: a regression loss for locating bounding boxes and a classification loss for predicting the relevant box categories. The formula for calculating the loss function is as follows:

[0103]

[0104] in, and The regression losses are for the 3D and 2D localization bounding boxes, respectively. and The α1, α2, and α3 are the classification losses for predicting the relevant box categories in 3D and 2D, respectively, and are three parameters that weight the relative importance of these losses.

[0105] Furthermore, in some embodiments, the method 100 provided in this disclosure can also combine point clouds with multi-view images, where the interaction between points and images, as well as the interaction between multi-view images, can further improve performance. When the input image of a single scene has different views, these images can be first connected along the width side to obtain a wide image. Then, method 100 is used to perform object detection.

[0106] The object detection method provided in this disclosure uses conditional object queries to associate 3D features with 2D features, thereby effectively fusing point clouds and images and obtaining better object detection results. Furthermore, the 3D features and 2D features are further associated through the projection of the 2D image onto the 3D points. In addition, the object detection method provided in this disclosure is the first to use a pure Transformer to fuse 3D point clouds and images. While traditional Transformer-based fusion sequentializes different modalities and then fuses them using a full attention mechanism, this disclosure directly associates features from different modalities, achieving better results. Furthermore, this disclosure can also fuse multiple images and point clouds.

[0107] Figure 2B A comparative test diagram of the object detection method (BrT) provided in the embodiments of this disclosure with other models is shown in the table.

[0108] from Figure 2B As can be seen, compared with the previous method, the object detection method (BrT) provided in this disclosure embodiment has an improvement of 2%-2.9% mAP (mean average accuracy).

[0109] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0110] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0111] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object detection method described in any of the above embodiments.

[0112] Figure 3This illustration shows a hardware structure diagram of a more specific electronic device 300 provided in this embodiment. This device 300 can be used to implement... Figure 1 Method 100. The device 300 may further include: a processor 302, a memory 304, an input / output interface 306, a communication interface 308, and a bus 310. The processor 302, memory 304, input / output interface 306, and communication interface 308 are interconnected within the device via the bus 310.

[0113] The processor 302 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0114] The memory 304 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 304 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 304 and is called and executed by the processor 302.

[0115] Input / output interface 306 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0116] The communication interface 308 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0117] Bus 310 includes a pathway for transmitting information between various components of the device (e.g., processor 302, memory 304, input / output interface 306, and communication interface 308).

[0118] It should be noted that although the above-described device only shows the processor 302, memory 304, input / output interface 306, communication interface 308, and bus 310, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0119] The electronic devices described above are used to implement the corresponding method 100 in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides an object detection device. Figure 4 A schematic diagram of the structure of an exemplary device 400 provided in an embodiment of this disclosure is shown.

[0121] refer to Figure 4 The object detection device 400 may include:

[0122] The data receiving module 402 is configured to receive 3D data and image data for object detection;

[0123] The feature calculation module 404 is configured to: calculate a first feature of the three-dimensional data and a second feature of the image data; calculate a first object query feature based on the three-dimensional data; and calculate a second object query feature corresponding to the image data based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data.

[0124] The object detection module 406 is configured to: calculate attention based on the first feature, the second feature, the first object query feature and the second object query feature to obtain attention data; and output object detection results based on the attention data.

[0125] In some embodiments, the object detection module 406 is configured to: calculate first attention data based on the first feature and the first object query feature; calculate second attention data based on the second feature and the second object query feature; and calculate third attention data and fourth attention data based on the first feature, the second feature, the first object query feature and the second object query feature.

[0126] In some embodiments, the object detection module 406 is configured to: use a predetermined model to calculate attention based on the first feature, the second feature, the first object query feature, and the second object query feature to obtain attention data.

[0127] In some embodiments, the predetermined model includes a first model unit group for processing the first feature, a second model unit group for processing the second feature, a third model unit group for processing the first object query feature, and a fourth model unit group for processing the third object query feature.

[0128] Wherein, the first model unit in the first model unit group is configured to calculate the first attention data based on the first feature and the first object query feature; the second model unit in the second model unit group is configured to calculate the second attention data based on the second feature and the second object query feature; the third model unit in the third model unit group is configured to calculate the third attention data based on the first feature, the second feature, the first object query feature and the second object query feature; and the fourth model unit in the fourth model unit group is configured to calculate the fourth attention data based on the first feature, the second feature, the first object query feature and the second object query feature.

[0129] In some embodiments, the object detection module 406 is configured to: output a three-dimensional object detection result based on the third attention data; and output a two-dimensional object detection result based on the fourth attention data.

[0130] In some embodiments, the object detection module 406 is configured to: add the projection feature of the second feature to the first feature according to the projection relationship between the three-dimensional data and the image data; and calculate attention based on the first feature with the added projection feature, the second feature, the first object query feature and the second object query feature to obtain attention data.

[0131] In some embodiments, the 3D data is a point cloud, and the image data is an RGB image.

[0132] In some embodiments, the feature calculation module 404 is configured to: sample multiple suggestion points from the point cloud based on the kNN algorithm; add biases learned based on the features of the multiple suggestion points to the multiple suggestion points to obtain multiple corrected suggestion points; and calculate the first object query feature based on the multiple corrected suggestion points.

[0133] In some embodiments, the feature calculation module 404 is configured to calculate the second object query feature based on the plurality of corrected suggestion points and the projection relationship between the three-dimensional data and the image data.

[0134] In some embodiments, the first object query feature and the second object query feature have the same location embedding.

[0135] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0136] The apparatus of the above embodiments is used to implement the corresponding method 100 in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0137] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method 100 as described in any of the above embodiments.

[0138] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0139] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the method 100 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0141] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0142] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0143] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An object detection method, comprising: Receive 3D data and image data for object detection; Calculate the first feature of the three-dimensional data and the second feature of the image data; Calculate the corresponding first object query feature based on the three-dimensional data; Based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data, the second object query feature corresponding to the image data is calculated; Based on the first feature, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data. as well as Based on the attention data, output the object detection result; The three-dimensional data is a point cloud, and the image data is an RGB image; The calculation of the corresponding first object query features based on the three-dimensional data includes: Based on the kNN algorithm, multiple suggestion points are sampled from the point cloud; By adding the learned bias based on the features of the multiple suggestion points to the multiple suggestion points, multiple corrected suggestion points are obtained; and Based on the multiple corrected suggestion points, the query features of the first object are calculated; Specifically, based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data, the second object query feature corresponding to the image data is calculated, including: Based on the multiple corrected suggested points and the projection relationship between the 3D data and the image data, the second object query feature is calculated.

2. The method as described in claim 1, wherein, Based on the first feature, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data, including: Based on the first feature and the first object query feature, calculate the first attention data; Based on the second feature and the second object query feature, calculate the second attention data; and Based on the first feature, the second feature, the first object query feature, and the second object query feature, calculate the third attention data and the fourth attention data.

3. The method as described in claim 1, wherein, Based on the first feature, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data, including: Using a predetermined model, attention is calculated based on the first feature, the second feature, the first object query feature, and the second object query feature to obtain attention data.

4. The method of claim 3, wherein, The predetermined model includes a first model unit group for processing the first feature, a second model unit group for processing the second feature, a third model unit group for processing the first object query feature, and a fourth model unit group for processing the second object query feature. Wherein, the first model unit in the first model unit group is configured to calculate the first attention data based on the first feature and the first object query feature; the second model unit in the second model unit group is configured to calculate the second attention data based on the second feature and the second object query feature; the third model unit in the third model unit group is configured to calculate the third attention data based on the first feature, the second feature, the first object query feature and the second object query feature; and the fourth model unit in the fourth model unit group is configured to calculate the fourth attention data based on the first feature, the second feature, the first object query feature and the second object query feature.

5. The method according to any one of claims 2-4, wherein, Based on the attention data, the object detection results are output, including: Based on the third attention data, output the 3D object detection result; and Based on the fourth attention data, output the two-dimensional object detection result.

6. The method of claim 1, wherein, Based on the first feature, the second feature, the first object query feature, and the second object query feature, attention is calculated to obtain attention data, including: Based on the projection relationship between the 3D data and the image data, the projection feature of the second feature is added to the first feature; and Based on the first feature, the second feature, the first object query feature, and the second object query feature with the added projection features, attention is calculated to obtain attention data.

7. The method of claim 1, wherein, The first object query feature and the second object query feature have the same location embedding added.

8. An object detection device, comprising: The data receiving module is configured to receive 3D data and image data for object detection; The feature calculation module is configured to: calculate a first feature of the three-dimensional data and a second feature of the image data; Calculate the corresponding first object query feature based on the three-dimensional data; Furthermore, based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data, the second object query feature corresponding to the image data is calculated; The object detection module is configured to: calculate attention based on the first feature, the second feature, the first object query feature, and the second object query feature to obtain attention data; And, based on the attention data, output the object detection result; The three-dimensional data is a point cloud, and the image data is an RGB image; The calculation of the corresponding first object query features based on the three-dimensional data includes: Based on the kNN algorithm, multiple suggestion points are sampled from the point cloud; By adding the learned bias based on the features of the multiple suggestion points to the multiple suggestion points, multiple corrected suggestion points are obtained; and Based on the multiple corrected suggestion points, the query features of the first object are calculated; Specifically, based on the three-dimensional data and the projection relationship between the three-dimensional data and the image data, the second object query feature corresponding to the image data is calculated, including: Based on the multiple corrected suggested points and the projection relationship between the 3D data and the image data, the second object query feature is calculated.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 7.

Images