Method, apparatus and system for 3D object detection

Abstract

One aspect of the present invention discloses a 3D object detection method performed by a server. The 3D object detection method comprises the steps of: extracting a point cloud from an image; obtaining an embedding cloud based on the point cloud; using the embedding cloud to extract geometric features of the point cloud and local features of voxels generated by dividing the point cloud; fusing the geometric features and the local features to obtain a feature map; inputting the feature map into a backbone network to obtain high-dimensional features for object detection; and inputting the high-dimensional features into a detection network to detect objects.

Description

Technology Field

[0001] The present invention relates to a 3D object detection method, apparatus, and system, and more specifically, to a method, apparatus, and system for detecting the location, size, and type of a 3D object by fusing geometric features of a point cloud and local features of a voxel. Background Technology

[0002] 3D object detection technology plays a key role in various fields such as autonomous driving, robot vision, and augmented reality. In particular, 3D object detection utilizing data such as point clouds acquired from LiDAR sensors has the advantage of enabling accurate three-dimensional spatial recognition.

[0003] Point cloud-based 3D object detection methods are broadly classified into point-based and voxel-based methods. Point-based methods detect objects by directly utilizing the geometric features of point clouds, allowing for the effective use of positional relationships and structural features between points. On the other hand, voxel-based methods divide 3D space into grids of a fixed size and extract features by integrating points within each grid, offering the advantage of high computational efficiency.

[0004] A representative example of a voxel-based approach is Point Pillars. Point Pillars is a model for high-efficiency real-time processing that performs 2D CNN-based object detection using a pillar structure that projects a point cloud vertically for efficient computation. While this enables efficient processing of 3D data, it has limitations such as information loss during the voxelization process and performance degradation in complex environments.

[0005] Meanwhile, in point-based methods, research is underway to effectively model relationships between points by utilizing attention mechanisms such as Point Transformers. While this enables more precise extraction of geometric features from point clouds, it has the disadvantage of high computational costs when processing large-scale point clouds. Therefore, there is a need for technology that improves object detection speed and accuracy by fusing methods utilizing point clouds and point pillars. Prior art literature

[0006] Korean Published Patent No. 10-2024-0118620 The problem to be solved

[0007] The present invention aims to solve the aforementioned problems by providing a method, apparatus, and system for detecting the location, size, and type of a 3D object by fusing the geometric features of a point cloud and the local features of a voxel. means of solving the problem

[0008] According to one aspect of the present invention, a 3D object detection method performed by a server is provided. The 3D object detection method comprises the steps of: extracting a point cloud from an image; obtaining an embedding cloud based on the point cloud; using the embedding cloud, extracting geometric features of the point cloud and local features of voxels generated by dividing the point cloud; fusing the geometric features and the local features to obtain a feature map; inputting the feature map into a backbone network to obtain high-dimensional features for object detection; and inputting the high-dimensional features into a detection network to detect objects.

[0009] In one aspect, the step of acquiring the feature map comprises: generating a plurality of pillars from the embedding cloud; generating a plurality of local feature maps by extracting a plurality of local features of each of the plurality of pillars; generating a geometric feature map by extracting geometric features of the plurality of pillars; and generating the feature map by fusing the plurality of local feature maps and the geometric feature map.

[0010] In another aspect, the step of generating the plurality of local feature maps includes the step of calculating the center coordinates of the plurality of pillars and calculating the distance between the center coordinates and a point to extract the plurality of local features, and the step of generating the plurality of local feature maps by performing pooling on the extracted plurality of local features.

[0011] In another aspect, the step of generating the geometric feature map comprises the step of extracting the geometric feature containing spatial information identical to the local feature using the index of the point extracted as a result of performing the pooling, and the step of generating the geometric feature map based on the extracted geometric feature.

[0012] In another aspect, the step of generating multiple pillars includes generating P pre-configured pillars from the embedding cloud, wherein the number of points of each of the P pillars is generated to be N.

[0013] In another aspect, each point of the plurality of pillars is a 9-dimensional vector represented as Point D = {x, y, z, r, xc, yc, zc xp, yp}, wherein r is the reflectance rate, xc, yc, and zc are distances from the center coordinates of the point, and xp and yp are distances from the center x, y of the pillar, respectively.

[0014] In another aspect, the aforementioned multiple local feature maps may be feature maps in the form of a 2D Bird's Eye View.

[0015] In another aspect, the step of generating the embedding cloud includes the step of generating the embedding cloud by inputting the point cloud into a point transformer.

[0016] In another aspect, the step of generating the embedding cloud includes the step of initializing the point transformer and the step of performing encoding on the point cloud through the point transformer.

[0017] In another aspect, the point transformer can serialize the point cloud by arbitrarily combining at least two of the Z-order, Hilbert, Trans Z-order, and Trans Hilbert methods.

[0018] In another aspect, the step of acquiring the high-dimensional feature includes extracting a multi-resolution feature map from the feature map through a convolution operation of the backbone network and combining the multi-resolution feature maps to generate the high-dimensional feature.

[0019] In another aspect, the backbone network includes a first sub-network that gradually reduces the spatial resolution of the feature map while increasing the number of channels of the feature map, and a second sub-network that combines the multi-resolution feature maps.

[0020] In another aspect, the step of detecting the object comprises: obtaining a plurality of anchor boxes that predict the location, size, and type of the object from the detection network; removing at least one duplicate anchor box among the anchor boxes using a Non-Maximum Suppression (NMS) algorithm; and obtaining at least one of the remaining anchor boxes from which the duplicate anchor box has been removed, having a probability greater than or equal to a reference, as the detection result.

[0021] In another aspect, prior to extracting the point cloud, the method further includes the step of receiving the image from the image device.

[0022] According to another aspect of the present invention, a 3D object detection device that fuses a point cloud and a voxel is provided. The 3D object detection device that fuses a point cloud and a voxel includes a communication module that receives an image from an image device and a processor that extracts a point cloud from the image, obtains an embedding cloud based on the point cloud, extracts geometric features of the point cloud and local features of voxels generated by dividing the point cloud using the embedding cloud, obtains a feature map by fusing the geometric features and the local features, inputs the feature map into a backbone network to obtain high-dimensional features for object detection, and inputs the high-dimensional features into a detection network to detect an object.

[0023] In one aspect, the processor includes an embedding unit that vectorizes geometric features of the point cloud, a feature map generation unit that generates the feature map by fusing the geometric features and the local features, a backbone that generates the high-dimensional features using the backbone network, and a detection unit that predicts the location, size, and type of the object using the detection network. Effects of the invention

[0024] Embodiments of the present invention may have effects including the following advantages. However, since the embodiments of the present invention are not required to include all of these, the scope of the present invention should not be understood as being limited by them.

[0025] A 3D object detection method, device, and system according to one embodiment of the present invention has the effect of accurately detecting 3D objects even in complex environments.

[0026] In addition, efficient object detection operations can be performed based on a feature map that fuses geometric and local features.

[0027] In addition, the time required for 3D object detection can be reduced. Brief explanation of the drawing

[0028] FIG. 1 is a schematic diagram showing a 3D object detection system according to one embodiment of the present invention. Figure 2 is a diagram showing the structure of a point transformer. Figure 3 is a diagram showing the serialization pattern of a point cloud. Figure 4 is a diagram showing patch grouping of a point cloud. Figure 5 is a diagram showing the embedding of a point cloud. Figure 6 is a diagram showing the generation of a feature map. Figure 7 is a diagram showing a backbone network. Figure 8 is a diagram showing a detection network. Figure 9 is a diagram illustrating the process of extracting a multi-resolution feature map. Figure 10 is a diagram showing an example of a detection area according to the size of a multi-resolution feature map. Figure 11 is a diagram showing the removal of duplicate anchor boxes. Figure 12 is a diagram showing a comparison of the object detection results of the conventional method and the object detection results of the proposed method. FIG. 13 is a flowchart of a 3D object detection method according to an embodiment of the present invention. FIG. 14 is a block diagram of a 3D object detection device according to one embodiment of the present invention. Specific details for implementing the invention

[0029] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail.

[0030] However, this is not intended to limit the invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0031] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0032] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0033] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0035] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0036] FIG. 1 is a schematic diagram showing a 3D object detection system according to one embodiment of the present invention.

[0037] Referring to FIG. 1, the 3D object detection system (100) according to the present embodiment is a system for detecting 3D objects based on a point cloud extracted from an image. In order to efficiently perform calculations on the point cloud, voxelization is performed such as with point pillars to extract local features of the voxels, and fusion with the geometric features of the point cloud can efficiently detect 3D objects. That is, unlike a method that uses only point pillars, the advantages of object detection calculations using the local features of point pillars and object detection calculations using the geometric features of the point cloud are combined to perform more efficient object detection calculations, and based on this, 3D objects can be accurately detected in real time.

[0038] A 3D object detection system (100) that fuses such point clouds and voxels may include an image device (110), an object detection server (120), and a user terminal (130).

[0039] An imaging device (110) is a device that captures images of an object. The imaging device (110a, 110b) can be implemented as a device equipped with a camera, such as a camera, camcorder, CCTV, or smart terminal. In addition, the imaging device (110) may include not only devices capable of capturing general images as described above, but also devices such as LiDAR. That is, a device that performs the operation of emitting a laser pulse and measuring the time it takes for it to be reflected back, thereby acquiring 3D point cloud data as a result, may also be included in the category of the imaging device (110). For the same reason, a 3D scanning device or a distance measuring sensor device may also be included in the category of the imaging device (110).

[0040] The imaging device (110) may capture surrounding objects at an installed location and then transmit the captured images to an object detection server (120), or it may be mounted on a moving body (e.g., a vehicle) to capture objects around a moving path and then transmit the captured images to an object detection server (120). The image transmission operation of the imaging device (110) may be performed periodically or non-periodically, but preferably, it may be operated to transmit to the object detection server (120) immediately after capturing for real-time object detection.

[0041] The object detection server (120) is a device that detects 3D objects by extracting a point cloud from an image. The object detection server (120) can extract a point cloud from an image received from an image device (110). If the image device (110) is a general camera-based device, the object detection server (120) can directly extract the point cloud, and if the image device (110) is a device such as a LiDAR, it can receive the point cloud extracted from the image device. The object detection server (120) can perform four operations—embedding operation, feature map generation operation, high-dimensional feature generation operation, and object detection operation—to detect 3D objects in real time using the point cloud.

[0042] First, the embedding operation may be an operation that learns the structure of the point cloud and the relationship between points to extract geometric features, and then merges them with the location information of the points. Then, the feature map generation operation may be an operation that generates point pillars using embedding data and calculates the center coordinates of the point pillars and the distance between the center coordinates and the points to extract local features. Subsequently, the local features can be formed into a local feature map, and pooling can be applied to the geometric features to perform spatial alignment of the local feature map and the geometric feature map, thereby fusing the geometric features and the local features. Then, the high-dimensional feature generation operation may be an operation that extracts spatial structural information of the point cloud from the feature map through a 2D CNN-based backbone network and extracts local and global features through multi-scale convolution to effectively extract high-dimensional features for object detection. Then, the object detection operation may be an operation that analyzes high-dimensional features through a detection network to predict the location, size, and type of the object. The object detection server (120) can accurately detect 3D objects in real time by sequentially performing these four operations. The object detection server (120) can transmit the object detection result to the user terminal (130).

[0043] The object detection server (120) can be implemented as a large-capacity PC, a server-class PC, a workstation, etc. The embodiments of the object detection server (120) described below are embodiments for the case of a server-class PC, but other types of terminals may also be applied.

[0044] The user terminal (130) is a device that receives the detection results of a 3D object from the object server (120). Depending on the environment in which the system of FIG. 1 is implemented, the imaging device (110) and the user terminal (130) may be implemented as a single device. That is, a camera or LiDAR sensor mounted on the user terminal (130) may be used as the imaging device. The user (administrator) can analyze the state of the space where the image was captured based on the object detection results received from the object detection server (120). For example, it can analyze whether an accident has occurred.

[0045] The user terminal (130) can be implemented as a device such as a mobile terminal, smart terminal, laptop, or PC, and in some cases, can be implemented as a terminal mounted on a means of transportation such as a vehicle. The embodiments of the user terminal (130) described below are embodiments in the case of a smart terminal, but other types of terminals may also be applied.

[0046] Figure 2 is a diagram showing the structure of a point transformer.

[0047] Referring to FIG. 2, the object detection server (120) can embed a point cloud extracted from an image using a Point Transformer. That is, the point cloud can be input into the Point Transformer to generate an embedding cloud. The example in FIG. 2 shows the architecture of a Point Transformer V3, and the object detection server (120) can vectorize the geometric structure of the point cloud using this Point Transformer. For example, the structure and geometric information of the point cloud can be extracted, the point cloud data can be serialized, and the structure and geometric features of the point cloud can be effectively and efficiently extracted using a serialized attention mechanism.

[0048] Point transformers can be designed to effectively handle the misalignment, density variations, and geometric relationships of point clouds. Point transformers can perform initialization and encoding phases. Point transformers can learn relative positional information between points and learn the relationship between a reference point and adjacent points based on a self-attention mechanism. In other words, for computational efficiency, the point cloud can be aligned in the initial phase to learn the relationship between the reference point and adjacent points based on patches.

[0049] Figure 3 is a diagram showing the serialization pattern of a point cloud.

[0050] Referring to FIG. 3, the point transformer can serialize a point cloud by arbitrarily combining at least two of the Z-order, Hilbert, Trans Z-order, and Trans Hilbert methods during the initialization phase. That is, it can align and structure an unstructured initial point cloud. After the initialization phase, the point transformer can perform an encoding phase to arrange the point cloud in an aligned order.

[0051] Figure 4 is a diagram showing patch grouping of a point cloud.

[0052] Referring to Fig. 4, the point transformer can process (divide) a serialized point cloud (which can be called a point sequence since it is serialized) into patches of a fixed size. It can also group the patches and change their positions to expand the information exchange and reception area between patches. Furthermore, to improve generalization capabilities, the serialization pattern order can be randomly changed, and a sparse convolution layer can be used to efficiently calculate the geometric structure of the point cloud and the relative positions between points. In other words, it can perform reordering and padding on the patches.

[0053] Referring again to FIG. 3, the encoder stage of the point transformer can improve learning stability and efficiency compared to the existing point transformer by simplifying the block structure and using a Pre-Norm structure, and computational efficiency can also be improved by using grid pooling. During the process of extracting geometric features, the point transformer performs grid pooling equal to the size of the pillar, thereby making the spatial resolution and data structure of geometric feature information and local feature information identical. As a result, geometric information and local information of the pillar can be effectively combined by linking feature information.

[0054] Figure 5 is a diagram showing the embedding of a point cloud.

[0055] Referring to FIG. 5, the object detection server (120) can obtain an embedding cloud by embedding the geometric structure of a point cloud using a point transformer. For example, an nx 4 point cloud and an nx G geometric embedding feature can be combined to create an nx (4+G) form. That is, an embedding cloud can be generated by combining the embedded geometric feature information and the location information of the point cloud.

[0056] Figure 6 is a diagram showing the generation of a feature map.

[0057] Referring to FIG. 6, the object detection server (120) can use an embedding cloud to extract geometric features of a point cloud and local features of a voxel generated by dividing the point cloud. Then, a feature map can be obtained by fusing the geometric features and the local features. For example, after generating multiple pillars from the embedding cloud, multiple local features of each of the multiple pillars can be extracted to generate multiple local feature maps. Then, geometric features of the multiple pillars can be extracted to generate a geometric feature map, and the multiple local feature maps and the geometric feature map can be fused to generate the feature map.

[0058] The object detection server (120) can generate P Pillars containing geometric features from the embedding cloud. P may be a pre-set number, and the object detection server (120) can generate them by limiting the number of points of each of the P Pillars to N. The object detection server (120) can divide the P Pillars into a Pillar containing local features (P, N, D) and a Pillar containing geometric features (P, N, G). Each point of the multiple Pillars for generating a local feature map is a 9-dimensional vector and can be represented as Point D = {x, y, z, r, xc, yc, zc xp, yp}. Here, r is the reflectance rate, xc, yc, and zc are distances from the center coordinates of the point, and xp and yp represent distances from the center x, y of the pillar, respectively. The object detection server (120) can calculate the center coordinates of multiple pillars and calculate the distance between the center coordinates and the point to extract the multiple local features.

[0059] The object detection server (120) can generate a tensor of size (P, N, D) by limiting the number of non-empty pillars to P as previously mentioned, while limiting the number of points per pillar to N. That is, if a pillar has too many points, random sampling is performed, and if it has too few, zero padding is performed to keep the number of points constant. Afterwards, a tensor of size (P, N, C) can be generated by applying a linear layer, batch normalization, and ReLU through PointNet to each point. Then, a local feature map (P, C) in the form of a 2D bird's eye view can be generated through Max Pooling. That is, multiple local feature maps can be generated by performing pooling on multiple extracted local features. The object detection server (120) can generate a geometric feature map (P, C+G) by extracting geometric features that have spatial information such as local features using the indices of points extracted during the pooling process. Finally, it can generate a feature map (C+G, H, W) by fusing the two features.

[0060] In this way, prior to extracting high-dimensional features, an effective feature map including global, local, and individual features can be obtained by combining local features derived from the positional information of each of multiple pillars with the geometric structure of the point cloud and the relationships between the points into a feature map. This feature map can be used to detect objects in complex environments and can complement the features of pillars that become non-uniform and sparse due to voxelization.

[0061] Figure 7 is a diagram showing a backbone network.

[0062] Referring to FIG. 7, the object detection server (120) can input a feature map into a backbone network to obtain high-dimensional features for object detection. The object detection server (120) can extract a multi-resolution (scale) feature map from the feature map through convolution operations of the backbone network and combine the multi-resolution feature maps to generate high-dimensional features.

[0063] As illustrated in FIG. 7, the backbone network may include a first sub-network that gradually reduces the spatial resolution of the feature map while increasing the number of channels of the feature map, and a second sub-network that combines multiple resolution feature maps. The first sub-network may be a Top-Down Network, and the second sub-network may be a Feature Pyramid Network (FPN).

[0064] The first sub-network is composed of convolution layers for feature extraction and can increase the number of channels of the feature map while gradually reducing the spatial resolution. The second sub-network can detect objects of various scales by combining multi-resolution feature maps. That is, the object detection server (120) can extract multi-resolution feature maps from the feature map through the convolution operation of the first sub-network and can generate high-dimensional features by combining multi-resolution feature maps through the operation of the second sub-network.

[0065] For example, three blocks can be generated to extract high-dimensional features from local-geometric feature maps (C+G, H, W). The convolution operation used to generate each block can be defined as Block(S, L, F). Here, S represents the stride size, L represents the number of 3 x 3 2D convolution layers, and F represents the size of the output channels. Batch Normalization (BN) and the ReLU function can be applied after every 2D convolution operation performed within Block(S, L, F). The sizes of the generated blocks are (C+G, H / 2, W / 2), (2(C+G), H / 4, W / 4), and (4(C+G), H / 8, W / 8), respectively, and all can be up-sampled into blocks of size (2(C+G), H / 2, W / 2) by applying Transposed 2D Convolution to the blocks. In this case, the upsampling operation can be defined as Up(Sin, Sout, F), where Sin and Sout represent the initial stride and final stride, respectively, and F represents the size of the final output channel. The three blocks can be concatenated to form one large tensor. The size of the concatenated tensor is (6(C+G), H / 2, W / 2).

[0066] Figure 8 is a diagram showing a detection network.

[0067] Referring to FIG. 8, the object detection server (120) can detect an object by inputting high-dimensional features into a detection network. The object detection server (120) can obtain multiple anchor boxes that predict the location, size, and type of the object from the detection network, and can obtain the anchor box with the highest probability of being an object among the remaining anchor boxes as a detection result after removing at least one duplicate anchor box among the anchor boxes using a Non-Maximum Suppression (NMS) algorithm. The object detection server (120) can pre-set a threshold value for the probability and can obtain at least one anchor box among the anchor boxes from which duplicates have been removed that has an accuracy greater than or equal to the threshold value as a detection result.

[0068] Detection networks can be referred to as detectors. Detectors can be broadly classified into two types: 2-stage detectors and 1-stage detectors. 2-stage detectors identify areas within an image where objects are presumed to exist based on similar colors or textures, such as region proposals. While they demonstrate high accuracy by providing the model with various views, they have the disadvantage of slow object detection speeds due to the significant time required to extract and process these region proposals. On the other hand, 1-stage detectors like YOLO v1 process the entire original image using a unified network, resulting in very fast object detection speeds. However, accuracy suffers because they provide the model with relatively fewer views by selecting only two bounding boxes per grid cell. As such, accuracy and speed in object detection generally have a trade-off relationship.

[0069] Referring again to FIG. 8, the detection network can be implemented as a Single Shot Detector (SSD) model. The SSD model is a 1-stage detector with an integrated network structure that utilizes various views, and can achieve high accuracy and high speed.

[0070] As shown in Fig. 8, the SSD model has a structure in which a pretrained VGG16 is used as the base network and an auxiliary network is added. The auxiliary network consists of general convolutional layers, and the fc layer appearing in the latter part of the base network can be replaced with a convolutional layer and connected to the auxiliary network. In other words, the object detection speed can be improved by replacing the fc layer with a conv layer during the process of connecting the two networks. The SSD model includes feature maps obtained from the convolutional layer in the middle of the convolutional network, allowing a total of six feature maps of different scales (resolutions) to be used for prediction. Additionally, the location of an object can be estimated using a default box, which is a bounding box with a different scale and aspect ratio for each cell of the feature map.

[0071] The core idea of ​​the SSD model is to use feature maps of various scales. Existing models use single-scale feature maps passed through a convolutional network for object detection. For example, the YOLO v1 model uses only 7x7 (x30) feature maps for object detection. Using only single-scale feature maps in this way can make it difficult to detect objects of various sizes. Therefore, this drawback can be overcome by extracting feature maps from the intermediate convolutional layers of the SSD model and using them for object detection.

[0072] Figure 9 is a diagram illustrating the process of extracting a multi-resolution feature map.

[0073] For example, when the input image size is 300x300, a feature map of size 38x38x(512) can be extracted from the conv4_3 layer of the base network. Then, a feature map of size 19x19x(1024) can be extracted from the conv7 layer of the base network. Next, feature maps of sizes 10x10x(512), 5x5x(256), 3x3x(256), and 1x1x(256) can be extracted from the conv8_2, conv9_2, conv10_2, and conv11_2 layers of the auxiliary network, respectively. Through this process, a total of six multi-resolution feature maps with different resolutions can be obtained. Therefore, it is possible to detect objects of a wider variety of sizes by using feature maps of various scales.

[0074] Figure 10 is a diagram showing an example of a detection area according to the size of a multi-resolution feature map.

[0075] Referring to Fig. 10, to detect objects of various sizes, a default box with a different scale and aspect ratio is generated for each cell of the feature map. The default box is conceptually similar to the anchor box used in the Faster R-CNN model, but differs in that it is applied to feature maps of different sizes. The SSD model can generate a default box for each cell of a total of six multi-resolution feature maps: 38x38, 19x19, 10x10, 5x5, 3x3, and 1x1. Here, the scale sk of the default box can be a ratio to the original image. For example, when the size of the original image is 300x300, s=0.1, and the aspect ratio is 1:1, the size of the default box can be 30x30 (=300x0.1 x 300x0.1). In other words, as the scale of the feature map decreases, the scale of the default box can increase, which means that larger objects can be detected as the size of the feature map decreases.

[0076] Each feature map applies a different number of default boxes. The first (38x38) and last (1x1) feature maps apply a total of 4 default boxes, including boxes with aspect ratios of 1:1, 1:2, and 1:1 / 2, as well as an additional box used when the aspect ratio is 1:1; this means that 4 default boxes are generated for each cell of the feature map. All 6 default boxes mentioned above can be applied to the remaining 4 feature maps. When default boxes are applied to feature maps of different scales, the total number of default boxes can be 8732 (= 38x38x4 + 19x19x6 + 10x10x6 + 5x5x6 + 3x3x6 + 1x1x4).

[0077] The detection network can apply a 3x3 (stride=1, padding=1) convolution operation after extracting feature maps of different scales for final prediction. In this case, when the number of default boxes is k and the number of classes to be predicted is c, the number of channels of the output feature map is designed to be k(4+c) so that k default boxes are generated in each cell of the feature map, and for each box, 4 offsets and class scores can be predicted.

[0078] Figure 11 is a diagram showing the removal of duplicate anchor boxes.

[0079] Referring to Fig. 11, when detecting an object, the detection network can remove at least one duplicate anchor box among multiple anchor boxes that predict the location, size, and type of the object by applying a Non-Maximum Suppression (NMS) algorithm to the last prediction. If the detection network displays all anchor boxes, the accuracy of object detection may decrease because an excessive number of anchor boxes may overlap for a single object. To solve this problem, the NMS algorithm can be used to remove at least one anchor box in a similar location among the detected anchor boxes and select the most suitable anchor box (one with an accuracy above a certain threshold).

[0080] This operation can be performed in the same way when using bounding boxes as well as anchor boxes. As shown in the example in Fig. 11, the NMS Non-maximum suppression algorithm can first generate multiple overlapping bounding boxes for a single object and then remove duplicate bounding boxes so that only one bounding box remains per object. In this case, more bounding boxes can be removed as the confidence score threshold is higher and the iou threshold is lower. Through this process, overlapping bounding boxes can be removed and only the minimum optimal bounding box can be returned.

[0081] Figure 12 is a diagram showing a comparison of the object detection results of the conventional method and the object detection results of the proposed method.

[0082] Referring to FIG. 12, the detection results of a conventional point pillar-based object recognition algorithm and an object detection algorithm according to the present invention are compared for each of two images. The left side shows the detection results of the conventional algorithm, and the right side shows the detection results of the algorithm according to the present invention. A total of 89,828 LiDAR images were used for the dataset for training and verification of object detection, which includes approximately 7,500 KITTI data. The dataset consists of Testing, training GT, training, and image set.

[0083] Even when detecting 3D objects from images of complex environments as shown in Fig. 12, the object detection algorithm according to the present invention effectively processes sparse or irregularly distributed points, achieving an increase in mAP of 2.13% compared to existing algorithms, and in particular, achieved a recognition rate of about 90% in the case of automobiles.

[0084] FIG. 13 is a flowchart of a 3D object detection method according to an embodiment of the present invention.

[0085] Referring to FIG. 13, the 3D object detection method includes steps S1310 through S1350. Steps S1310 through S1350 are described as operations performed by the object detection server (120) described through FIG. 1.

[0086] In step S1310, the object detection server (120) receives an image from the image device.

[0087] In step S1320, the object detection server (120) extracts a point cloud from the image and obtains an embedding cloud based on the point cloud.

[0088] In step S1330, the object detection server (120) uses an embedding cloud to extract geometric features of the point cloud and local features of the voxels generated by dividing the point cloud, and then fuses the geometric features and local features to obtain a feature map.

[0089] In step S1340, the object detection server (120) inputs the feature map into the backbone network to obtain high-dimensional features for object detection.

[0090] In step S1350, the object detection server (120) inputs high-dimensional features into the detection network to detect objects.

[0091] FIG. 14 is a block diagram of a 3D object detection device according to one embodiment of the present invention.

[0092] Referring to FIG. 14, the 3D object detection device (1400) includes a communication module (1410), a memory (1420), and a processor (1430). The 3D object detection device (1400) that fuses a point cloud and a voxel can be implemented as an object detection server (120) described through FIG. 1.

[0093] The communication module (1410) receives an image from the image device (110). The communication module (1410) transmits the object detection result of the processor (1430) to the user terminal (130). The communication module (1410) may be implemented as a modem or module capable of communicating with the image device (110) or the user terminal (130) of FIG. 1.

[0094] The memory (1420) stores a set of instructions for 3D object detection. The memory (1420) can be implemented as a memory device such as RAM, ROM, etc.

[0095] The processor (1430) extracts a point cloud from an image. Then, it obtains an embedding cloud based on the point cloud. Then, using the embedding cloud, it extracts geometric features of the point cloud and local features of the voxels generated by segmenting the point cloud. Then, it obtains a feature map by fusing the geometric features and local features, and inputs the feature map into a backbone network to obtain high-dimensional features for object detection. Then, it inputs the high-dimensional features into a detection network to detect objects. The processor (1430) can be implemented as a CPU, GPU, or AP.

[0096] Meanwhile, the processor (1430) can be implemented with an embedding unit (1430a), a feature map generation unit (1430b), a back unit (1430c), and a detection unit (1430d) depending on its function.

[0097] The embedding unit (1430a) vectorizes the geometric features of the point cloud. The point cloud can be input into the point transformer of the embedding unit (1430a) to generate an embedding cloud.

[0098] The feature map generation unit (1430b) generates a feature map by fusing geometric features and local features. The feature map generation unit (1430b) can generate a feature map in which geometric features and local features are fused using PointNet.

[0099] The backbone headquarters (1430c) generates high-dimensional features using a backbone network. The backbone headquarters (1430c) can generate high-dimensional features by sequentially using a backbone network including two sub-networks to generate multi-resolution feature maps, and then combining the multi-resolution feature maps.

[0100] Detection (1430d) predicts the location, size, and type of an object using a detection network. Detection (1430d) is based on a pre-trained VGG16, but an SSD model connected to an auxiliary network with the fc layer changed to a conv layer can be used as the detection network.

[0101] Meanwhile, the 3D object detection device (1400) that fuses the point cloud and voxel described in FIG. 14 may be implemented as a program (or application) that includes an algorithm executable on a computer.

[0102] Although the invention has been described with reference to the drawings and embodiments, this does not imply that the scope of protection of the present invention is limited by the drawings or embodiments. Those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the invention as described in the following claims.

Claims

Claim 1 A 3D object detection method performed by a server, comprising: a step of extracting a point cloud from an image; a step of obtaining an embedding cloud based on the point cloud; a step of extracting geometric features of the point cloud and local features of voxels generated by dividing the point cloud using the embedding cloud; a step of obtaining a feature map by fusing the geometric features and the local features; a step of obtaining high-dimensional features for object detection by inputting the feature map into a backbone network; and a step of detecting an object by inputting the high-dimensional features into a detection network. Claim 2 A 3D object detection method according to claim 1, wherein the step of acquiring the feature map comprises: generating a plurality of pillars from the embedding cloud; generating a plurality of local feature maps by extracting a plurality of local features of each of the plurality of pillars; generating a geometric feature map by extracting geometric features of the plurality of pillars; and generating the feature map by fusing the plurality of local feature maps and the geometric feature map. Claim 3 A 3D object detection method according to claim 2, wherein the step of generating the plurality of local feature maps comprises: a step of calculating the center coordinates of the plurality of pillars and calculating the distance between the center coordinates and a point to extract the plurality of local features; and a step of generating the plurality of local feature maps by performing pooling on the extracted plurality of local features. Claim 4 A 3D object detection method according to claim 3, wherein the step of generating the geometric feature map comprises: a step of extracting the geometric feature containing spatial information identical to the local feature using the index of the point extracted as a result of performing the pooling; and a step of generating the geometric feature map based on the extracted geometric feature. Claim 5 A 3D object detection method according to claim 2, wherein the step of generating a plurality of pillars includes generating P pillars pre-set from the embedding cloud, and generating the number of points of each of the P pillars to be N. Claim 6 A 3D object detection method according to claim 2, wherein each point of the plurality of pillars is a 9-dimensional vector represented by Point D = {x, y, z, r, xc, yc, zc xp, yp}, wherein r is the reflectance rate, xc, yc, and zc are distances from the center coordinates of the point, and xp and yp are distances from the center x, y of the pillar, respectively. Claim 7 A 3D object detection method according to claim 2, wherein the plurality of local feature maps are feature maps in the form of a 2D Bird's Eye View. Claim 8 A 3D object detection method according to claim 1, wherein the step of generating the embedding cloud includes the step of generating the embedding cloud by inputting the point cloud into a point transformer. Claim 9 A 3D object detection method according to claim 8, wherein the step of generating the embedding cloud comprises: the step of initializing the point transformer; and the step of performing encoding for the point cloud through the point transformer. Claim 10 A 3D object detection method according to claim 8, wherein the point transformer serializes the point cloud by arbitrarily combining at least two of Z-order, Hilbert, Trans Z-order, and Trans Hilbert methods. Claim 11 A 3D object detection method according to claim 1, wherein the step of acquiring the high-dimensional feature comprises the step of extracting a multi-resolution feature map from the feature map through a convolution operation of the backbone network and combining the multi-resolution feature map to generate the high-dimensional feature. Claim 12 A 3D object detection method according to claim 11, wherein the backbone network comprises a first sub-network that gradually reduces the spatial resolution of the feature map while increasing the number of channels of the feature map, and a second sub-network that combines the multi-resolution feature maps. Claim 13 A 3D object detection method according to claim 1, wherein the step of detecting the object comprises: obtaining a plurality of anchor boxes that predict the location, size, and type of the object from the detection network; removing at least one duplicate anchor box among the anchor boxes using a Non-Maximum Suppression (NMS) algorithm; and obtaining at least one of the remaining anchor boxes from which the duplicate anchor box has been removed, wherein the probability is greater than or equal to a reference, as the detection result. Claim 14 A 3D object detection method according to claim 1, further comprising the step of receiving the image from an imaging device prior to extracting the point cloud. Claim 15 A 3D object detection device comprising: a communication module that receives an image from an image device; and a processor that extracts a point cloud from the image, obtains an embedding cloud based on the point cloud, extracts geometric features of the point cloud and local features of voxels generated by dividing the point cloud using the embedding cloud, obtains a feature map by fusing the geometric features and the local features, inputs the feature map into a backbone network to obtain high-dimensional features for object detection, and inputs the high-dimensional features into a detection network to detect an object. Claim 16 In claim 15, the processor comprises: an embedding unit that vectorizes geometric features of the point cloud; a feature map generation unit that generates the feature map by fusing the geometric features and the local features; a backbone unit that generates the high-dimensional features using the backbone network; and a detection unit that predicts the location, size, and type of the object using the detection network, a 3D object detection device.

Images