Method and device for detecting objects in point clouds that can reduce computation and memory usage

Abstract

A method for detecting objects in a point cloud performed by a processor is disclosed. The method for detecting objects in a point cloud includes the steps of generating a feature map from a 3D point cloud, performing a convolution operation on the feature map to output a final feature map, and detecting objects in the final feature map.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for object detection in a point cloud, and more specifically, to a method and apparatus for object detection in a point cloud capable of reducing computation and memory usage. Background Technology

[0002] LiDAR-based object detection technology is used in the field of autonomous driving to recognize objects in the surrounding environment in real time and to estimate their location and size. To this end, processing is required to accurately extract information regarding the presence and boundaries of objects from point clouds. However, since point clouds have irregular distributions and low density characteristics, an effective feature extraction method is essential. Prior art literature

[0003] Korean Registered Patent Publication No. 10-244672 (September 14, 2022) The problem to be solved

[0004] The technical problem that the present invention aims to solve is to provide a method and apparatus for object detection in a point cloud that can reduce computation and memory usage. means of solving the problem

[0005] An object detection method in a point cloud performed by a processor according to an embodiment of the present invention includes the steps of generating a feature map from a 3D point cloud, performing a convolution operation on the feature map to output a final feature map, and detecting an object in the final feature map.

[0006] The step of outputting the final feature map comprises: a step of performing a first convolution operation on the feature map to output a first feature map having the same scale as the feature map; a step of performing a second convolution operation on the first feature map to output a second feature map having a scale smaller than the scale of the first feature map; a step of performing a third convolution operation on the second feature map to output a third feature map having a scale smaller than the scale of the second feature map; a step of performing a fourth convolution operation on the third feature map to output a fourth feature map having a scale smaller than the scale of the third feature map; a step of performing a fifth convolution operation on the second feature map to output a fifth feature map; a step of upscaling the third feature map to generate a sixth feature map; a step of upscaling the fourth feature map to generate a seventh feature map; and the fifth feature map, It includes the step of generating the final feature map by connecting the sixth feature map and the seventh feature map.

[0007] The scale of the above-mentioned 6th feature map and the scale of the above-mentioned 7th feature map are the same.

[0008] In the step of detecting objects in the final feature map above, the objects are detected using a neural network, a heatmap is used to detect the objects, and an elliptical Gaussian heatmap is used as a training image to train the heatmap.

[0009] A computing device according to an embodiment of the present invention includes a processor that executes object detection commands in a point cloud, and a memory that stores said commands.

[0010] The above commands are implemented to generate a feature map from a 3D point cloud, perform a convolution operation on the feature map to output a final feature map, and detect an object in the final feature map.

[0011] The commands for outputting the final feature map described above perform a first convolution operation on the feature map to output a first feature map having the same scale as the feature map, perform a second convolution operation on the first feature map to output a second feature map having a scale smaller than the scale of the first feature map, perform a third convolution operation on the second feature map to output a third feature map having a scale smaller than the scale of the second feature map, perform a fourth convolution operation on the third feature map to output a fourth feature map having a scale smaller than the scale of the third feature map, perform a fifth convolution operation on the second feature map to output a fifth feature map, upscale the third feature map to generate a sixth feature map, upscale the fourth feature map to generate a seventh feature map, and the fifth feature map, the sixth feature map, and the It is implemented to generate the above final feature map by connecting the 7th feature map.

[0012] The above-mentioned fifth feature map, the above-mentioned sixth feature map, and the above-mentioned seventh feature map are independent of each other.

[0013] The commands for detecting objects in the above final feature map are implemented such that the object is detected using a neural network, a heatmap is used to detect the object, and an elliptical Gaussian heatmap is used as a training image to train the heatmap. Effects of the invention

[0014] The object detection method and apparatus in a point cloud according to an embodiment of the present invention have the effect of improving learning efficiency, as feature maps having different resolutions can be directly upsampled or, through a refinement process, each of the feature maps having different resolutions can be independently trained to emphasize the characteristics of a specific class.

[0015] In addition, the object detection method and device in a point cloud according to an embodiment of the present invention have the effect of reducing the amount of computation by eliminating unnecessary computations.

[0016] Therefore, the present invention eliminates unnecessary operations, thereby reducing the amount of computation and improving learning efficiency. Brief explanation of the drawing

[0017] Detailed descriptions of each drawing are provided to help to more fully understand the drawings cited in the detailed description of the present invention. Figure 1 shows a block diagram of a lidar system according to an embodiment of the present invention. FIG. 2 shows a structural diagram for detecting objects in a point cloud according to an embodiment of the present invention. Figure 3 shows a detailed structural diagram of the pillar extraction layer illustrated in Figure 2. FIG. 4 shows a flowchart for explaining an object detection method in a point cloud according to an embodiment of the present invention. Figure 5 shows a flowchart for explaining in detail the operation of outputting a feature vector for each of the fillers illustrated in Figure 4. Figure 6 shows a block diagram for explaining the operation of the backbone illustrated in Figure 2. Figure 7 shows training images to explain the operation of the detector head illustrated in Figure 2. Specific details for implementing the invention

[0018] Specific structural or functional descriptions regarding embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0019] Embodiments 3 according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0020] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0021] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0022] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0023] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0024] Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the attached drawings.

[0025] Figure 1 shows a block diagram of a lidar system according to an embodiment of the present invention.

[0026] Referring to FIG. 1, the lidar system (100) means a system capable of detecting an object (30) and tracking the object (30). The object (30) is tracked by a lidar sensor (21) to prevent accidents such as collisions with a moving vehicle (20).

[0027] The system (100) includes a lidar sensor (21) and a computing device (10).

[0028] A LiDAR sensor (21) is installed on a moving vehicle (20). The vehicle (20) may be an autonomous vehicle. According to an embodiment, the LiDAR sensor (21) may be installed on fixed infrastructure such as a pole or a tower. The LiDAR sensor (21) emits laser light from the environment toward an object (e.g., vehicle (30)) to generate a 3D point cloud (25). The 3D point cloud (25) includes a set of LiDAR points. The set of LiDAR points may be referred to as LiDAR point data (25). The set of LiDAR points includes a plurality of LiDAR points. In FIG. 1, reference numeral 23 indicates a LiDAR point. A LiDAR point (23) may be expressed in (x, y, z) 3D coordinates. In this specification, LiDAR point, LiDAR point data, point cloud, 3D point cloud, 3D LiDAR point cloud, or LiDAR point cloud may be understood as the same concept. The lidar sensor (21) can be used to detect the vehicle (30) and track the movement of the vehicle (30) for the safety of the autonomous vehicle (20).

[0029] The vehicles (20, 30) drive on the road (101).

[0030] The computing device (10) may be implemented as a hardware module combined with other hardware inside the vehicle (20), or as an independent hardware device. For example, the computing device (10) may be implemented in the Electronic Control Unit (ECU) of the vehicle (20). Additionally, the computing device (10) may be implemented as an external electronic device such as a computer, laptop, PC, server, or tablet PC.

[0031] A computing device (10) includes a processor (11) and a memory (13). The processor (11) executes object detection commands in a point cloud. The memory (13) stores the commands. The operations of the computing device (10) can be understood as the operations of the processor (11).

[0032] The processor (11) receives a 3D point cloud (25) from the lidar sensor (21). Each lidar point included in the 3D point cloud (25) can be represented by 3D coordinates (x, y, z). Additionally, each lidar point can have an intensity.

[0033] FIG. 2 shows a structural diagram for detecting objects in a point cloud according to an embodiment of the present invention.

[0034] Referring to FIGS. 1 and FIGS. 2, the processor (11) receives a point cloud (201) generated by a LiDAR sensor (21). The point cloud (201) is a set of data points in space representing the surface of an object.

[0035] The processor (11) divides the XY plane in the 3D point cloud (201) into multiple grids to form vertical columns along the Z-axis. The vertical columns may be referred to as pillars. In FIG. 2, a vertical filter (203) is shown for illustrative purposes.

[0036] The processor (11) applies each of the vertical columns to the encoder (300) to output a feature vector (330) for each of the vertical columns. The feature vector (330) for each of the vertical columns is rearranged to generate a 2D pseudo-image. The 2D pseudo-image refers to a feature map. In the present invention, the 2D pseudo-image is constructed by rearranging the feature vectors extracted for each vertical column onto a grid in the XY plane. The 2D pseudo-image may be referred to as a BEV (Bird's Eye View) feature map. The 2D pseudo-image may be converted into a BEV.

[0037] The processor (11) applies a 2D similar image to the backbone (207) and downsamples the 2D similar image multiple times to obtain feature maps of various resolutions, and then upsamples the multiple feature maps to the same size after downsampling to generate a final feature map through splicing. The backbone (207) may be a 2D CNN.

[0038] The processor (11) applies the final feature map output from the backbone (207) to the detection head (209) to estimate objects in the point cloud (211).

[0039] The specific operations of the encoder (300) are described below.

[0040] The encoder (300) includes a multi-layer perceptron (301) and a filler extraction layer (310).

[0041] The processor (11) samples a random number (e.g., 20) of LiDAR points included in each of the vertical columns, and then applies a first neural network to each of the sampled LiDAR points to generate k-dimensional (k is a natural number) point feature vectors. The k-dimensional refers to the feature dimension. This is point-wise feature extraction. The first neural network is a multi-layer perceptron (301).

[0042] The processor (11) applies k-dimensional point feature vectors output from the multi-layer perceptron (301) to the filler extraction layer (310) to output a feature vector (330). The feature vector (330) represents a k-dimensional feature for a single filler.

[0043] Figure 3 shows a detailed structural diagram of the pillar extraction layer illustrated in Figure 2.

[0044] Referring to FIGS. 1 to 3, n k-dimensional point feature vectors (311) are generated from a multi-layer perceptron (301). The n, or the k, is a natural number. The n represents the number of sampled LiDAR points. For example, n can be 20.

[0045] The processor (11) divides n k-dimensional point feature vectors (311) into M groups. M is a natural number. For example, when M is 4, the n k-dimensional point feature vectors (311) may include a plurality of k / 4-dimensional sub-point feature vectors (311-1, 311-2, 311-3, and 311-4). The n k-dimensional point feature vectors (311) may be divided into 4 groups (311-1, 311-2, 311-3, and 311-4). For example, one k-dimensional point feature vector (312) may be divided into 4 sub-point feature groups (312-1, 312-2, 312-3, and 312-4).

[0046] The processor (11) applies n k / M dimension point feature vectors (311-1, 311-2, 311-3, and 311-4) divided into M groups to a convolution block (313) to output n compressed feature vectors (315) divided into M groups. The convolution block (313) includes a plurality of shared 1 x 1 convolution layers. Sharing means that the plurality of 1 x 1 convolution layers include the same weights. The number of compressed feature vectors (315) is n.

[0047] The processor (11) applies n compressed feature vectors (315) to a softmax function (317) to output attention scores (319) for the n compressed feature vectors (315).

[0048] The attention scores (319) include multiple attention scores (319-1, 319-2, 319-3, 319-4) for M (e.g., 4) groups. The sum of the attention scores (319-1) is 1. The multiple attention scores (319-1, 319-2, 319-3, 319-4) may have different values.

[0049] One k-dimensional point feature vector (312) can be divided into four sub-point feature groups (312-1, 312-2, 312-3, and 312-4).

[0050] Since the four sub-point feature groups (311-1, 311-2, 311-3, and 311-4) perform a weighted sum with multiple attention scores (319-1, 319-2, 319-3, 319-4) having different values, a feature vector (330) can be extracted by considering multiple perspectives even when using only one filler extraction layer (310). Multiple perspectives refer to different attention scores (319-1, 319-2, 319-3, 319-4). One feature vector (330) is extracted for one filler. One feature vector (330) has k dimensions.

[0051] The processor (11) generates group-wise weighted sums (321, 323, 325, 327) for each of n k / M dimensional point feature vectors (311-1, 311-2, 311-3, and 311-4) divided into M groups and for each of multiple attention scores (319-1, 319-2, 319-3, 319-4) divided into M groups. For example, a weighted sum can be performed on the k / M dimensional point feature vector (311-1) with the attention scores (319-1). A weighted sum is performed to generate feature vectors (321, 323, 325, and 327). The weighted sums (321, 323, 325, 327) refer to the feature vectors (321, 323, 325, and 327) shown in FIG. 3.

[0052] The processor (11) performs a concatenation operation to combine weighted sums (321, 323, 325, 327) to output a single feature vector (330).

[0053] According to an embodiment, the processor (11) can perform cross-attention using a single feature vector (330) corresponding to a specific vertical column as a query and a set of column-specific feature vectors generated for all vertical columns as a key and value, thereby allowing global context information contained in the set of column-specific feature vectors to be reflected in a single feature vector (330). That is, an updated feature vector (330) that takes global context information into account is output.

[0054] After generating point feature vectors by applying each of the LiDAR points included in each of the vertical columns to the first neural network, a set of column-specific feature vectors can be generated by performing pooling, max pooling, or mean pooling operations on the point feature vectors generated for each vertical column.

[0055] The above global context information refers to feature information that includes interrelated spatial information, such as the location, arrangement, and structural relationships of objects, across a wide spatial range, such as an entire vertical column, rather than being limited to local areas like individual points or individual vertical columns within a point cloud or feature map.

[0056] Additionally, the processor (11) performs cross-attention using a set of point feature vectors generated by applying a first neural network to each of the LiDAR points sampled from the entire point cloud (201) as a query and as a key and value, using one feature vector (330) as a query.

[0057] The above cross attention uses a single feature vector (330) as a query and applies a first neural network to each of the LiDAR points sampled by an arbitrary number from the entire point cloud to generate a set of point feature vectors as a key and value, thereby performing a reverse reference so that point-unit fine shape information is reflected in the single feature vector (330). That is, an updated feature vector (330) with fine shape information reflected is output.

[0058] Micro-geometry information refers to feature information representing precise geometric characteristics, such as the detailed shape, boundaries, curvature, and surface structure of individual points or local areas.

[0059] The above cross-attention consists of multiple heads. The first head outputs an updated feature vector (330) that takes into account global context information, and the second head outputs an updated feature vector (330) that reflects fine shape information. The outputs of each head can be sequentially connected or weighted averaged and integrated to generate a final fused feature vector (not shown).

[0060] The processor (11) detects objects in the 3D point cloud (211) using the final fused feature vector. That is, instead of the feature vector (330) in the backbone (207), the final fused feature vector can be converted into a BEV image and input into the backbone (207).

[0061] The bidirectional reference structure of the present invention improves the accuracy of object identification in boundary areas and prevents the omission of small objects by mutually reflecting point-unit fine shape information and column (or pillar)-unit global context information.

[0062] In addition, unlike conventional unidirectional attention structures, the present invention allows global context information and fine shape information to be integrated in a balanced manner by having one feature vector (330) refer to point-unit detailed information and simultaneously having one feature vector (330) refer to column-unit global context.

[0063] Through such a bidirectional reference structure, complex spatial relationships in point cloud data can be effectively modeled, while significantly reducing computational load compared to fully performing global self-attention, thereby improving learning efficiency and inference speed.

[0064] FIG. 4 shows a flowchart for explaining an object detection method in a point cloud according to an embodiment of the present invention.

[0065] Referring to FIGS. 1 to 5, the processor (11) receives a 3D point cloud (25) from the lidar sensor (21) (S100).

[0066] The processor (11) divides the XY plane in the 3D point cloud (201) into multiple grids to generate pillars (S200).

[0067] The processor (11) applies each of the above fillers to the encoder (300) and outputs a feature vector (330) for each of the fillers (S300).

[0068] The processor (11) generates a final feature map by applying the feature vector (330) to the backbone (207) (S400). According to an embodiment, the final fused feature vector may be applied to the backbone (207) instead of the feature vector (330) to generate the final feature map. That is, the output of the encoder (300) may be the final fused feature vector.

[0069] The processor (11) applies the final feature map output from the backbone (207) to the detection head (209) to estimate objects in the point cloud (211) (S500).

[0070] Figure 5 shows a flowchart for explaining in detail the operation of outputting a feature vector for each of the fillers illustrated in Figure 4.

[0071] Referring to FIGS. 1 to 5, the processor (11) samples the LiDAR points included in each filler in an arbitrary number (e.g., 20), and then applies the sampled LiDAR points to a first neural network to generate k-dimensional point feature vectors (S310).

[0072] The processor (11) calculates attention scores for the point feature vectors (S320). Detailed operations for calculating the attention scores are described below.

[0073] The processor (11) divides n k-dimensional point feature vectors (311) into M groups (S321). The n k-dimensional point feature vectors (311) can be divided into a plurality of k / 4-dimensional sub-point feature vectors (311-1, 311-2, 311-3, and 311-4). The number of the plurality of k / 4-dimensional sub-point feature vectors (311-1, 311-2, 311-3, and 311-4) is n.

[0074] The processor (11) applies n k / M dimension point feature vectors (311-1, 311-2, 311-3, and 311-4) divided into M groups to a convolution block (313) to output n compressed feature vectors (315) in M ​​dimensions (S323). The n compressed feature vectors (315) divided into M groups refer to n M dimension compressed feature vectors (315).

[0075] The processor (11) applies n M-dimensional compressed feature vectors (315) to a softmax function (317) to calculate attention scores (319) for the n compressed feature vectors (315) (S325).

[0076] The processor (11) outputs a feature vector (330) using the calculated attention scores (319) (S330). Detailed operations for outputting the feature vector (330) are described below.

[0077] The processor (11) generates group-wise weighted sums (321, 323, 325, 327) for each of n k / M dimension point feature vectors (311-1, 311-2, 311-3, and 311-4) divided into M groups and each of multiple attention scores (319-1, 319-2, 319-3, 319-4) divided into M groups (S331).

[0078] The processor (11) performs a concatenation operation to combine weighted sums (321, 323, 325, 327) to output a single feature vector (330) (S33).

[0079] Unlike the prior art that uses multiple layers to utilize max pooling, the present invention uses only one filler extraction layer (310). By using only one filler extraction layer (310) instead of multiple layers, the computation time can be reduced.

[0080] Since max pooling reflects information from only one of the points within a voxel or pillar, subsequent MLP layers must process each point feature and the max pooling result by concatenating them, which leads to increased memory usage and computational inefficiency due to the use of redundant information.

[0081] In addition, since max pooling selects only the feature with the largest value among all points, voxel features may be distorted by outlier points that do not appear during training.

[0082] The present invention considers information on all points and applies learned attention weights according to importance, thereby being robust against outliers and insensitive to noise. Since the present invention can apply greater weight to important points in a single filler (203), the main features of the object are emphasized, contributing to an improvement in overall performance.

[0083] Unlike max pooling, the present invention generates voxel-level features or pillar-level features through ascension without the need to concatenate all point features, thereby reducing unnecessary computation and memory waste in subsequent layers.

[0084] In addition, the present invention has the effect of enabling robust feature extraction against outliers by using weighted sums rather than max pooling.

[0085] Figure 6 shows a block diagram for explaining the operation of the backbone illustrated in Figure 2.

[0086] Referring to FIGS. 1, FIGS. 2, and FIGS. 6, the processor (11) generates a feature map (401) from a 3D point cloud (201). The feature map (401) may be generated by an encoder (300) shown in FIG. 2. The feature map (401) refers to a 2D pseudo-image or BEV (Bird's Eye View) feature map as described in FIG. 2.

[0087] The processor (11) performs a convolution operation on the feature map (401) to output a final feature map (419). The operation of generating the final feature map (419) from the feature map (401) is performed by the backbone (400). In FIG. 2, the backbone is shown by reference numeral 207, but in FIG. 6, the backbone is shown by reference numeral 400.

[0088] The detailed operation of the backbone (400) is described below.

[0089] The processor (11) performs a first convolution operation on a feature map (401) to output a first feature map (403) having the same scale as the feature map (401) (S410). For example, when the feature map (401) that is the input to the first convolution operation is 32 x 32, the kernel is 3 x 3, the stride is 1, and the padding is 1, the first feature map (403) that is the output of the first convolution operation may be 32 x 32. The feature map (401) has high resolution and has low-level features such as edges, corners, or textures. Therefore, the feature map (401) is used to detect small objects (e.g., pedestrians).

[0090] The processor (11) performs a second convolution operation on a first feature map (403) and outputs a second feature map (405) having a scale smaller than that of the first feature map (403) (S420). For example, when the first feature map (403) which is the input to the second convolution operation is 32 x 32, the kernel is 3 x 3, the stride is 2, and the padding is 1, the second feature map (405) which is the output of the second convolution operation may be 16 x 16.

[0091] The processor (11) performs a third convolution operation on the second feature map (405) and outputs a third feature map (407) having a scale smaller than that of the second feature map (405) (S430). For example, when the second feature map (405), which is the input to the third convolution operation, is 16 x 16, the kernel is 3 x 3, the stride is 2, and the padding is 1, the third feature map (407), which is the output of the third convolution operation, may be 8 x 8.

[0092] The processor (11) performs a fourth convolution operation on the third feature map (407) and outputs a fourth feature map (409) having a scale smaller than that of the third feature map (407) (S440).

[0093] For example, when the third feature map (407), which is the input to the fourth convolution operation, is 8 x 8, the kernel is 3 x 3, the stride is 2, and the padding is 1, the fourth feature map (409), which is the output of the fourth convolution operation, may be 4 x 4. The third feature map (407) has low resolution and has high-level features such as object parts or the entire object. Therefore, the feature map (401) is used to detect large objects (e.g., buses, or trucks).

[0094] In the present invention, convolution operations are performed on feature maps having different scales to enable the detection of features of various objects ranging from small objects to large objects.

[0095] The processor (11) performs a fifth convolution operation on the second feature map (405) to output a fifth feature map (411) (S450). The fifth convolution operation refers to a 1 x 1 convolution operation. The fifth feature map (411) may be 16 x 16. The fifth convolution operation is used to match the number of channels of the fifth feature map (411), the sixth feature map (413), and the seventh feature map (415).

[0096] The processor (11) upscales the third feature map (407) to generate the sixth feature map (413) (S460). According to an embodiment, the processor (11) may upscale the third feature map (407) and perform a seventh convolution operation to generate the sixth feature map (413). The sixth feature map (413) may be 16 x 16.

[0097] The processor (11) upscales the fourth feature map (409) to generate the seventh feature map (415) (S470). According to an embodiment, the processor (11) may upscale the fourth feature map (409) and perform an eighth convolution operation to generate the seventh feature map (415). The seventh feature map (415) may be 16 x 16. The scales of the fifth feature map (411), the sixth feature map (413), and the seventh feature map (415) are all the same. The fifth feature map (411), the sixth feature map (413), and the seventh feature map (415) are independent of each other. That is, it means that the class information in the fifth feature map (411), the sixth feature map (413), and the seventh feature map (415) is independent of each other.

[0098] The processor (11) concatenates the fifth feature map (411), the sixth feature map (413), and the seventh feature map (415) to generate a final feature map (419) (S480).

[0099] Unlike conventional methods, the present invention configures class information independently in feature maps having different scales. Therefore, unlike conventional methods, unnecessary transit operations are eliminated, reducing computational load and minimizing loss of representation resolution.

[0100] The processor (11) applies the final feature map (419) output from the backbone (207) to the detection head (209) to estimate objects in the point cloud (211).

[0101] The detection head (209) includes a plurality of convolution layers (not shown). For example, the detection head (209) may include a plurality of 3 x 3 convolution layers (not shown).

[0102] The input of the detection head (209) is the final feature map (419). The output of the detection head (209) may be a heatmap, the size of the bounding box of the object detected in the heatmap, the class score of the detected object, or the rotation of the bounding box. The heatmap represents a probability map for the center of the detected object. The probability map is represented by a Gaussian distribution. When viewed from above, the probability map appears as a circle. The size of the object's bounding box includes length, width, and height. The rotation of the bounding box may be represented by the sine and cosine values ​​of the yaw.

[0103] Figure 7 shows training images to explain the operation of the detector head illustrated in Figure 2.

[0104] Referring to FIGS. 1 to 7, in the prior art, a probability map represented by a Gaussian distribution was used as a training image to train the heatmap. However, in the case of objects with large differences in length and width, such as buses, if a circular probability map of the Gaussian distribution is used as a training image, there is a possibility that detection errors may occur due to a discrepancy with the object shape.

[0105] In the present invention, the shape recognition rate of objects such as buses can be improved by using an elliptical probability map that reflects the shape of objects with large differences in length and width, such as buses, as a training image. That is, in the present invention, not only circular Gaussian heatmaps but also elliptical Gaussian heatmaps are used as training images. FIG. 7(a) is a training image representing a circular Gaussian heatmap, and FIG. 7(b) is a training image representing an elliptical Gaussian heatmap.

[0106] The above training output image represents the Ground Truth. An elliptical Gaussian heatmap is a two-dimensional probability distribution generated by reflecting the width and length of an object's bounding box relative to the object's center point, and it means that the equivalent line of the probability distribution has an elliptical shape. The above elliptical Gaussian heatmap can be expressed as shown in the following mathematical formula.

[0107] [Mathematical Formula]

[0108] G(x, y) = exp( - [ (x - x o )² / (2σ x 2) + (y - y o )² / (2σ y ²) ] )

[0109] The above G(x, y) represents the pixel value in the elliptical Gaussian heatmap, the above (xo, yo) represents the center coordinates of the object, and the above σ x The dispersion range determined according to the width of the bounding box, the above σ y represents the range of dispersion determined by the length of the bounding box.

[0110] In the present invention, to more accurately recognize the shape of an object, an elliptical Gaussian heatmap is additionally trained as an auxiliary task in addition to the primary detection purpose. Through this, auxiliary supervision can be performed so that shape information reflecting the sizes of various bounding boxes of the object is learned using the final feature map (419) output from the backbone (207). The auxiliary training is performed in parallel with center point prediction, and the learning loss is backpropagated to the final feature map (419) to ensure that shape information is richly included. As a result, the accuracy of center point detection and bounding box regression for objects with large differences in length and width (e.g., buses, trucks, etc.) can be improved.

[0111] In addition, the prior art used a focal loss function that sets only the correct pixel to 1 and applies a weight of (1-ground truth) to the remaining pixels. The prior art had a problem in that when predicting in an environment different from the training environment, the reliability was greatly reduced even if the features of the final feature map (419) output from the backbone (207) were only slightly different.

[0112] However, in the present invention, the loss function is designed so that the predicted heatmap follows a distribution closer to ground truth by applying the mean square error (MSE) loss function to a probability distribution containing continuous values ​​between 0 and 1. Therefore, unlike the prior art, the present invention has the effect that a significant decrease in reliability does not occur even if the features of the final feature map (419) output from the backbone (207) differ slightly when predicting in an environment different from the learning environment.

[0113] Unlike images, LiDAR point data, which is a 3D point cloud, has the characteristic that the size of an object remains constant regardless of the observation distance. Considering this characteristic of LiDAR point data, the present invention can secure sufficient expressive power with only a simple neural network structure. Therefore, the present invention makes it possible to maintain detection performance while reducing the computational load and memory usage of the neural network.

[0114] In feature maps having different resolutions, the class information of the features is configured independently. Therefore, the present invention enables the learning of different object classes for each scale. For example, in a feature map with low resolution, high-level features are learned to detect relatively large objects such as buses or trucks. On the other hand, in a feature map with high resolution, low-level features are learned to detect relatively small objects such as pedestrians. In other words, the present invention maintains independent representations between resolutions.

[0115] Feature maps with different resolutions are directly upsampled or converted directly into final feature maps through a refinement process. The refinement process refers to an additional processing operation that supplements or strengthens already extracted features into a more meaningful and less noisy form, and may refer to a 1 x 1 convolution operation.

[0116] Therefore, the present invention eliminates unnecessary operations, thereby reducing the amount of computation and improving learning efficiency.

[0117] In LiDAR-based object detection, physical size characteristics of classes such as vehicles, pedestrians, and bicycles are distributed differently depending on the scale. Considering this environment, the present invention allows feature maps with different resolutions to be trained to emphasize the characteristics of a specific class, thereby improving training efficiency.

[0118] The present invention has been described with reference to an exemplary embodiment illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0119] 100: LiDAR system; 10: Computing device; 11: Processor; 13: Memory;

Claims

Claim 1 A method for object detection in a point cloud performed by a processor comprises: a step of generating a feature map from a 3D point cloud; a step of performing a convolution operation on the feature map to output a final feature map; and a step of detecting an object in the final feature map, wherein the step of outputting the final feature map comprises: a step of performing a first convolution operation on the feature map to output a first feature map having the same scale as the feature map; a step of performing multi-stage downsampling on the first feature map to generate a plurality of feature maps having different scales; and a step of performing a 1x1 convolution operation or upsampling on the feature maps having different scales to convert them into a plurality of feature maps having the same scale and number of channels and maintaining their respective class information independently. The method includes the step of generating a final feature map by splicing a plurality of feature maps having the same scale and number of channels, wherein the step of generating a plurality of feature maps having different scales comprises: a step of performing a second convolution operation on a first feature map to output a second feature map having a scale smaller than the scale of the first feature map; and a step of performing a third convolution operation on the second feature map to output a third feature map having a scale smaller than the scale of the second feature map. The method includes the step of performing a fourth convolution operation on the third feature map to output a fourth feature map having a scale smaller than the scale of the third feature map, and the step of converting into a plurality of feature maps having the same scale and number of channels and each maintaining independent class information includes: the step of performing a fifth convolution operation on the second feature map to output a fifth feature map; and the step of upscaling the third feature map to generate a sixth feature map having the same scale and number of channels as the fifth feature map and maintaining independent class information for the fifth feature map.A method for detecting objects in a point cloud, comprising the step of upscaling the fourth feature map to generate a seventh feature map having the same scale and number of channels as the fifth feature map, and maintaining independent class information for the fifth feature map and the sixth feature map. Claim 2 delete Claim 3 delete Claim 4 A method for detecting objects in a point cloud according to claim 1, wherein the step of detecting an object in the final feature map is to detect the object using a neural network, a heatmap is used to detect the object, and an elliptical Gaussian heatmap is used as a training image to train the heatmap. Claim 5 A processor that executes object detection commands in a point cloud; and includes a memory for storing the above commands, wherein the commands are implemented to generate a feature map from a 3D point cloud, perform a convolution operation on the feature map to output a final feature map, and detect an object in the final feature map; the commands for outputting the final feature map are implemented to include the step of performing a first convolution operation on the feature map to output a first feature map having the same scale as the feature map, performing multi-stage downsampling on the first feature map to generate a plurality of feature maps having different scales, performing a 1x1 convolution operation or upsampling on the feature maps having different scales to convert them into a plurality of feature maps having the same scale and number of channels and maintaining their respective class information independently, and splicing the plurality of feature maps having the same scale and number of channels to generate a final feature map; and the commands for generating a plurality of feature maps having different scales are implemented to include the step of performing a second convolution operation on the first feature map to generate a second feature map having a scale smaller than the scale of the first feature map The method is implemented to output a map, perform a third convolution operation on the second feature map to output a third feature map having a scale smaller than the scale of the second feature map, and perform a fourth convolution operation on the third feature map to output a fourth feature map having a scale smaller than the scale of the third feature map, and the commands for converting into a plurality of feature maps having the same scale and number of channels and maintaining independent class information for each feature map are implemented to perform a fifth convolution operation on the second feature map to output a fifth feature map, upscale the third feature map to generate a sixth feature map having the same scale and number of channels as the fifth feature map and maintaining independent class information for the fifth feature map, and upscale the fourth feature map,A computing device implemented to generate a seventh feature map that maintains independent class information for the fifth feature map and the sixth feature map, wherein the scale and number of channels of the fifth feature map are the same. Claim 6 delete Claim 7 delete Claim 8 A computing device in which, in the fifth aspect, commands for detecting an object in the final feature map are implemented such that the object is detected using a neural network, a heatmap is used to detect the object, and an elliptical Gaussian heatmap is used as a training image to train the heatmap.

Images