Local expansion and inter-row sampling method for 3D target detection
Through local expansion and cross-row sampling methods, combined with space-content adaptive interactive processing, the problem of difficulty in extracting long-distance feature dependency relationships in 3D object detection is solved, which improves detection performance and reduces consumption.
Patent Information
- Application Number
- CN202411821111.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-04-08
AI Technical Summary
The existing 3D object detection method is difficult to extract long-distance feature dependencies, and the self-attention mechanism is insufficient in fusing long-distance feature dependencies, resulting in the lack of spatial relative position information and insufficient fusion of context information.
A local expansion and cross-row sampling method is proposed. Sparse features are extracted through a 3D sparse backbone network, and partially cut and expand processing is performed. Combined with space-content adaptive interaction processing and cross-row adaptive downsampling, enhancing the spatial information expression ability and receptive field of the feature.
It effectively alleviates the loss of long-distance feature dependency caused by sparse convolution, improves the performance of 3D object detection, reduces the time and space consumption of detection, and makes significant progress in experiments on NuScenes and Kitti datasets.
Smart Images
Figure CN119785337A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision applications, and in particular relates to a local expansion and cross-row sampling method for 3D target detection. Background Art
[0002] With the development of technologies such as autonomous driving, drones, and intelligent transportation systems, 3D object detection plays a vital role in these fields. 3D object detection refers to the detection and identification of objects through three-dimensional point cloud data collected by sensors (such as LiDAR, millimeter-wave radar, etc.). Compared with traditional 2D image detection, 3D object detection can provide more accurate spatial position information and more accurately describe the size, shape, and position of objects. Therefore, it is widely used in scenarios such as autonomous driving that require high-precision perception capabilities.
[0003] At present, existing high-performance 3D object detection methods usually redesign the backbone network or use sparse detectors to alleviate the problem of long-distance feature dependency loss caused by the sparse distribution of points in 3D scenes. At the same time, the adaptive fusion mechanism has great advantages and potential in fusing long-distance feature dependencies. However, existing methods have great limitations in design ideas and the balance between time and space, resulting in unsatisfactory detection results. Summary of the invention
[0004] The purpose of the present invention is to address the reasons why existing 3D target detection is difficult to extract long-distance feature dependencies and the shortcomings of the idea of self-attention in extracting long-distance feature dependencies, mainly the lack of spatial relative position information and insufficient fusion of context information. A local expansion and cross-row sampling method for 3D target detection is proposed. This method can solve the problems of missing spatial relative position information and insufficient fusion of context information, alleviate the loss of long-distance dependencies caused by sparse convolution, greatly improve the performance of 3D target detection, and reduce the time and space consumption of detection.
[0005] In order to achieve the above object, the present invention adopts the following technical solution:
[0006] A local expansion and cross-row sampling method for 3D object detection includes the following steps:
[0007] Step A: Given a radar point cloud input, first use the 3D sparse backbone network to extract sparse feature maps. The sparse features are compressed into dense BEV features x before being fed to the 2D dense backbone. Then, the BEV features x after passing through the 2D dense backbone network are locally segmented into blocks, divided into b×b local regions, and it is ensured that there is no overlap between these local regions, and that each feature is completely divided into a local region. Finally, the local region is expanded with an expansion scale of e. After each local region is expanded, an expansion window is formed, that is, the expansion window consists of a local region and an expansion region around the local region.
[0008] Step B: Perform space-content adaptive interactive processing on each expansion window generated in step A. The specific steps are as follows:
[0009] Step B01: for each local area in the expansion window, each feature block therein is used as a reference feature block, and the boundary of the reference feature block is expanded with an expansion scale of e to generate an expanded block T;
[0010] Step B02: Design a masking scheme, mark all feature blocks in the local area as masks, and cover all feature blocks in the local area. The masking scheme allows the reference feature blocks to better integrate the surrounding environment features in the calculation of step B04;
[0011] Step B03: After expansion, each reference feature block includes a part of the expansion area and its own feature block. In order to merge the features of the expansion area into the reference feature block, the self-attention mechanism is improved to form a space-content adaptive interactive processing mechanism. The weight formula in the adaptive interactive processing mechanism is designed as follows:
[0012] f(i,j,p,q) self-att(ours) =softmax pq ((W Q x ij ) T W K x pq +(W Q x ij ) T l p-i,q-j )W V (1)
[0013] Among them, f(i,j,p,q) is the defined 2D spatial pooling function, which can generate a weight for each dilated block T of size k*k and centered at (i,j) C represents the number of channels in the deep learning network. in and C out Respectively represent the number of channels of deep learning network input and output; WQ , W K , W V are learned linear transformations that are shared across all content and spatial locations. They help the BEV feature x in step A generate queries, keys, and values when transformed, which makes the weights generated by the 2D spatial pooling function f(i,j,p,q) unfixed, adaptively integrating the surrounding environment and remote dependencies of each feature, while reducing the number of parameters; i and j in the formula represent the position index of the benchmark feature block in the BEV feature x, and p and q represent the position index of each feature block in the dilated block centered on (i,j) relative to (i,j) in the BEV feature; x ij represents the features of BEV feature x at index positions p and q, x pq represents the features of BEV feature x at index positions p and q; l p-i,j-q is relative to x ij The spatial position scalar probability of is obtained by learning the relative position; (W Q x ij ) T W K x pq It is used to capture the content-content interaction between query features and key features in local blocks. (W Q x ij ) T l p-i,q-j It is used to capture the feature interaction between the spatial position and content relative to the query feature;
[0014] Step B04: After obtaining the weight corresponding to the BEV feature x, the adaptive interaction processing mechanism uses the weight to fuse the BEV feature with the surrounding environment. The formula is as follows:
[0015]
[0016] Among them, y ij Represents feature x ij The features that integrate the surrounding environment and detail information are generated through the adaptive interactive processing mechanism; the meanings of the remaining parameters are consistent with those of the parameters in B03;
[0017] Step B05: According to steps B03 and B04, the reference feature block y that integrates the surrounding environment features is obtained. ij , all the reference feature blocks in an expansion window are pieced together to replace the feature blocks in the local area. After processing all the expansion windows, an adaptive interactive feature that integrates the surrounding environment and detail information is obtained.
[0018] Step C: Perform cross-row adaptive downsampling processing on each expansion window generated in step A. The specific steps are as follows:
[0019] Step C01: Divide each expansion window generated in step A into blocks according to the side length (b+2e) / b to form expansion window blocks;
[0020] Step C02: independently encode the feature blocks in each expansion window block, and process them from left to right and from top to bottom in sequence; then sample the feature block with the smallest code number, that is, the feature block in the upper left corner of the expansion window block, as the base feature block of the block;
[0021] Step C03: perform an expansion operation on the expansion window block, so that its boundary expands outward by a distance e, and the expanded area is called the block expansion area; and perform steps B03 and B04 on the expanded expansion window block and the block base feature block therein, so that the block base feature block fully integrates the features of the block expansion area;
[0022] Step C04: After calculating the interactions between all the sub-base feature blocks and the surrounding features, merge them and fill them back into the position of the local area in step A; finally, the BEV feature that integrates the feature context information and the remote feature dependency is obtained, and is merged with the adaptive interaction feature obtained in step B and sent to the detection head for target detection.
[0023] In step A, the entire BEV feature map is divided into a number of local areas of size b×b. In order to ensure that the local blocks divide the entire BEV feature without duplication or omission, the side length of the local blocks can be divided by the side length of the global BEV feature.
[0024] The specific steps of step A, step B, and step C regarding the value of the expansion scale e are as follows:
[0025] In order to make the number of sub-block feature blocks in each expansion window consistent with the local block size in step C, the value of e needs to satisfy the condition of (2*e)%b=0.
[0026] The local expansion operation can help enhance the expression ability of spatial information in the feature extraction process, so that the model can better capture target information of different scales and greatly reduce the amount of calculation; while cross-row sampling helps to expand the receptive field, thereby improving the robustness and accuracy of detection. By combining these methods, the problem of loss of long-distance feature dependencies caused by point cloud data sparsity and sparse convolution can be effectively improved, and the target detection performance of large, long-distance and highly environmentally relevant objects can be improved. This has important application value for autonomous driving, intelligent monitoring and other application scenarios involving 3D perception.
[0027] Compared with the prior art, the present invention has the following advantages:
[0028] First, in order to solve the problem of loss of long-distance feature dependencies in sparse convolution, the present invention proposes a corresponding solution. The method effectively utilizes the voxel point cloud feature learning method and combines it with the self-attention mechanism to improve the three-dimensional target detection performance with controllable memory consumption.
[0029] Second, the present invention proposes a local block and expansion scheme, and improves the self-attention method, incorporating the factors of spatial relative position into the weight generation process. This not only avoids the huge computational consumption, but also includes rich scene context, greatly improving the 3D detection performance.
[0030] Third, the present invention proposes a cross-row self-attention downsampling process for each local area, which aggregates richer contextual information from scenes with larger receptive fields to further avoid the problem of losing long-range dependencies between features.
[0031] Fourth, the present invention has conducted extensive experiments on the NuScenes dataset and the Kitti dataset. The processing speed reaches 6.8 frames per second, the official accuracy index of the NuScenes dataset reaches 71.5%, and the average detection accuracy of the Kitti dataset reaches 84.1%. The results of both datasets exceed the detection accuracy of the best methods in the past, especially in the detection of large, long-distance and highly spatial environment-related targets. Significant progress has been made. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is an overall flow chart of the method of the present invention.
[0033] Figure 2 The structural schematic and specific implementation details of the method of the present invention are shown.
[0034] Figure 3 It shows the real form of the point cloud space from the BEV perspective.
[0035] Figure 4 The specific implementation details of space-content adaptive interaction are demonstrated.
[0036] Figure 5 The effectiveness of the present invention in detecting large targets (trucks) at long distances is demonstrated, wherein the left figure is the detection result of the best existing single-modal 3D target detection method, and the right figure is the detection result of the present invention.
[0037] Figure 6 The effectiveness of the present invention in long-distance target (car) detection is demonstrated, wherein the left figure is the detection result of the best existing single-modal 3D target detection method, and the right figure is the detection result of the present invention.
[0038] Figure 7The effectiveness of the present invention in detecting targets (pedestrians) with a high degree of environmental relevance is demonstrated, wherein the left figure is the detection result of the best single-modal 3D target detection method in the past, and the right figure is the detection result of the present invention.
[0039] Figure 8 The effectiveness of the present invention in detecting long-distance targets (cars) and large targets (trucks) is demonstrated, wherein the left figure is the detection result of the best single-modal 3D target detection method in the past, and the right figure is the detection result of the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the described embodiments are only preferred examples of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0041] The present invention proposes a local expansion and cross-row sampling method for 3D object detection. The whole process of the method is as follows: Figure 1 shown.
[0042] The method mainly includes the following steps:
[0043] Step A: Given a radar point cloud input, such as Figure 3 The morphology of the point cloud space from the BEV perspective is shown. First, a 3D sparse backbone network is used to extract sparse feature maps. The sparse features are compressed into dense BEV features x before being fed into the 2D dense backbone. Figure 2 As shown in the figure, the structure and specific implementation details of the network are shown; then, the BEV feature x after passing through the 2D dense backbone network is locally divided into blocks, divided into b×b local areas, and it is ensured that there is no overlap between these local areas, and that each feature is completely divided into a local area; finally, the local area is expanded with an expansion scale of e, and each local area is expanded to form an expansion window, that is, the expansion window consists of a local area and the expansion area around the local area; in Figure 2 In the actual operation example, the side length of the segmentation block is defined as 2, and the entire BEV feature space is divided into multiple local areas of [2,2,c]. Here and below, c represents the number of channels of the deep learning network. Finally, the block expansion is performed according to the distance of 1 feature block to generate multiple expansion windows;
[0044] Step B: Perform space-content adaptive interactive processing on each expansion window generated in step A. The specific steps are as follows:
[0045] Step B01: Figure 4As shown, for each local area in the expansion window, each feature block therein is used as a reference feature block, and the boundary of the reference feature block is expanded with an expansion scale of e to generate an expansion block T;
[0046] Step B02: Figure 4 The idea of designing a masking scheme is shown in FIG. All feature blocks in the local area are marked as masks to cover all feature blocks in the local area. The masking scheme allows the reference feature blocks to better integrate the surrounding environment features in the calculation of step B04.
[0047] Step B03: After expansion, each reference feature block includes a part of the expansion area and its own feature block. In order to merge the features of the expansion area into the reference feature block, the self-attention mechanism is improved to form a space-content adaptive interactive processing mechanism. The weight formula in the adaptive interactive processing mechanism is designed as follows:
[0048] f(i,j,p,q) self-att(ours) =softmax pq ((W Q x ij ) T W K x pq +(W Q x ij ) T l p-i,q-j )W V (1)
[0049] Among them, f(i,j,p,q) is the defined 2D spatial pooling function, which can generate a weight for each dilated block T of size k*k and centered at (i,j) C represents the number of channels in the deep learning network. in and C out Respectively represent the number of channels of deep learning network input and output; W Q , W K , W V are learned linear transformations that are shared across all content and spatial locations. They help the BEV feature x in step A generate queries, keys, and values when transformed, which makes the weights generated by the 2D spatial pooling function f(i,j,p,q) unfixed, adaptively integrating the surrounding environment and remote dependencies of each feature, while reducing the number of parameters; i and j in the formula represent the position index of the benchmark feature block in the BEV feature x, and p and q represent the position index of each feature block in the dilated block centered on (i,j) relative to (i,j) in the BEV feature; x ij represents the features of BEV feature x at index positions p and q, x pqrepresents the features of BEV feature x at index positions p and q; l p-i,j-q is relative to x ij The spatial position scalar probability of is obtained by learning the relative position; (W Q x ij ) T W K x pq It is used to capture the content-content interaction between query features and key features in local blocks. (W Q x ij ) T l p-i,q-j It is used to capture the feature interaction between the spatial position and content relative to the query feature;
[0050] Step B04: After obtaining the weight corresponding to the BEV feature x, the adaptive interaction processing mechanism uses the weight to fuse the BEV feature with the surrounding environment. The formula is as follows:
[0051]
[0052] Among them, y ij Represents feature x ij The features that integrate the surrounding environment and detail information are generated through the adaptive interactive processing mechanism; the meanings of the remaining parameters are consistent with those of the parameters in B03;
[0053] Step B05: According to steps B03 and B04, the reference feature block y that integrates the surrounding environment features is obtained. ij , all the reference feature blocks in an expansion window are pieced together to replace the feature blocks in the local area. After processing all the expansion windows, the adaptive interactive features that integrate the surrounding environment and detail information are obtained. Figure 7 As shown in the figure, the effectiveness of the present invention in detecting targets (pedestrians) with high environmental relevance is demonstrated. The left figure is the detection result of the best single-modal 3D target detection method in the past, and it is obvious that the target is not detected; the right figure is the detection result of the present invention, and the detection effect is very good. It proves that after the method of the present invention adds spatial content adaptive processing, the deep learning network using the method of the present invention has a greatly improved detection effect for small targets at a long distance and targets with high environmental relevance.
[0054] Step C: Perform cross-row adaptive downsampling processing on each expansion window generated in step A. The specific steps are as follows:
[0055] Step C01: Figure 2 As shown in the cross-row adaptive downsampling section, for the expansion window formed in step A, Figure 2For example, the expansion window scale is [4,4,c]. First, it is divided into blocks with a side length of 2, called expansion window blocks, and the scale is [2,2,2,c].
[0056] Step C02: independently encode the feature blocks in each expansion window block, and process them from left to right and from top to bottom in sequence; then sample the feature block with the smallest code number, that is, the feature block in the upper left corner of the expansion window block, as the block base feature block; Figure 2 As shown in Figure 1, the scale of the block base feature block obtained by an expansion window block is [2,2,1,1,c].
[0057] Step C03: perform an expansion operation on the expansion window block, so that its boundary expands outward by a distance e, and the expanded area is called the block expansion area; and perform steps B03 and B04 on the expanded expansion window block and the block base feature block therein, so that the block base feature block fully integrates the features of the block expansion area;
[0058] Step C04: After calculating the interaction between all the sub-block feature blocks and the surrounding features, merge them and fill them back to the position of the local area in step A; finally, the BEV feature that integrates the feature context information and the remote feature dependency is obtained, and is merged with the adaptive interaction feature obtained in step B and sent to the detection head for target detection. Figure 5 The figure shows the effectiveness of the present invention in detecting large targets (trucks) at long distances. The left figure shows the detection result of the best existing single-mode 3D target detection method, which obviously does not detect the target; the right figure shows the detection result of the present invention, which has a very good detection effect. Figure 6 As shown in the figure, the effectiveness of the present invention in detecting long-distance targets (cars) is demonstrated. The left figure is the detection result of the best existing single-mode 3D target detection method, which obviously does not detect the target; the right figure is the detection result of the present invention, which has a very good detection effect. Figure 8 As shown in the figure, the effectiveness of the present invention in detecting long-distance targets (cars) and large targets (trucks) is demonstrated. The left figure is the detection result of the best single-modal 3D target detection method in the past, which obviously did not detect the target; the right figure is the detection result of the present invention, which has a very good detection effect. It proves that the present invention method shows excellent detection results for large targets and long-distance targets.
[0059] Preferably, in step A, the entire BEV feature map is divided into a number of local regions of size b×b. In order to ensure that the local blocks divide the entire BEV feature without duplication or omission, the side length of the local blocks can be divided by the side length of the global BEV feature. For example, if the side length N of the global BEV feature is 4, then the side length b of the local block must be divisible by N, so b here can be 1, 2 or 4. In the actual operation example, we choose b to be 2.
[0060] Preferably, the specific steps of determining the value of the expansion scale e in step A, step B, and step C are as follows:
[0061] In order to make the number of sub-block feature blocks in each expansion window consistent with the local block size in step C, the value of e needs to satisfy the condition of (2*e)%b=0. For example, when b is defined as 2, e can be selected as 1, 2, 4, etc. In the actual operation example, b is selected as 2 and e is selected as 1.
Claims
1. A local expansion and cross-row sampling method for 3D object detection, characterized in that: The steps include: Step A: Given a radar point cloud input, first use the 3D sparse backbone network to extract sparse feature maps. The sparse features are compressed into dense BEV features x before being fed to the 2D dense backbone. Then, the BEV features x after passing through the 2D dense backbone network are locally segmented into blocks, divided into b×b local regions, and it is ensured that there is no overlap between these local regions, and that each feature is completely divided into a local region. Finally, the local region is expanded with an expansion scale of e. After each local region is expanded, an expansion window is formed, that is, the expansion window consists of a local region and an expansion region around the local region. Step B: Perform space-content adaptive interactive processing on each expansion window generated in step A. The specific steps are as follows: Step B01: for each local area in the expansion window, each feature block therein is used as a reference feature block, and the boundary of the reference feature block is expanded with an expansion scale of e to generate an expanded block T; Step B02: Design a masking scheme, mark all feature blocks in the local area as masks, and cover all feature blocks in the local area. The masking scheme allows the reference feature blocks to better integrate the surrounding environment features in the calculation of step B04; Step B03: After expansion, each reference feature block includes a part of the expansion area and its own feature block. In order to merge the features of the expansion area into the reference feature block, the self-attention mechanism is improved to form a space-content adaptive interactive processing mechanism. The weight formula in the adaptive interactive processing mechanism is designed as follows: f(i,j,p,q) self-att(ours) =softmax pq ((W Q x ij ) T W K x pq +(W Q x ij ) T l p-i,q-j )W V (1) Among them, f(i,j,p,q) is the defined 2D spatial pooling function, which is a dilated block of size k*k centered at (i,j). T Generate a weight C represents the number of channels in the deep learning network. in and C out Respectively represent the number of channels of deep learning network input and output; W Q , W K , W V are learned linear transformations that are shared across all content and spatial locations. They help the BEV feature x in step A generate queries, keys, and values when transformed, which makes the weights generated by the 2D spatial pooling function f(i,j,p,q) unfixed, adaptively integrating the surrounding environment and remote dependencies of each feature, while reducing the number of parameters; i and j in the formula represent the position index of the benchmark feature block in the BEV feature x, and p and q represent the position index of each feature block in the dilated block centered on (i,j) relative to (i,j) in the BEV feature; x ij represents the features of BEV feature x at index positions p and q, x pq represents the features of BEV feature x at index positions p and q; l p-i,j-q is relative to x ij The spatial position scalar probability of is obtained by learning the relative position; (W Q x ij ) T W K x pq It is used to capture the content-content interaction between query features and key features in local blocks. (W Q x ij ) T l p-i,q-j It is used to capture the feature interaction between the spatial position and content relative to the query feature; Step B04: After obtaining the weight corresponding to the BEV feature x, the adaptive interaction processing mechanism uses the weight to fuse the BEV feature with the surrounding environment. The formula is as follows: Among them, y ij Represents feature x ij The features that integrate the surrounding environment and detail information are generated through the adaptive interactive processing mechanism; the meanings of the remaining parameter symbols are consistent with those of the parameter symbols in B03; Step B05: According to steps B03 and B04, the reference feature block y that integrates the surrounding environment features is obtained. ij , all the reference feature blocks in an expansion window are pieced together to replace the feature blocks in the local area. After processing all the expansion windows, an adaptive interactive feature that integrates the surrounding environment and detail information is obtained; Step C: Perform cross-row adaptive downsampling processing on each expansion window generated in step A. The specific steps are as follows: Step C01: Divide each expansion window generated in step A into blocks according to the side length (b+2e) / b to form expansion window blocks; Step C02: independently encode the feature blocks in each expansion window block, and process them from left to right and from top to bottom in sequence; then sample the feature block with the smallest code number, that is, the feature block in the upper left corner of the expansion window block, as the base feature block of the block; Step C03: perform an expansion operation on the expansion window block, so that its boundary expands outward by a distance e, and the expanded area is called the block expansion area; and perform steps B03 and B04 on the expanded expansion window block and the block base feature block therein, so that the block base feature block fully integrates the features of the block expansion area; Step C04: After calculating the interactions between all the sub-base feature blocks and the surrounding features, merge them and fill them back into the position of the local area in step A; finally, the BEV feature that integrates the feature context information and the remote feature dependency is obtained, and is merged with the adaptive interaction feature obtained in step B and sent to the detection head for target detection.
2. The local dilation and cross-row sampling method for 3D object detection according to claim 1, characterized in that: In step A, the entire BEV feature map is divided into a number of local areas of size b×b. In order to ensure that the local blocks divide the entire BEV feature without duplication or omission, the side length of the local blocks can be divided by the side length of the global BEV feature.
3. The local dilation and cross-row sampling method for 3D object detection according to claim 1, characterized in that: The specific steps of determining the value of the expansion scale e in step A, step B, and step C are as follows: In order to make the number of sub-block feature blocks in each expansion window consistent with the local block size in step C, the value of e needs to satisfy the condition of (2*e)%b=0.