Three-dimensional target detection method, device and system

By introducing window division and linear grouping recurrent neural networks in three-dimensional object detection, combined with voxel merging and diffusion technology, the problem of large overhead of long-distance dependency modeling and computing in traditional methods is solved, and more efficient three-dimensional object detection is achieved , suitable for autonomous driving systems.

CN118918577APending Publication Date: 2024-11-08HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410935345.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing three-dimensional object detection method performs poorly when dealing with long-distance dependencies, has a large calculation overhead, and is difficult to achieve efficient object recognition and tracking in complex scenarios. Especially in autonomous driving systems, traditional methods cannot effectively model objects. Global context and complex spatial relationships.

Method used

Using a method based on window division and linear grouping recurrent neural network, the calculation overhead of sparse convolution operations is reduced by introducing voxel merging and diffusion technology, and the local three-dimensional voxel features are extracted through three-dimensional descriptors and autoregression methods to enhance the complexity Characteristic representation ability of scenes.

Benefits of technology

It improves the computing efficiency and real-time nature of three-dimensional object detection, improves the feature representation ability of complex scenarios, and shows better performance. It is suitable for three-dimensional object detection tasks in autonomous driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918577A_ABST
    Figure CN118918577A_ABST
Patent Text Reader

Abstract

The invention relates to a three-dimensional target detection method, device and system. The method mainly comprises the following steps: dividing an initial three-dimensional voxel feature according to a window-based three-dimensional voxel division mode to obtain a three-dimensional voxel feature of a first stage; performing local feature extraction on the three-dimensional voxel features of the first stage through a three-dimensional descriptor to obtain three-dimensional voxel features of a second stage; performing feature generation on the three-dimensional voxel features of the second stage through diffusion and combination to obtain three-dimensional voxel features of a third stage; performing feature extraction on the three-dimensional voxel features of the third stage through a backbone network based on window division and a linear grouping recurrent neural network to obtain three-dimensional voxel features of a fourth stage; and processing the three-dimensional voxel features in the fourth stage through an aerial view backbone network and a detection head to obtain a final detection result. The method shows more excellent performance in a three-dimensional target detection task, and provides new theoretical and technical support for the field.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional target detection method, device and system. Background Art

[0002] The rapid development of autonomous driving technology marks a revolutionary change in the field of transportation and transport. With the advancement of sensor technology, artificial intelligence and real-time data processing capabilities, autonomous driving systems have gradually moved from the laboratory stage to real-world applications. In the current scenario, autonomous driving technology not only means safer roads and more efficient traffic flow, but it can also significantly reduce traffic accidents, improve urban air quality, and bring people a more convenient travel experience. Engineers need to consider many factors when designing and optimizing autonomous driving systems, such as sensor accuracy, environmental perception capabilities, reliability of decision-making algorithms, system security, and laws and regulations to ensure their stability and credibility. Therefore, the promotion of autonomous driving technology will have a far-reaching impact on society, and will also promote the modernization and intelligent development of transportation systems, becoming an important part of sustainable urban life in the future.

[0003] The most critical perception system in autonomous driving refers to the ability to perceive and understand the surrounding environment through sensors and algorithms. These systems usually include multiple technologies such as radar, camera, lidar and ultrasonic sensors to obtain and analyze information about roads, obstacles, pedestrians and other vehicles in real time. The development of perception systems has gone through a process from single sensor to multi-sensor fusion. With the advancement of sensor technology and the reduction of cost, autonomous vehicles can perceive and understand complex traffic scenes more accurately. From traditional mechanical radar to modern high-resolution cameras and advanced deep learning algorithms, the capabilities of perception systems are constantly improving, allowing autonomous vehicles to operate safely in different weather conditions and road conditions. Perception systems are of great significance in autonomous driving. They not only help vehicles monitor and identify various dynamic and static objects in the surrounding environment in real time, but also predict the behavior of other traffic participants to make timely driving decisions. This high-precision environmental perception is the cornerstone of the safety and reliability of autonomous vehicles, which can greatly reduce the incidence of traffic accidents and improve the efficiency and smoothness of the traffic system. The role of perception systems is not limited to the installation of sensors and the acquisition of data. They also provide vehicles with highly autonomous driving capabilities through real-time data processing and advanced algorithm analysis. This autonomous capability allows self-driving vehicles to drive safely in complex urban environments and on highways, creating a safer and more convenient travel experience for passengers and pedestrians.

[0004] Currently, 3D object detection in the field of autonomous driving faces key challenges in feature extraction and long-range relationship modeling using transformer models. Traditional methods attempt to process point cloud data through local self-attention mechanisms (such as cylinder- or voxel-based segmentation) to promote interactions between features. However, due to computational resource constraints, these methods are usually only able to process small-scale feature groups and fail to effectively utilize the potential of transformer models in handling global long-range dependencies. Specifically, current research has not yet addressed how to achieve accurate modeling of relationships between distant objects and structures in complex scenes while maintaining computational efficiency. Modeling long-range relationships needs to go beyond the limitations of local regions and involve complex spatial and semantic dependencies between multiple objects and structural elements. In addition, modeling long-range relationships requires not only large-scale and diverse datasets, but also further optimization of data representation and algorithm design to ensure effective implementation and application in various complex real-world scenarios. Therefore, future research directions will focus on how to effectively integrate the advantages of transformer models and overcome computational limitations to achieve more accurate and efficient 3D object detection and recognition in a wider range of and complex environments.

[0005] Three-dimensional object detection methods in autonomous driving scene perception systems face two core problems: First, these methods usually perform poorly when dealing with long-range dependencies and fail to effectively model the global context and complex spatial relationships of objects. Traditional three-dimensional object detection methods lack an effective attention mechanism to deal with long-range dependencies between objects, which leads to poor performance when capturing long-range relationships and complex spatial structures of objects. For example, in autonomous driving, effectively identifying and tracking distant vehicles or pedestrians is crucial for safe driving, but traditional methods often fail to meet this requirement. Second, these methods are challenging when dealing with high computational overhead, especially when dealing with highly sparse point cloud data. Although the method using attention mechanism can effectively extract long-range dependencies between features, it also requires more computing resources and computational overhead. When dealing with large-scale point cloud data, this method often requires complex computational graph structures and a large number of computing units, which limits its application and deployment efficiency in actual autonomous driving systems. In summary, future research needs to further optimize and innovate three-dimensional object detection methods on the basis of overcoming the lack of long-range dependency modeling capabilities and high computational overhead, so as to improve the perception and decision-making capabilities of autonomous driving systems in complex environments.

[0006] The current problems with voxel partitioning methods are, first of all, the lack of effective use of voxel height information in the partitioning process, which limits the integrity and accuracy of information when processing stereo scenes. Secondly, when some methods support voxel partitioning within large groups, they may result in too many voxels in a single group, which affects the computational efficiency and processing power of the model. In addition, when using sparse convolution for upsampling and downsampling operations, due to the sparsity of point clouds, a large computational overhead will be incurred, limiting the real-time performance and deployment efficiency of the network. In contrast, current networks based on sparse convolutions face challenges in directly extracting general features, and it is difficult to effectively capture the complex structures and local features in point clouds.

[0007] In view of this, how to overcome the defects of the prior art and solve at least part of the above-mentioned technical problems is a difficult problem to be solved in this technical field. Summary of the invention

[0008] In view of the defects or improvement needs in the prior art, in order to solve the problem of high computational overhead of sparse convolution in the current point cloud data processing. The present invention provides a three-dimensional target detection method, device and system, which is based on window partitioning and linear grouping recurrent neural network, and effectively reduces the computational overhead and resource consumption of sparse convolution operations by introducing voxel merging and diffusion technology, and improves the computational efficiency and real-time performance of the model. In addition, by introducing three-dimensional descriptors, local three-dimensional voxel features can be accurately extracted; through voxel generation technology, the voxel features are expanded using autoregressive methods, further enhancing the feature characterization capabilities of complex scenes. Compared with traditional methods, the present invention shows better performance in three-dimensional target detection tasks, and provides new theoretical and technical support for further promoting research and application in this field.

[0009] The present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a three-dimensional target detection method, comprising:

[0011] The initial three-dimensional voxel features are divided according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage;

[0012] The three-dimensional voxel features of the first stage are locally extracted through the three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage;

[0013] The three-dimensional voxel features of the second stage are diffused and merged to generate features, thereby obtaining the three-dimensional voxel features of the third stage;

[0014] The 3D voxel features of the third stage are extracted through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the 3D voxel features of the fourth stage;

[0015] The 3D voxel features of the fourth stage are processed through the bird's-eye view backbone network and the detection head to obtain the final detection results.

[0016] In some embodiments, dividing the initial three-dimensional voxel features according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage specifically includes:

[0017] Voxelize the input point cloud to obtain the initial three-dimensional voxel features;

[0018] The three-dimensional space is divided into non-overlapping windows according to the spatial shape of the initial three-dimensional voxel features, and two different sorting methods, x-axis priority and y-axis priority, are used for the three-dimensional voxel features in each window to obtain different voxel feature sorting orders;

[0019] The sorted voxel features are grouped, that is, the sorted voxel features are divided into equal numbers, and those that are less than the equal number are filled with zero features, so as to obtain the three-dimensional voxel features of the first stage.

[0020] In some embodiments, the three-dimensional descriptor includes a three-dimensional sub-manifold sparse convolution, a layer normalization, and an activation function arranged in sequence.

[0021] In some embodiments, the step of generating features by diffusing and merging the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage specifically includes:

[0022] Obtaining the coordinates of high-response features in the three-dimensional voxel features of the second stage;

[0023] The coordinates of the high-response features are diffused to the adjacent coordinates, and the features of the diffused coordinates are assigned zero values; the new features and new coordinates obtained by diffusion are merged into the features and coordinates of the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage after the voxels are generated.

[0024] In some embodiments, extracting the three-dimensional voxel features of the third stage through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage specifically includes:

[0025] generating an absolute position code according to the coordinates of the three-dimensional voxel features of the third stage, and adding the absolute position code to the three-dimensional voxel features of the third stage to obtain the three-dimensional voxel features of the first intermediate stage;

[0026] The three-dimensional voxel features of the first intermediate stage are serialized and modeled through layers based on a linear grouped recurrent neural network, each network layer of the linear grouped recurrent neural network includes a bidirectional recurrent neural network in different directions, and the serialized modeled three-dimensional voxel features are downsampled by voxel merging; the serialization modeling and downsampling operations are repeated twice to obtain the three-dimensional voxel features of the second intermediate stage;

[0027] The three-dimensional voxel features of the second intermediate stage are expanded and upsampled by voxels to obtain voxel features of the corresponding resolution. The features before downsampling are added back through jump connections to obtain new features. The new features are feature modeled through layers based on linear grouping recurrent neural networks. The upsampling and feature modeling operations are repeated twice to obtain the three-dimensional voxel features of the fourth stage.

[0028] In some embodiments, the three-dimensional voxel features of the fourth stage are processed through the bird's-eye view backbone network and the detection head to obtain the final detection result specifically includes:

[0029] The 3D voxel features of the fourth stage are converted into bird's-eye view features through the bird's-eye view backbone network;

[0030] The bird's-eye view features are used for target detection through the detection head module to obtain the final detection results.

[0031] In some embodiments, the detection head module first predicts whether each voxel or pixel position contains the target center point, and outputs a confidence score to characterize the existence or not of the target center point; for the position predicted to contain the target center point, the detection head module further predicts the position and size of the target's bounding box, adjusts the position of the box through the regression model, fits the target's true bounding box, and obtains the final detection result.

[0032] In a second aspect, the present invention further provides a three-dimensional object detection device, for implementing the three-dimensional object detection method described in the first aspect, the device comprising:

[0033] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the three-dimensional target detection method described in the first aspect.

[0034] In a third aspect, the present invention further provides a three-dimensional target detection system, applying the three-dimensional target detection method as described in the first aspect, the system includes a first stage module, a second stage module, a third stage module, a fourth stage module and a final detection result acquisition module, wherein:

[0035] The first stage module is used to: divide the initial three-dimensional voxel features according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage;

[0036] The second stage module is used to: extract local features of the three-dimensional voxel features of the first stage through the three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage;

[0037] The third stage module is used to: generate features by diffusing and merging the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage;

[0038] The fourth stage module is used to: extract the three-dimensional voxel features of the third stage through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage;

[0039] The final detection result acquisition module is used to process the three-dimensional voxel features of the fourth stage through the bird's-eye view backbone network and the detection head to obtain the final detection result.

[0040] In a fourth aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the three-dimensional target detection method described in the first aspect.

[0041] Compared with the prior art, the beneficial effects of the present invention are: providing a three-dimensional target detection method, device and system, based on window partitioning and linear grouping recurrent neural network, by introducing voxel merging and diffusion technology, effectively reducing the computational overhead and resource consumption of sparse convolution operations, and improving the computational efficiency and real-time performance of the model. In addition, by introducing three-dimensional descriptors, local three-dimensional voxel features can be accurately extracted; through voxel generation technology, the voxel features are expanded using autoregressive methods, further enhancing the feature characterization capabilities of complex scenes. Compared with traditional methods, the present invention shows better performance in three-dimensional target detection tasks, and provides new theoretical and technical support for further promoting research and applications in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required to be used in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A flowchart of a three-dimensional target detection method provided in Example 1 of the present invention;

[0044] Figure 2 The overall flow chart of the detection method based on window partitioning and linear grouping recurrent neural network provided in Example 2 of the present invention;

[0045] Figure 3 A schematic diagram of a network block based on window partitioning and linear grouping recurrent neural network provided in Example 2 of the present invention;

[0046] Figure 4 A schematic diagram of a network layer based on a window partitioning and linear grouping recurrent neural network provided in Example 2 of the present invention;

[0047] Figure 5 A schematic diagram of a voxel merging and voxel expansion module provided in Embodiment 2 of the present invention;

[0048] Figure 6 A schematic diagram of a three-dimensional description submodule provided in Example 2 of the present invention;

[0049] Figure 7 A schematic diagram of a module of a three-dimensional object detection system provided in Example 3 of the present invention;

[0050] Figure 8 A schematic diagram of the structure of a three-dimensional target detection device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can also be made. These all belong to the protection scope of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. In addition, although the functional module division may be performed in the device schematic diagram, and the logical order may be shown in the flow chart, in some cases, it may be different from the module division in the device, or the steps shown or described in the order in the flow chart may be performed.

[0052] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0054] The following first explains and illustrates the technical terms of the present invention:

[0055] CenterHead: A detection head structure commonly used in 3D object detection. CenterHead performs bounding box regression based on the prediction of the center position to accurately locate the position and size of the target object in 3D space. This design combines the characteristics of 3D point cloud data with the needs of object detection, aiming to improve detection accuracy and efficiency, and is suitable for various application scenarios that require detection and identification of objects from point clouds, such as autonomous driving, robot visual navigation, etc.

[0056] Recurrent Neural Network (RNN): A neural network structure specifically designed to process sequential data, such as text, speech, time series, etc. Its uniqueness lies in the recurrent connection, which enables the network to remember previous information and apply it to the current input, thereby capturing the temporal dependencies in the sequence data. This ability makes RNN widely used in natural language processing, time series prediction and other fields. At the same time, its variants such as long short-term memory network (LSTM) and gated recurrent unit (GRU) further improve the ability to model long-term dependencies by solving the gradient vanishing problem.

[0057] Bidirectional recurrent neural network: A recurrent neural network that combines forward and backward, taking into account both past and future information of sequence data, and comprehensively understanding and processing sequences by combining hidden states in the forward and backward directions. It is widely used in language modeling, named entity recognition, and other tasks that require global sequence understanding.

[0058] Submanifold sparse convolution: A convolutional neural network technique specifically designed for processing point cloud data, which effectively captures and utilizes local features and spatial relationships in unstructured data by considering the sparse nature and submanifold structure of point clouds. Coupled with its focus on valid points in point cloud data, this technique is widely used in 3D vision and machine learning tasks, such as point cloud segmentation, classification, and semantic segmentation, providing efficient and accurate solutions for fields such as autonomous driving, robot navigation, and medical image processing.

[0059] ScatterAdd: An operation in programming and computing that is usually used to scatter values ​​into a tensor according to specified indices and add them together. Specifically, its function is to add the input value to the specified position of the output tensor according to the given index. In machine learning and deep learning, ScatterAdd is often used to process sparse data or scenarios where tensors need to be dynamically updated according to certain conditions or indices. For example, it can be used to update gradient information or process pixel values ​​in non-contiguous areas in image processing.

[0060] Skip connection: A technique in deep neural networks that is used to solve problems such as gradient vanishing and network training difficulties. It directly adds the features of the input or intermediate layer to the output of the subsequent layer so that information can be directly transferred across layers, which helps reduce the impact of information loss and gradient vanishing, while improving the training efficiency and convergence speed of the network. Skip connections are common in structures such as residual networks and U-Net. In residual networks, skip connections allow some layers to bypass some convolutional layers and directly add input or intermediate features to the output in order to learn residual information more effectively. In U-Net, skip connections are used to connect the feature maps of the encoder (downsampling path) with the feature maps of the decoder (upsampling path), which helps with accurate semantic segmentation and image reconstruction tasks. The use of skip connections allows the network to learn features more deeply and efficiently, thereby improving the performance and effects of many computer vision and natural language processing tasks.

[0061] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0062] Embodiment 1:

[0063] like Figure 1 As shown, an embodiment of the present invention provides a three-dimensional target detection method, which includes the following steps.

[0064] Step 100: The initial three-dimensional voxel features are divided according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage. Specifically, it includes: voxelizing the input point cloud to obtain the initial three-dimensional voxel features; dividing the three-dimensional space into non-overlapping windows according to the spatial shape of the initial three-dimensional voxel features, and using two different sorting methods of x-axis priority and y-axis priority for the three-dimensional voxel features in each window to obtain different voxel feature sorting orders; then, grouping the sorted voxel features, that is, dividing the sorted voxel features into equal numbers, and filling the less than equal numbers with feature zero filling, so as to obtain the three-dimensional voxel features of the first stage.

[0065] Step 200: extract local features of the three-dimensional voxel features of the first stage through a three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage. The three-dimensional descriptor includes a three-dimensional sub-manifold sparse convolution, a layer normalization, and an activation function arranged in sequence.

[0066] Step 300: Generate features of the three-dimensional voxel features of the second stage by diffusion and merging to obtain the three-dimensional voxel features of the third stage. Specifically, it includes: obtaining the coordinates of the high-response features in the three-dimensional voxel features of the second stage; diffusing the coordinates of the high-response features to adjacent coordinates, and assigning zero values ​​to the features of the diffused coordinates; merging the new features and new coordinates obtained by diffusion into the features and coordinates of the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage after voxel generation.

[0067] Step 400: extracting the features of the three-dimensional voxel features of the third stage through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage. Specifically, the steps include: generating an absolute position code according to the coordinates of the three-dimensional voxel features of the third stage, adding the absolute position code to the three-dimensional voxel features of the third stage, and obtaining the three-dimensional voxel features of the first intermediate stage; serializing and modeling the three-dimensional voxel features of the first intermediate stage through a layer based on a linear grouping recurrent neural network, each network layer of the linear grouping recurrent neural network includes a bidirectional recurrent neural network in different directions, and downsampling the serialized modeled three-dimensional voxel features through voxel merging; repeating the serialization modeling and downsampling operations twice to obtain the three-dimensional voxel features of the second intermediate stage; expanding the three-dimensional voxel features of the second intermediate stage through voxels and upsampling to obtain voxel features of the corresponding resolution, adding the features before downsampling back through jump connections to obtain new features, and performing feature modeling on the new features through a layer based on a linear grouping recurrent neural network, and repeating the upsampling and feature modeling operations twice to obtain the three-dimensional voxel features of the fourth stage.

[0068] Step 500: Process the 3D voxel features of the fourth stage through the bird's-eye view backbone network and the detection head to obtain the final detection result. Specifically, it includes: converting the 3D voxel features of the fourth stage into bird's-eye view perspective features through the bird's-eye view backbone network; performing target detection on the bird's-eye view perspective features through the detection head module to obtain the final detection result. The detection head module first predicts whether each voxel or pixel position contains the target center point, and outputs a confidence score to characterize the existence or non-existence of the target center point; for the position predicted to contain the target center point, the detection head module further predicts the position and size of the target's bounding box, adjusts the position of the box through the regression model, fits the target's real bounding box, and obtains the final detection result.

[0069] In summary, this embodiment provides a three-dimensional target detection method based on window partitioning and linear grouping recurrent neural network. By introducing voxel merging and diffusion technology, the computational overhead and resource consumption of sparse convolution operations are effectively reduced, and the computational efficiency and real-time performance of the model are improved. In addition, by introducing three-dimensional descriptors, local three-dimensional voxel features can be accurately extracted; through voxel generation technology, the voxel features are expanded using autoregressive methods, further enhancing the feature characterization capabilities of complex scenes. Compared with traditional methods, the present invention shows better performance in three-dimensional target detection tasks, and provides new theoretical and technical support for further promoting research and applications in this field.

[0070] Embodiment 2:

[0071] This embodiment 2 provides a more detailed description of the three-dimensional object detection method provided on the basis of embodiment 1.

[0072] like Figure 2 As shown, in general, this embodiment includes five implementation steps: (1) performing a voxel partitioning method based on window partitioning to obtain serialized voxel features; (2) extracting spatial position information through a three-dimensional descriptor; (3) performing autoregressive diffusion of three-dimensional voxel features through feature generation to make the three-dimensional voxel features denser; (4) using a network block based on window partitioning and a linear recurrent neural network; (5) obtaining detection results through a bird's-eye view backbone network and a detection head network. The following describes the specific implementation method around these five steps.

[0073] (1) A voxel partitioning method based on window partitioning of three-dimensional voxel features is performed to obtain serialized voxel features, including the following sub-steps.

[0074] (1.1) First, the input point cloud is voxelized to obtain three-dimensional voxel features. The input point cloud is voxelized and three-dimensional voxel features are obtained through a simple linear layer; specifically, first, the size of each voxel is defined, usually a cube, and the entire three-dimensional space is gridded according to this size. Next, each point in the point cloud is classified and assigned to the voxel to which it belongs, usually based on the distance between the point and the voxel boundary or center. Subsequently, the final three-dimensional voxel features can be extracted within each voxel.

[0075] (1.2) The three-dimensional space is divided into non-overlapping windows according to the spatial shape of the three-dimensional voxel features. For the three-dimensional voxel features in each window, we use two different sorting methods: x-axis priority and y-axis priority to obtain different voxel feature sorting orders. Specifically, according to the spatial shape of the three-dimensional voxel features, the three-dimensional space is divided into (T x ,T y ,T z ) windows that do not overlap in spatial position. Secondly, the three-dimensional voxel features inside the window are sorted according to their corresponding spatial coordinates in different sorting methods of x-axis priority sorting and y-axis priority sorting to obtain different sorting methods of voxel features in the window. According to the sorting method obtained above, the three-dimensional voxel features in the window are sorted, and then sorted according to the coordinates of the window to obtain the overall sorting method of the three-dimensional voxel coordinates.

[0076] (1.3) Grouping the sorted voxel features, that is, dividing the sorted voxel features into equal numbers, and filling the less than equal numbers with zeros, thereby obtaining the three-dimensional voxel features of the first stage, so as to save computational overhead and optimize data processing efficiency. Specifically, according to the above-mentioned overall sorting method, the three-dimensional voxel features are divided into groups of the same size, and the mapping coordinates corresponding to the corresponding sorting are obtained. The grouping method of the same size can greatly save computational overhead.

[0077] (2) Extracting spatial position information through 3D descriptors: Using 3D descriptors to extract local features of 3D voxel features, new 3D voxel features are obtained, such as Figure 6 As shown in Figure 1, the 3D descriptor includes a submanifold sparse convolution, a layer normalization, and a GELU activation function. Specifically, the 3D voxel feature is first represented as P = (f i (x,y,z) i ), where f i Represents spatial position (x, y, z) iThe three-dimensional voxel feature is passed through a 3D descriptor to obtain a new three-dimensional voxel feature, which solves the problem that the adjacent voxel features in the spatial position are far apart in sequence due to the voxel partitioning method based on window partitioning.

[0078] (3) Performing autoregressive diffusion of 3D voxel features through feature generation to make the 3D voxel features denser. The process of performing feature diffusion of 3D voxel features through feature generation includes the following sub-steps.

[0079] (3.1) First, the area for feature diffusion is determined by the feature response of the three-dimensional voxel feature. The high response area in the three-dimensional voxel feature is usually the foreground area. The specific calculation is as follows:

[0080]

[0081] in, is the output feature F obtained by the i-th recurrent neural network layer based on window partitioning and linear grouping i The feature response of is, and C is the channel dimension of the output feature.

[0082] (3.2) After that, the features are sorted in descending order, and the highest m voxels are selected as the foreground area, where m depends on the diffusion ratio. The specific operation is as follows:

[0083]

[0084] m=r*L

[0085] in yes The first m features of the foreground voxel F are arranged in descending order, where m is the number of diffuse features, r is the diffusion ratio, and L is the number of non-empty voxels. m The corresponding coordinates of are defined as P m , diffuse P along four different offsets (i.e., [-1,-1,0], [-1,1,0], [1,-1,0], [1,1,0]) on the x-axis, y-axis, and z-axis m To obtain the coordinates of the diffused 3D voxels.

[0086] (3.3) Finally, the features corresponding to the three-dimensional voxel coordinates obtained after diffusion are initialized to zero, that is, the three-dimensional voxel features after the feature generation module are obtained.

[0087] (4) Through the network block based on window partitioning and linear recurrent neural network. The process of extracting features of three-dimensional voxel features through the network block based on window partitioning and linear grouping recurrent neural network includes the following sub-steps.

[0088] (4.1) The coordinates of the voxel feature are converted into an absolute position code and added to the three-dimensional voxel feature. By embedding the spatial position information of each voxel into the feature representation, the model's ability to understand and express the spatial structure is improved. The coordinates of the input three-dimensional voxel feature are upgraded through a linear layer to obtain an absolute position code, which is directly added to the three-dimensional voxel feature. The generated absolute position code is added or concatenated with the original three-dimensional voxel feature, so that the new feature vector contains not only the feature information of the voxel itself, but also its position information in the entire space.

[0089] (4.2) The new three-dimensional voxel features are serialized and modeled through network layers based on linear grouping recurrent neural networks. One network layer includes two bidirectional recurrent neural networks with window-based voxel partitioning in different directions. Downsampling is performed by voxel merging, and the operation is repeated twice. Specifically, the three-dimensional voxel features pass through the first network layer based on window partitioning and linear grouping recurrent neural networks. Each network layer includes a bidirectional recurrent neural network with x-direction partitioning and y-direction partitioning, and downsampling with a step size of [2,2,2] is performed by voxel merging to obtain new three-dimensional voxel features. Repeat the above operation to obtain three-dimensional voxel features with a resolution of 1 / 8 and 1 / 64 of the previous resolution.

[0090] like Figure 3 As shown in Figure 1, the new 3D voxel features are first modeled sequentially through a network layer based on a linear grouped recurrent neural network. Figure 4 As shown in , each network layer includes a bidirectional recurrent neural network in different directions. The obtained 3D voxel features extract local 3D features through three-dimensional descriptors, and then perform downsampling operations through voxel merging to reduce the dimension and complexity of the data, and merge the features to generate a higher level of abstract feature representation. Figure 5 As shown, according to the set step size, the corresponding merged coordinates are calculated:

[0091]

[0092] Among them, (x′, y′, z′) i is the coordinate of the merged point cloud set, (s x ,s y ,s z ) are the step lengths in different directions respectively. Sum the features corresponding to the merged point cloud coordinates to obtain the merged point cloud features:

[0093] f′ i =ScatterAdd({f j |(x j ,y j ,z j )})

[0094] Among them, ScatterAdd is the operation of adding corresponding points. Repeat the above operations of serialization modeling and voxel merging twice to gradually extract and enhance the feature representation of the data.

[0095] (4.3) The features that have been downsampled twice by voxel merging are upsampled by voxel expansion to obtain voxel features of the corresponding resolution, and the features before downsampling are added back by skip connections to obtain new features. The new features are then feature modeled by a layer based on a linear grouping recurrent neural network, and the operation is repeated twice. Specifically, the three-dimensional voxel features with a resolution of 1 / 64 are restored to a resolution of 1 / 8 of the original three-dimensional features by voxel expansion, and at the same time, the three-dimensional voxel features of the same resolution before feature merging are added to the three-dimensional voxel features after voxel expansion by skip connections. The three-dimensional voxel features are passed through the third network layer based on window partitioning and linear grouping recurrent neural network. Repeat the above operations to obtain three-dimensional voxel features of the original resolution size.

[0096] This step performs voxel expansion on the features obtained by voxel merging and restores them to the original voxel resolution, increasing the spatial detail and precision of the features so as to more accurately capture and express the subtle features in the original data. In the accelerated expansion, the features before voxel merging are added back through skip connections. This step retains the information of the original high-resolution features and improves the network's expressiveness in detailed and local structures. Subsequently, the reconstructed features are feature modeled through layers based on linear grouped recurrent neural networks to obtain a higher level of abstract expression. Repeat the above voxel expansion voxel merging, voxel expansion, skip connections, and network layers of linear grouped recurrent neural networks twice to gradually enhance the feature representation and restore the 3D voxel features to the resolution before entering the network block.

[0097] (5) Obtain the detection results through the bird's-eye view backbone network and the detection head network. Specifically, it includes the following sub-steps.

[0098] (5.1) The 3D voxel features processed by the backbone network based on window partitioning and linear grouping recurrent neural network can be further converted into bird's-eye view features. This process usually involves compressing the 3D voxel features into a 2D plane and then extracting the bird's-eye view features through the bird's-eye view backbone network, which can reduce the dimensionality of the data and provide a more intuitive and easy-to-interpret feature representation.

[0099] (5.2) The extracted features are used as the detection head module for target detection through CenterHead. CenterHead is known for its ability to accurately locate the center point of the target and effectively adjust the bounding box. It first predicts whether each voxel or pixel position contains the center point of the target and outputs a confidence score to characterize the existence or non-existence of the center point. For the position predicted to contain the center point of the target, CenterHead further predicts the position and size of the target's bounding box, and fine-tunes the position of the box through the regression model to best fit the target's true bounding box to obtain the final target detection output.

[0100] In summary, the present embodiment proposes a method based on window partitioning and linear grouping recurrent neural network, which effectively reduces the computational overhead and resource consumption of sparse convolution operations by introducing voxel merging and diffusion technology, and improves the computational efficiency and real-time performance of the model. In addition, by introducing three-dimensional descriptors, local three-dimensional voxel features can be accurately extracted; through voxel generation technology, the voxel features are expanded using autoregressive methods, further enhancing the feature characterization capabilities of complex scenes. Compared with traditional methods, the present invention shows better performance in three-dimensional target detection tasks, and provides new theoretical and technical support for further promoting research and applications in this field.

[0101] Embodiment 3:

[0102] Based on the three-dimensional object detection method provided in the above embodiment 1, embodiment 3 of the present invention further provides a three-dimensional object detection system. Figure 7 As shown, the system includes a first-stage module, a second-stage module, a third-stage module, a fourth-stage module and a final detection result acquisition module, wherein: the first-stage module is used to divide the initial three-dimensional voxel features according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage; the second-stage module is used to extract the features of the three-dimensional voxel features of the first stage locally through the three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage; the third-stage module is used to generate features of the three-dimensional voxel features of the second stage through diffusion and merging to obtain the three-dimensional voxel features of the third stage; the fourth-stage module is used to extract features of the three-dimensional voxel features of the third stage through the backbone network based on window division and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage; the final detection result acquisition module is used to process the three-dimensional voxel features of the fourth stage through the bird's-eye view backbone network and the detection head to obtain the final detection result. The specific functions of each module of the above system correspond to the introduction of the method steps in Example 1, and will not be repeated here.

[0103] In addition, based on the three-dimensional object detection method provided in the above embodiment 1, the present invention also provides a three-dimensional object detection device that can be used to implement the above method and system, such as Figure 8 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The three-dimensional object detection device of this embodiment includes one or more processors 21 and a memory 22. Figure 8 A processor 21 is taken as an example.

[0104] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0105] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the three-dimensional object detection method in Example 1. The processor 21 executes various functional applications and data processing of the three-dimensional object detection device by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, the three-dimensional object detection method in Example 1 is implemented.

[0106] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The program instructions / modules are stored in the memory 22, and when executed by one or more processors 21, the three-dimensional object detection method in the above embodiment 1 is executed, for example, the above described Figure 1 The steps shown.

[0108] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0109] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional target detection method, characterized in that: include: The initial three-dimensional voxel features are divided according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage; The three-dimensional voxel features of the first stage are locally extracted through the three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage; The three-dimensional voxel features of the second stage are diffused and merged to generate features, thereby obtaining the three-dimensional voxel features of the third stage; The 3D voxel features of the third stage are extracted through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the 3D voxel features of the fourth stage; The 3D voxel features of the fourth stage are processed through the bird's-eye view backbone network and the detection head to obtain the final detection results.

2. The three-dimensional target detection method according to claim 1, characterized in that: The initial three-dimensional voxel features are divided according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage, which specifically includes: Voxelize the input point cloud to obtain the initial three-dimensional voxel features; The three-dimensional space is divided into non-overlapping windows according to the spatial shape of the initial three-dimensional voxel features, and two different sorting methods, x-axis priority and y-axis priority, are used for the three-dimensional voxel features in each window to obtain different voxel feature sorting orders; The sorted voxel features are grouped, that is, the sorted voxel features are divided into equal numbers, and those that are less than the equal number are filled with zero features, so as to obtain the three-dimensional voxel features of the first stage.

3. The three-dimensional target detection method according to claim 1, characterized in that: The three-dimensional descriptor includes a three-dimensional sub-manifold sparse convolution, a layer normalization and an activation function which are arranged in sequence.

4. The three-dimensional target detection method according to claim 1, characterized in that: The step of generating features by diffusing and merging the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage specifically includes: Obtaining the coordinates of high-response features in the three-dimensional voxel features of the second stage; The coordinates of the high-response features are diffused to the adjacent coordinates, and the features of the diffused coordinates are assigned zero values; the new features and new coordinates obtained by diffusion are merged into the features and coordinates of the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage after the voxels are generated.

5. The three-dimensional target detection method according to claim 1, characterized in that: The three-dimensional voxel features of the third stage are subjected to feature extraction by a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage, which specifically includes: generating an absolute position code according to the coordinates of the three-dimensional voxel features of the third stage, and adding the absolute position code to the three-dimensional voxel features of the third stage to obtain the three-dimensional voxel features of the first intermediate stage; The three-dimensional voxel features of the first intermediate stage are serialized and modeled through layers based on a linear grouped recurrent neural network, each network layer of the linear grouped recurrent neural network includes a bidirectional recurrent neural network in different directions, and the serialized modeled three-dimensional voxel features are downsampled by voxel merging; the serialization modeling and downsampling operations are repeated twice to obtain the three-dimensional voxel features of the second intermediate stage; The three-dimensional voxel features of the second intermediate stage are expanded and upsampled by voxels to obtain voxel features of the corresponding resolution. The features before downsampling are added back through jump connections to obtain new features. The new features are feature modeled through layers based on linear grouping recurrent neural networks. The upsampling and feature modeling operations are repeated twice to obtain the three-dimensional voxel features of the fourth stage.

6. The three-dimensional target detection method according to claim 1, characterized in that: The three-dimensional voxel features of the fourth stage are processed by the bird's-eye view backbone network and the detection head to obtain the final detection results, which specifically include: The 3D voxel features of the fourth stage are converted into bird's-eye view features through the bird's-eye view backbone network; The bird's-eye view features are used for target detection through the detection head module to obtain the final detection results.

7. The three-dimensional target detection method according to claim 6, characterized in that: The detection head module first predicts whether each voxel or pixel position contains the target center point, and outputs a confidence score to characterize the existence or not of the target center point; for the position predicted to contain the target center point, the detection head module further predicts the position and size of the target's bounding box, adjusts the position of the box through the regression model, fits the target's real bounding box, and obtains the final detection result.

8. A three-dimensional object detection device, used to implement the three-dimensional object detection method according to any one of claims 1 to 7, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the three-dimensional target detection method described in any one of claims 1-7.

9. A three-dimensional object detection system, using the three-dimensional object detection method according to any one of claims 1 to 7, characterized in that: The system includes a first-stage module, a second-stage module, a third-stage module, a fourth-stage module, and a final detection result acquisition module, wherein: The first stage module is used to: divide the initial three-dimensional voxel features according to the window-based three-dimensional voxel division method to obtain the three-dimensional voxel features of the first stage; The second stage module is used to: extract local features of the three-dimensional voxel features of the first stage through the three-dimensional descriptor to obtain the three-dimensional voxel features of the second stage; The third stage module is used to: generate features by diffusing and merging the three-dimensional voxel features of the second stage to obtain the three-dimensional voxel features of the third stage; The fourth stage module is used to: extract the three-dimensional voxel features of the third stage through a backbone network based on window partitioning and linear grouping recurrent neural network to obtain the three-dimensional voxel features of the fourth stage; The final detection result acquisition module is used to process the three-dimensional voxel features of the fourth stage through the bird's-eye view backbone network and the detection head to obtain the final detection result.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the three-dimensional target detection method as described in any one of claims 1-7.