Point cloud processing method and device, automatic identification system and lidar

Abstract

A point cloud processing method and device, an automatic recognition system, and a lidar are disclosed. The point cloud processing method includes: acquiring a two-dimensional bird's-eye view corresponding to the point cloud; performing a first feature extraction based on the two-dimensional bird's-eye view to obtain a first feature map; and performing target detection on the first feature map to obtain a target detection result. This approach reduces hardware development costs while achieving efficient three-dimensional target detection.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to a point cloud processing method and device, an automatic identification system, and a lidar. Background Technology

[0002] Point cloud-based 3D target detection is one of the key aspects of realizing intelligent applications such as autonomous navigation. To improve 3D target detection capabilities, some existing 3D target detection methods perform feature extraction processing on point clouds.

[0003] However, the efficiency of existing 3D target detection methods is not ideal. This is because, compared to one-dimensional and two-dimensional data, 3D point clouds are more complex in feature extraction, resulting in huge computational overhead. Furthermore, due to factors such as non-uniform sampling in space, limited detection range, pose variations, and occlusion from external objects, the data structure of point clouds is more sparse and loose, and suffers from perspective distortion, which hinders feature extraction and consequently affects the accuracy of 3D target detection results.

[0004] To address the aforementioned issues, several methods have been proposed to convert point clouds into volumes. For example, some voxel-based 3D object detection methods divide the point cloud into multiple spatial cells and rasterize the points in each spatial cell to obtain voxels. Then, feature extraction and object detection are performed on the voxels.

[0005] Existing voxel-based 3D object detection methods include: VoxelNet (an end-to-end voxel object detection network) algorithm, VoxelGrid-type algorithms in PCL (The Point Cloud Library), SECOND (Sparsely Embedded Convolutional Detection) algorithm, PV-RCNN (PointVoxel-Region-Convolutional Neural-Networks) algorithm, etc.

[0006] However, since these voxel-based 3D object detection methods all require rasterization of point clouds, quantization errors occur, leading to information loss, reduced accuracy, and affecting the accuracy of 3D object detection results. Furthermore, for some methods with complex network architectures, feature extraction of 3D data already incurs significant computational overhead; rasterization of point clouds will further increase the computational load and algorithm complexity.

[0007] In summary, existing 3D target detection methods still suffer from low target detection efficiency. Summary of the Invention

[0008] In view of this, embodiments of this specification provide a point cloud processing method and device, an automatic recognition system, and a lidar, which can reduce hardware development costs while achieving efficient 3D target detection.

[0009] This specification provides a point cloud processing method, characterized by comprising:

[0010] Obtain a 2D bird's-eye view of the point cloud;

[0011] Based on the two-dimensional bird's-eye view, the first feature is extracted to obtain the first feature map;

[0012] Target detection is performed on the first feature map to obtain the target detection result.

[0013] Optionally, obtaining the two-dimensional bird's-eye view corresponding to the point cloud includes:

[0014] Obtain the depth image corresponding to the point cloud;

[0015] The depth image is subjected to a second feature extraction to obtain a second feature map;

[0016] The second feature map is subjected to perspective transformation to obtain the two-dimensional bird's-eye view.

[0017] Optionally, obtaining the two-dimensional bird's-eye view corresponding to the point cloud includes:

[0018] Obtain the depth image corresponding to the point cloud;

[0019] The depth image is subjected to second feature extraction in multiple stages, wherein the feature dimensions of the intermediate results of the second features in each stage are different;

[0020] In the multiple stages, at least one intermediate result of the second feature of a specified stage is obtained and subjected to perspective transformation to obtain the corresponding stage perspective view, so as to determine the two-dimensional bird's-eye view corresponding to the depth image.

[0021] Optionally, the step of obtaining at least one intermediate result of the second feature of a specified stage in the plurality of stages, performing perspective transformation to obtain a corresponding stage perspective view, and determining the two-dimensional bird's-eye view corresponding to the depth image, includes:

[0022] In the multiple stages, the intermediate result of the second feature of a specified stage is obtained and subjected to perspective transformation to obtain a corresponding stage perspective view, which serves as the two-dimensional bird's-eye view.

[0023] Optionally, the step of obtaining at least one intermediate result of the second feature of a specified stage in the plurality of stages, performing perspective transformation to obtain a corresponding stage perspective view, and determining the two-dimensional bird's-eye view corresponding to the depth image, includes:

[0024] In the multiple stages, intermediate results of the second feature of multiple specified stages are obtained, and perspective transformation is performed on each intermediate result of the second feature to obtain the corresponding multiple stage perspective views;

[0025] Based on the multiple stage perspective views, multiple two-dimensional bird's-eye views are determined;

[0026] The step of extracting the first feature based on the two-dimensional bird's-eye view to obtain the first feature map includes:

[0027] Based on multiple two-dimensional bird's-eye view images, a first feature is extracted to obtain the first feature map.

[0028] Optionally, the step of extracting the first feature based on multiple two-dimensional bird's-eye views to obtain the first feature map includes:

[0029] Perform first feature extraction on each of the two-dimensional bird's-eye views to obtain the corresponding first sub-feature map;

[0030] The first feature map is determined based on multiple first sub-feature maps.

[0031] Optionally, the step of extracting the first feature based on multiple two-dimensional bird's-eye views to obtain the first feature map includes:

[0032] At least one stage of first feature extraction is performed on each of the two-dimensional bird's-eye views;

[0033] According to the set selection stage, the corresponding first feature intermediate result is obtained; wherein, the feature dimension of the first feature intermediate result of the selection stage is determined; and the feature dimension of the first feature intermediate result corresponding to the same selection stage is the same.

[0034] The intermediate results of the first feature corresponding to the same selection stage are integrated to obtain the stage sub-feature results;

[0035] The first feature map is determined based on the stage sub-feature results.

[0036] Optionally, the step of obtaining a perspective transformation of at least one intermediate result of a second feature in a specified stage during the plurality of stages to obtain a corresponding stage perspective view includes:

[0037] In the plurality of stages, at least one intermediate result of the second feature in a specified stage is obtained and upsampled so that the size of the intermediate result of the second feature is consistent with the size of the depth image;

[0038] Based on the correspondence between the upsampled second feature intermediate result and the point cloud, a perspective transformation is performed on the upsampled second feature intermediate result to obtain the two-dimensional bird's-eye view.

[0039] Optionally, the two-dimensional bird's-eye view corresponding to the depth image is obtained in the following way:

[0040] Based on the depth image, determine the image to be processed;

[0041] The pixels in the image to be processed are subjected to perspective transformation to obtain the corresponding perspective transformation points. The pixels include values ​​of multiple feature dimensions.

[0042] Obtain the perspective transformation points to the preset grid, and select the values ​​of the perspective transformation points in the grid that are in the same feature dimension to obtain the characterization points, so as to characterize all perspective transformation points in the grid.

[0043] The two-dimensional bird's-eye view is generated based on the representation points corresponding to the grid.

[0044] Optionally, obtaining the two-dimensional bird's-eye view corresponding to the depth image includes:

[0045] The depth image is subjected to multi-stage second feature extraction using an encoder;

[0046] The encoder is constructed based on encoder parameters, and the output of the encoder is subjected to feature reconstruction and supervised learning to adjust the encoder parameters.

[0047] Optionally, the point cloud processing method further includes:

[0048] Obtain a second feature map corresponding to the depth image, wherein the second feature map is obtained by extracting a second feature from the depth image;

[0049] The second feature map is used for target classification and recognition to obtain the target classification result;

[0050] Based on the target classification results and the target detection results, the classification information of each point in the point cloud is determined.

[0051] Optionally, the step of performing target detection on the first feature map to obtain target detection results includes:

[0052] Obtain the attention distribution weights corresponding to the first feature map, wherein the attention distribution weights are calculated on the two-dimensional bird's-eye view through a preset attention learning mechanism to characterize the importance of each perspective transformation point in the two-dimensional bird's-eye view;

[0053] Based on the attention distribution weights corresponding to the first feature map, target detection is performed on the first feature map to obtain the target detection result.

[0054] This specification also provides a point cloud processing device, including:

[0055] The image acquisition unit is suitable for acquiring a two-dimensional bird's-eye view corresponding to a point cloud.

[0056] The feature processing unit is adapted to perform a first feature extraction based on the two-dimensional bird's-eye view to obtain a first feature map;

[0057] The target detection unit is adapted to perform target detection on the first feature map to obtain the target detection result.

[0058] This specification also provides an automatic identification system, comprising: a point cloud acquisition device and a data processing device, wherein the data processing device is connected to the point cloud acquisition device, wherein:

[0059] The point cloud acquisition device is suitable for acquiring point clouds;

[0060] The data processing device is adapted to perform the steps of the point cloud processing method described in any of the above embodiments to process the point cloud acquired by the point cloud acquisition device and obtain the target detection result.

[0061] This specification also provides a lidar system, including multiple transmitters, multiple detectors, and a data processing device, wherein:

[0062] The transmitter and the detector are corresponding; the transmitter is suitable for transmitting detection signals from multiple angles; the detector is suitable for acquiring echo signals from multiple angles.

[0063] The data processing device is adapted to perform data processing based on the detection signal of the transmitter and the echo signal of the corresponding detector to obtain a point cloud, and to execute the steps of the point cloud processing method described in any of the above embodiments.

[0064] The data processing method provided in the embodiments of this specification obtains a two-dimensional bird's-eye view from the point cloud. Then, a first feature is extracted based on the two-dimensional bird's-eye view, and the obtained first feature map is used for target detection to obtain the target detection result. As can be seen from the above scheme, on the one hand, by performing feature extraction and target detection on the two-dimensional bird's-eye view, the accuracy of the target detection result is ensured, thus eliminating the need for rasterization of the point cloud, avoiding quantization errors, and improving information completeness and accuracy. On the other hand, converting the three-dimensional point cloud into a two-dimensional image can reduce the complexity of feature processing and the amount of computation, resulting in faster calculation speed, less time consumption, and reduced computer memory usage, thereby improving computing performance. Furthermore, the data processing method provided in the embodiments of this specification can be adapted to more existing hardware platforms, saving hardware development costs and time, and has a wider range of application prospects. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of this specification, the drawings used in the description of the embodiments of this specification or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of a data processing method in one of the embodiments of this specification.

[0067] Figure 2 This is a schematic diagram of the stage division of a neural network layer in an embodiment of this specification.

[0068] Figure 3 This is a first feature extraction framework diagram of a two-dimensional bird's-eye view in an embodiment of this specification.

[0069] Figure 4 This is a schematic diagram of a residual building block in an embodiment of this specification.

[0070] Figure 5 This is a schematic diagram of a residual building block in one of the embodiments of this specification.

[0071] Figure 6 This is a schematic diagram of the stage division of a residual building block in an embodiment of this specification.

[0072] Figure 7 This is a flowchart of a method for obtaining a two-dimensional bird's-eye view corresponding to the depth image in an embodiment of this specification.

[0073] Figure 8 This is a flowchart of another method for obtaining a two-dimensional bird's-eye view corresponding to the depth image in an embodiment of this specification.

[0074] Figure 9 This is a flowchart illustrating one embodiment of the present specification for obtaining a two-dimensional bird's-eye view corresponding to the depth image.

[0075] Figure 10 This is a flowchart illustrating another method for obtaining a two-dimensional bird's-eye view corresponding to the depth image in an embodiment of this specification.

[0076] Figure 11 This is a flowchart of one of the depth image processing procedures described in the embodiments of this specification.

[0077] Figure 12 This is another depth image processing flowchart described in the embodiments of this specification.

[0078] Figure 13 This is another depth image processing flowchart described in the embodiments of this specification.

[0079] Figure 14 This is a flowchart illustrating the process of extracting first features from multiple two-dimensional bird's-eye views in multiple stages, as described in an embodiment of this specification.

[0080] Figure 15 This is a flowchart illustrating a different method for performing first feature extraction in multiple stages from multiple two-dimensional bird's-eye view images, as described in the embodiments of this specification.

[0081] Figure 16 This is a flowchart illustrating another embodiment of the present specification for obtaining a two-dimensional bird's-eye view corresponding to the depth image.

[0082] Figure 17 This is a flowchart illustrating the training process of an encoder in one of the embodiments of this specification.

[0083] Figure 18 This is another point cloud processing method in the embodiments of this specification.

[0084] Figure 19 This is a classification information diagram in one of the embodiments of this specification.

[0085] Figure 20 This is a structural block diagram of a point cloud processing device according to an embodiment of this specification.

[0086] Figure 21 This is a structural block diagram of an image acquisition unit in an embodiment of this specification.

[0087] Figure 22 This is a structural block diagram of another image acquisition unit in the embodiments of this specification.

[0088] Figure 23 This is a schematic diagram illustrating the correspondence between point cloud, depth image, and two-dimensional bird's-eye view in one embodiment of this specification. Detailed Implementation

[0089] As can be seen from the background section, existing 3D target detection methods still suffer from low target detection efficiency.

[0090] To address the aforementioned technical problems, this specification provides a flowchart of a data processing method, as illustrated in the embodiments below. Figure 1 As shown, the method may include the following steps:

[0091] S11, obtain the two-dimensional bird's-eye view corresponding to the point cloud.

[0092] In practical implementation, a point cloud acquisition device can collect the spatial pose of external objects in the environment through multiple wire bundles, thereby forming a point cloud. Each point in the point cloud can include angle information and distance information. Further, the angle information can include: the horizontal angle (i.e., azimuth angle) of each point in the point cloud relative to the point cloud acquisition device, and the vertical angle (i.e., pitch angle) of each point in the point cloud relative to the point cloud acquisition device. The distance information is the distance from each point in the point cloud to the point cloud acquisition device.

[0093] For ease of description and understanding, a coordinate system can be established with the point cloud acquisition device as the origin. See the following for details. Figure 23 As shown, an xyz coordinate system is established with the point cloud acquisition device as the origin o. The x-axis is the normal vector direction of the surface of the point cloud acquisition device L used for emitting the line bundle, the y-axis is the horizontal direction of the point cloud acquisition device L, and the z-axis is the vertical direction of the point cloud acquisition device L.

[0094] The angle between the projection of a point pt in the point cloud onto the xoy plane (ty1) and the y-axis is the horizontal angle α; the angle between the projection of point pt onto the yoz plane (ty2) and the y-axis is the vertical angle β; the distance between point pt and the point cloud acquisition device L is calculated, yielding the value d. Therefore, the angle information of point pt can include: the horizontal angle α and the vertical angle β; the distance information can be the value d.

[0095] Furthermore, the angle and distance information of each point can be represented using coordinates, with reference to... Figure 23 The three-dimensional coordinates of point pt can be (d, β, α).

[0096] In practice, the 3D point cloud perspective can be transformed into a 2D bird's-eye view based on the angle and distance information of each point in the point cloud. The coordinate system of the 2D bird's-eye view can be established by referring to the plane corresponding to the top-down view in the coordinate system of the point cloud acquisition device, which facilitates obtaining the correspondence between the 2D bird's-eye view and the point cloud. Then, each point in the point cloud is projected onto the coordinate system of the plane where the 2D bird's-eye view is located, thereby obtaining the corresponding 2D bird's-eye view.

[0097] For example, continue to refer to Figure 23 Since the normal vector of the surface used to emit the beam in the point cloud acquisition device L is along the x-axis, and the xoy plane corresponds to the top-down view of the point cloud acquisition device, a coordinate system for the two-dimensional bird's-eye view, namely the x"o"y" axis coordinate system, is established with reference to the xoy plane of the point cloud acquisition device. Based on the angle and distance information of point pt in the point cloud, point pt can be projected onto the x"o"y" axis coordinate system to obtain pixel py. The grayscale value of pixel py corresponds to the distance information d. Similarly, all points in the point cloud are projected onto the x"o"y" axis coordinate system to obtain the two-dimensional bird's-eye view.

[0098] Therefore, projecting the point cloud onto the coordinate system of a two-dimensional bird's-eye view from a top-down perspective can ensure that the size of objects is consistent and avoid perspective distortion, thereby improving the completeness and accuracy of information and thus enhancing the accuracy and reliability of the first feature extraction result.

[0099] In practical implementation, the resolution of the two-dimensional bird's-eye view can be determined by preset two-dimensional bird's-eye view resolution parameters. Furthermore, the two-dimensional bird's-eye view resolution parameters can be set according to the requirements of the point cloud acquisition device's wiring harness and image resolution. The range of values ​​for the two-dimensional bird's-eye view resolution parameters can be greater than the resolution determined by the angle resolution and the number of wiring harnesses of the point cloud acquisition device, in order to obtain a clear two-dimensional bird's-eye view.

[0100] In one optional example, if the image resolution requirement is 1920*1800 and the number of wires in the point cloud acquisition device is 64, then the resolution parameter of the two-dimensional bird's-eye view can be set to 64*1800. That is, the resolution of the two-dimensional bird's-eye view corresponding to the point cloud is 64*1800. Thus, the resolution of the two-dimensional bird's-eye view can be quickly determined based on the number of wires in the point cloud acquisition device.

[0101] In another optional example, the required image resolution is 1920*1800, the number of wires in the point cloud acquisition device is 64, and the angular resolution is 0.2°. Therefore, the range of values ​​for the 2D bird's-eye view resolution parameter is greater than 64 / 0.2 = 320. For example, the 2D bird's-eye view resolution parameter can be set to 600*1800, meaning the resolution of the 2D bird's-eye view corresponding to the point cloud is 600*1800. Thus, by determining the lower limit of the 2D bird's-eye view resolution parameter based on the number of wires and the angular resolution of the point cloud acquisition device, the clarity of the 2D bird's-eye view is ensured.

[0102] It is understood that the above embodiments are for illustrative purposes only. In actual applications, the corresponding resolution parameters of the two-dimensional bird's-eye view can be set according to specific scenarios and needs, and this specification does not impose any restrictions on this.

[0103] S12, based on the two-dimensional bird's-eye view, perform first feature extraction to obtain a first feature map.

[0104] In practical implementation, feature extraction can be performed according to the set feature extraction algorithm to obtain a feature map. To facilitate the differentiation of feature extraction for different application objects, terms such as "first" and "second" can be added. For example, the feature extraction algorithm used for a two-dimensional bird's-eye view can be called the first feature extraction algorithm, the feature extraction based on the two-dimensional bird's-eye view can be called the first feature extraction, and the resulting feature map can be called the first feature map.

[0105] It should be noted that in the description of this specification, terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with terms such as "first" and "second" may explicitly or implicitly include one or more of that feature. Furthermore, terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or indicate importance.

[0106] In specific implementation, the feature dimension of the extracted first feature map can be consistent with the feature dimension of the two-dimensional bird's-eye view, or it can be different from the feature dimension of the two-dimensional bird's-eye view, such as being larger than the feature dimension of the two-dimensional bird's-eye view; and the size of the first feature map can be consistent with the size of the two-dimensional bird's-eye view, or it can be different from the size of the two-dimensional bird's-eye view, such as being smaller than the size of the two-dimensional bird's-eye view.

[0107] Furthermore, the feature dimension transformation of the first feature map can be inversely proportional to the size change; that is, when the feature dimension of the first feature map increases, the size of the first feature map decreases; when the feature dimension of the first feature map decreases, the size of the first feature map increases.

[0108] In practical applications, a corresponding first feature extraction algorithm can be executed through a neural network model to extract the first feature from the two-dimensional bird's-eye view. It should be noted that, for ease of description, in this specification, the neural network model used for first feature extraction can be referred to as an object detector, and the parameters used to construct the object detector can be referred to as object detector parameters.

[0109] The specific content of the target detector parameters is related to the functions required by the first feature extraction algorithm. For example, if the first feature extraction algorithm needs to implement convolution functionality, i.e., convolutional feature extraction, then the target detector parameters may include planar filter (i.e., two-dimensional convolution kernel) parameters, strides parameters, padding parameters, the number of feature dimensions used in the convolution, etc., so that the constructed target detector can perform first feature extraction with convolution functionality on the two-dimensional bird's-eye view.

[0110] It is understood that the parameters of the first feature extraction algorithm and the corresponding target detector can be set according to the actual application scenario and requirements, and this manual does not impose specific restrictions on them.

[0111] It should be noted that, in some embodiments, the neural network in the neural network model can be divided into different neural network layers according to the functions it performs. For example, the neural network used to implement the convolution function can be divided into Convolutional Neural Networks (CNN) layers. For ease of description, the output of each neural network layer can be referred to as intermediate results, and the output of the neural network model can be referred to as feature maps.

[0112] Based on this, the neural network in the object detector can be divided into different neural network layers according to their functions, and the output of the object detector is the first feature map.

[0113] Depending on the specific application scenario and requirements, the neural network layers in the object detector can be modified, replaced, or added. For example, the size of the planar filters in the convolutional neural network layers of the object detector can be modified; another example is replacing the fully connected neural network (FCNN) layers in the object detector with convolutional neural network layers; yet another example is adding generative adversarial network (GAN) layers to the object detector. This specification does not impose specific limitations on the neural network layers included in the object detector.

[0114] In practice, the complexity and computational load of two-dimensional feature extraction are far less than those of three-dimensional feature extraction. Taking convolutional feature extraction as an example, existing three-dimensional object detection algorithms use cubic filters (i.e., three-dimensional convolutional kernels) to extract three-dimensional convolutional features from each voxel. As the number of three-dimensional convolutional feature extractions increases, the amount of data increases by at least a cubic multiple. However, if the data processing method provided in the embodiments of this specification is used, a planar filter can be used to extract two-dimensional convolutional features from the first feature map. Even though the number of two-dimensional convolutional feature extractions increases by at least a square multiple, the overall increase in the amount of data from two-dimensional convolutional feature extraction is still far less than that from three-dimensional convolutional feature extraction.

[0115] As the accuracy of point cloud acquisition improves, the data volume gap between 3D feature extraction and 2D feature extraction will become increasingly larger. Therefore, the data processing method provided in the embodiments of this specification greatly reduces the computational complexity and has a significant optimization in computational performance.

[0116] S13, Perform target detection on the first feature map to obtain the target detection result.

[0117] The target detection result may include at least one of target bounding box recognition result and target classification result. The target bounding box may include feature information such as the three-dimensional coordinates (dx, dy, dz), the three-dimensional dimensions (h, w, l), and the rotation angle (yaw) along the axis perpendicular to the horizontal plane.

[0118] In practice, based on the set target detection algorithm, target detection can be performed on the first feature map to obtain the target detection result. Using the first feature map as the target detection object has lower computational complexity and faster processing efficiency compared to using the point cloud as the target detection object.

[0119] Furthermore, the target detection algorithm can be anchor-based, anchor-free, or a combination of both to obtain target bounding box recognition and target classification results.

[0120] In one feasible example, for an anchor-box-based target detection algorithm, target candidate boxes can be identified in the first feature map, and then optimization and classification processes can be performed based on the identified target candidate boxes.

[0121] Specifically, anchor frames of different sizes can be preset based on the candidate categories of the target. For example, if the candidate categories of the target include vehicles, people and non-motorized vehicles, then anchor frames corresponding to vehicles, people and non-motorized vehicles are set. Since there are relative size differences between vehicles, people and non-motorized vehicles, there are also size differences between the anchor frames corresponding to vehicles, people and non-motorized vehicles, thus obtaining anchor frames of different sizes.

[0122] Furthermore, by identifying the connected regions in the first feature map, an envelope box can be obtained. Then, multiple preset anchor boxes are matched with each envelope box to filter out the anchor boxes that meet the conditions, thereby obtaining the target candidate box.

[0123] Then, based on the coordinates of each candidate bounding box, the overlapping area between two candidate bounding boxes is determined, and the matching degree between them is calculated. Based on the matching degree, the candidate bounding boxes are filtered to remove overlapping ones. Box regression is then performed on the remaining candidate bounding boxes to obtain the target bounding box recognition result, and classification is performed on the remaining candidate bounding boxes to obtain the target classification result.

[0124] In another feasible example, for an anchorless object detection algorithm, key points can be identified in the first feature map, and then target candidate boxes can be generated and classified based on the identified key points.

[0125] Specifically, at least one of the target center point or target boundary point in the first feature map can be identified first. If the target center point is identified, a target candidate box is generated based on the set size distance. If the target boundary point is identified, a target candidate box is generated based on the set size parameters. Then, box regression processing is performed on the target candidate boxes to obtain the target box recognition result, and the target candidate boxes are classified to obtain the target classification result.

[0126] It is understood that the target detection algorithm can be set, modified or adjusted according to the actual application scenario and needs, and the embodiments in this specification do not limit this.

[0127] To enable those skilled in the art to better understand the advantages of the methods provided in the embodiments of this specification, detailed descriptions are given in specific application scenarios.

[0128] Taking the VoxelNet algorithm as an example, it requires rasterizing the point cloud to form multiple voxels, and then performing 3D feature extraction and target prediction on each voxel. Specifically, if an xyz-axis coordinate system is established with the point cloud acquisition device as the origin, where the range of the x and z axes is (-60, +60) and the range of the y axis is (-40, +40), and the unit of the xyz-axis coordinate system is meters, a cubic space with a length and width of 120 (meters) and a height of 80 (meters) is obtained. In this cubic space, if the Voxel algorithm is used to rasterize the point cloud according to a grid unit of 1×1×1 (cubic decimeters), 1200×1200×80 voxels will be obtained.

[0129] However, by using the data processing method provided in the embodiments of this specification, point clouds can be converted into two-dimensional bird's-eye view images. For example, the point cloud in the aforementioned cubic space is converted into a two-dimensional bird's-eye view image with 64×1800 pixels. Compared to 1200×1200×80 voxels, there is a one-thousandth difference in the number of voxels. Obviously, the computational cost of the data processing method provided in the embodiments of this specification is far less than that of the VoxelNet algorithm, reducing computational overhead. In addition, it avoids the quantization error problem caused by rasterization processing, improving information integrity and accuracy.

[0130] Furthermore, since the point cloud is converted into a two-dimensional bird's-eye view, the first feature extraction is performed using two-dimensional features. However, the voxels in the VoxelNet algorithm are still three-dimensional data, therefore three-dimensional feature extraction is performed. Compared to three-dimensional feature extraction, two-dimensional feature extraction is more mature and widely used on hardware platforms, such as Field Programmable Gate Array (FPGA) chips and Application Specific Integrated Circuit (ASIC) chips. It has strong hardware support and a wide variety of hardware options to meet different hardware needs, eliminating the need for hardware development and saving time and costs.

[0131] Hardware support for 3D feature extraction is relatively weak, especially for improved 3D feature extraction algorithms developed to address the sparsity of point clouds, such as algorithms that use sparse convolutional neural network layers for 3D feature extraction. These improved 3D feature extraction algorithms are currently difficult to deploy on existing hardware platforms, requiring time and cost to develop new hardware.

[0132] In summary, the data processing method provided in the embodiments of this specification achieves several advantages. First, by extracting features and detecting targets from the two-dimensional bird's-eye view, the accuracy of target detection results is ensured, eliminating the need for rasterization of the point cloud and avoiding quantization errors, thus improving information completeness and accuracy. Second, converting the three-dimensional point cloud into a two-dimensional image reduces the complexity of feature processing and the amount of computation, resulting in faster computation speed, less time consumption, and reduced computer memory usage, thereby improving computing performance. Furthermore, the data processing method provided in the embodiments of this specification is compatible with more existing hardware platforms, saving hardware development costs and time, and has broader application prospects.

[0133] In practice, the object detector may include multiple stacked neural network layers. Through these stacked neural network layers, the first feature can be extracted accurately, thereby enhancing the performance of the object detector.

[0134] Depending on the parameters of the target detector, the feature dimension of the intermediate results output by each neural network layer in the target detector can remain unchanged from the feature dimension of its input object, or the feature dimension of the intermediate results output by each neural network layer can be greater than the feature dimension of its input object.

[0135] For example, if a convolutional neural network layer in a target detector uses four planar filters (i.e., the target detector parameters include four planar filter parameters corresponding to the convolutional neural network) to convolve the input object, and the feature dimension of the input object of this neural network layer is 3, then the feature dimension of the intermediate result output by this convolutional neural network layer is 4, which is greater than the feature dimension of the input object of this neural network layer.

[0136] For example, if a convolutional neural network layer in a target detector uses four planar filters to convolve the input object, and the feature dimension of the input object of the neural network layer is 4, then the feature dimension of the intermediate result output by the convolutional neural network layer is 4, which is equal to the feature dimension of the input object of the neural network layer.

[0137] Furthermore, the intermediate results output by each neural network layer can be regarded as an intermediate feature map. The size of the intermediate results can remain unchanged from the size of its input object, or the size of the intermediate results output by each neural network layer can be smaller than the size of its input object.

[0138] For example, when a convolutional neural network layer in an object detector convolves with an input object, if the input object is padded (i.e., the padding parameter in the object detector parameters corresponding to the convolutional neural network is not 0), the size of the intermediate result output by the convolutional neural network layer can be equal to the size of its input object; however, if the input object is not padded (i.e., the padding parameter in the object detector parameters corresponding to the convolutional neural network is 0), the size of the intermediate result output by the convolutional neural network layer is smaller than the size of its input object.

[0139] The input object for each neural network layer can be a two-dimensional bird's-eye view, or it can be an intermediate result of the output of the two-dimensional bird's-eye view processed by other neural network layers.

[0140] Furthermore, the change in feature dimension of the intermediate results of each neural network layer in the target detector can be inversely proportional to the change in size; that is, when the feature dimension of the intermediate result increases, the size of the intermediate result decreases; when the feature dimension of the intermediate result decreases, the size of the intermediate result increases.

[0141] In practical implementation, in order to facilitate the control of feature dimensions and to more quickly adjust the parameters of neural network layers according to feature dimension requirements, multiple neural network layers can be divided into different stages based on the feature dimensions of the intermediate results output by each neural network layer in the neural network model.

[0142] For example, such as Figure 2 The diagram illustrates a stage division of a neural network layer. In stage ST1, the intermediate outputs of neural network layers 11 to 1B all have a feature dimension of W1; in stage ST2, the intermediate outputs of neural network layers 21 to 2C all have a feature dimension of W2. Therefore, from a stage perspective, W1 is the feature dimension of the output of stage ST1, and W2 is the feature dimension of the output of stage ST2. W1 and W2 are not the same.

[0143] Optionally, the stages can be sorted and connected according to the size of the feature dimension. For example, if W1 is greater than W2, then the input end of stage ST1 is the end closer to the input of the neural network model, and the output end of stage ST2 is the end closer to the output of the neural network model.

[0144] To facilitate the description of the output results at each stage, the output results of the aforementioned stages can be referred to as feature intermediate results, for example, such as Figure 2 As shown, the number of features in the intermediate feature result SC1 of stage ST1 is W1, and the number of features in the intermediate feature result SC2 of stage ST2 is W2. Accordingly, in the target detector, the output of each stage can be referred to as the first intermediate feature result.

[0145] For example, such as Figure 3 The diagram shown is a framework diagram for the first feature extraction of a two-dimensional bird's-eye view, combined with... Figure 1 After obtaining a two-dimensional bird's-eye view from the degree image corresponding to the point cloud, the first feature extraction in three stages can be performed on the two-dimensional bird's-eye view, namely... Figure 3 The algorithm involves stages 1 through 3 to obtain a first feature map. The size of the intermediate first feature map in stages 1 through 3 gradually decreases while the feature dimension gradually increases. The intermediate first feature map in stage 3 is then output as the first feature map. Object detection can then be performed on this first feature map to obtain the object detection result.

[0146] In practice, some useful information may gradually fade or be filtered out in stacked neural network layers. Therefore, too many neural network layers can easily cause gradient explosion or gradient vanishing problems, which are detrimental to model training. To avoid these problems, the object detector can adopt a residual network structure, thereby mapping some information to subsequent features during the first feature extraction process. This increases the model gradient, facilitates the construction of deeper object detectors, and can highlight subtle changes in features, which is beneficial for model training and learning.

[0147] Specifically, the target detector may include residual building blocks, such as Figure 4 The diagram shows a residual building block. In the residual building block BK1, the input object a is divided into two paths. One path continues to input the subsequent connected neural network layers 1 to A, while the other path skips A neural network layers and is superimposed with the intermediate result b output by the skipped neural network layer A to obtain the superimposed result c, which is the output of the residual building block BK1.

[0148] Where A is a positive integer; the input object a can be a two-dimensional bird's-eye view, or it can be an intermediate result obtained by processing the two-dimensional bird's-eye view by other neural network layers; the output of the residual building block BK1 can be used as the result of the target detector (i.e. the first feature map) or it can be output to other neural network layers for subsequent processing.

[0149] In practical applications, the target detector may include other neural network layers. Depending on the actual situation and requirements, the residual building block can be connected to other neural network layers in the target detector. Furthermore, the type of neural network layer in the residual building block can be set according to functional requirements. For example, if the residual building block is used for two-dimensional convolutional feature extraction, it may include convolutional neural network layers, batch normalization (BN) layers, and nonlinear mapping layers to achieve convolutional functionality.

[0150] In specific implementations, the residual building block may include multiple neural network layers with the same function, for example, such as Figure 5 The diagram shows a specific schematic of a residual building block. The residual building block BK2 includes three convolutional neural network layers, three batch normalization layers, and three nonlinear mapping layers. The input object d is divided into two paths. One path performs the first feature extraction in the order of convolutional neural network layer 1, batch normalization layer 1, nonlinear mapping layer 1, convolutional neural network layer 2, batch normalization layer 2, nonlinear mapping layer 2, convolutional neural network layer 3, batch normalization layer 3, and nonlinear mapping layer 3. The other path skips 9 neural network layers and then superimposes the intermediate result e output by nonlinear mapping layer 3 to obtain the superimposed result f, which is used as the result of the residual building block BK2.

[0151] The input object d can be a two-dimensional bird's-eye view, or it can be an intermediate result obtained by processing the two-dimensional bird's-eye view by other neural network layers. The output of the residual building block BK2 can be used as the result of the target detector (i.e., the first feature map) or it can be output to other neural network layers for further processing.

[0152] It is understood that the embodiments in this specification do not impose any restrictions on the connection relationship of the residual building blocks in the target detector, the type and number of neural networks contained in the residual building blocks, or the connection relationship of each neural network in the residual building blocks.

[0153] In practical implementation, the target detector may include multiple residual building blocks. The parameter types and values ​​of each residual building block are set according to actual requirements. Correspondingly, the output results of each residual building block (such as...) Figure 3 The result c after superposition and Figure 4 The feature dimensions of the result f) after overlay may be the same or different. Understandably, in practical applications, the appropriate number of residual building blocks can be set according to actual needs, and this specification does not impose specific restrictions on this.

[0154] In practical implementation, the above stages and residual building blocks can also be combined, such as Figure 6 The diagram illustrates a stage division of residual building blocks. Based on the feature dimensions of the output results of each residual building block, multiple residual building blocks are divided into different stages. In stage ST3, the feature dimensions of the output results of residual building blocks 11 to 1D are all W3, while in stage ST2, the feature dimensions of the output results of residual building blocks 21 to 2E are all W4. Therefore, from a stage perspective, W3 is the feature dimension of the intermediate results of stage ST3, and W4 is the feature dimension of the intermediate results of stage ST4. W3 and W4 are not the same.

[0155] In a feasible example, taking the object detector using ResNet (deep residual network) as an example, ResNet50 can be divided into three stages. After the two-dimensional bird's-eye view goes through the three stages, a total of 50 residual neural network processing processes are performed, and a first feature map with a predetermined number of feature dimensions is output.

[0156] In practice, the more layers a neural network has, the greater the computational overhead and the slower the object detector's processing speed. Conversely, if the number of neural network layers is too small, the object detector's processing results will be underfitted. Therefore, the number of neural network layers in the object detector is set according to the speed and processing result requirements to ensure the object detector's fitting performance and speed.

[0157] In a feasible example, taking the object detector using ResNet for first feature extraction as an example, the deeper the ResNet structure, the more layers the neural network has. For example, ResNet101 has more layers than ResNet18. In theory, the deeper the ResNet structure, the better the performance. However, if the ResNet structure is too deep, it will lead to slow operation (for example, ResNet101 is much slower than ResNet18). If the ResNet structure is not deep enough, it will be difficult to achieve a good fit (for example, ResNet18 is more likely to fail to fit than ResNet101). In this case, choosing a ResNet with an appropriate structural depth (such as ResNet50, which has a faster operation speed than ResNet101 and a higher probability of achieving a good fit than ResNet18) can ensure that the processing speed of the object detector meets the requirements, while also ensuring the fitting performance of the object detector.

[0158] In practice, the number of neural network layers can be set to a multiple of the initial feature dimension of the two-dimensional bird's-eye view. For example, if the initial feature dimension of the two-dimensional bird's-eye view is 4, which includes the x-coordinate, y-coordinate, z-coordinate, and reflectivity r, then multiples of 4 such as 16, 64, 192, and 1000 can be set.

[0159] In one specific implementation, during the process of directly converting a point cloud into a top-down 2D bird's-eye view, some points overlap, leading to a reduction in information and hindering feature extraction. However, in practical applications, point cloud acquisition devices are more likely to capture information from a forward-looking perspective. Therefore, to obtain richer information, improve data quality and feature extraction effectiveness, and facilitate more accurate target detection results, the point cloud can first be converted into a forward-looking depth image. After feature extraction from the depth image, perspective transformation and subsequent operations can be performed.

[0160] For example, such as Figure 7An example method for obtaining a two-dimensional bird's-eye view corresponding to the depth image is shown. According to... Figure 7 The method shown includes the following steps:

[0161] S71, Obtain the depth image corresponding to the point cloud.

[0162] S72, perform second feature extraction on the depth image to obtain a second feature map.

[0163] S73, perform perspective transformation on the second feature map to obtain the two-dimensional bird's-eye view.

[0164] In practice, based on the set depth imaging algorithm, the angle and distance information of each point in the point cloud can be converted into the pixel information of each pixel in the depth image, thereby converting the three-dimensional point cloud acquired by the point cloud acquisition device into a two-dimensional depth image (Range Image).

[0165] The angle information is used to determine the pixel in the depth image that corresponds to the point cloud; the distance information is used to determine the grayscale value of each pixel in the depth image.

[0166] In practice, the coordinate system of the depth image can be established by referring to the plane corresponding to the forward view in the coordinate system of the point cloud acquisition device, which makes it easier to obtain a depth image with richer information.

[0167] For example, continue to refer to Figure 23 Since the normal vector of the surface used to emit the beam in the point cloud acquisition device L is along the x-axis, and the yoz plane corresponds to the forward-looking view of the point cloud acquisition device, converting the point cloud into an image on the yoz plane yields rich information. Therefore, a coordinate system for the depth image, i.e., the x'o'y' axis coordinate system, can be established with reference to the yoz plane of the point cloud acquisition device. Point pt is then converted into a pixel px in the x'o'y' axis coordinate system, where the grayscale value of pixel px corresponds to the distance information of pt. This process is repeated to map all points in the point cloud to pixels in the x'o'y' axis coordinate system, thus obtaining the depth image.

[0168] In specific implementation, the depth imaging algorithm can be set according to the actual scene and needs. For example, the Range Image related class algorithm in PCL can be used to convert the point cloud into the corresponding depth image.

[0169] It is understood that the above-described depth imaging algorithm is only an example. In actual application of the embodiments of this specification, the depth imaging algorithm can be set and modified according to the actual scenario and needs. The embodiments of this specification do not limit this.

[0170] In specific implementations, the resolution of the depth image can be determined by preset depth image resolution parameters. Furthermore, the depth image resolution parameters can be set according to the wiring harness and image resolution requirements of the point cloud acquisition device. The value range of the depth image resolution parameters can be greater than the resolution determined by the angular resolution and the number of wiring harnesses of the point cloud acquisition device, in order to obtain a clear depth image.

[0171] In one optional example, if the required image resolution is 1920*1800 and the number of wires in the point cloud acquisition device is 64, then the depth image resolution parameter can be set to 64*1800, that is, the resolution of the depth image corresponding to the point cloud is 64*1800. Thus, the resolution of the depth image can be quickly determined based on the number of wires in the point cloud acquisition device.

[0172] In another optional example, the required image resolution is 1920*1800, the number of cabling in the point cloud acquisition device is 64, and the angular resolution is 0.2°. Therefore, the value range of the depth image resolution parameter is greater than 64 / 0.2 = 320. For example, the depth image resolution parameter can be set to 600*1800, meaning the resolution of the depth image corresponding to the point cloud is 600*1800. Thus, by determining the lower limit of the image resolution parameter based on the number of cabling and the angular resolution of the point cloud acquisition device, the clarity of the depth image is ensured.

[0173] It is understood that the above embodiments are for illustrative purposes only. In actual applications, the corresponding depth image resolution parameters can be set according to specific scenarios and needs, and this specification does not impose any restrictions on this.

[0174] In specific implementation, the second feature is extracted according to the set second feature extraction algorithm. The feature dimension of the extracted second feature map can be consistent with the feature dimension of the depth image, or it can be different from the feature dimension of the depth image, such as being larger than the feature dimension of the depth image; and the size of the second feature map can be consistent with the size of the depth image, or it can be different from the size of the depth image, such as being smaller than the size of the depth image.

[0175] Furthermore, the feature dimension transformation of the second feature map can be inversely proportional to the size change; that is, when the feature dimension of the second feature map increases, the size of the second feature map decreases; when the feature dimension of the second feature map decreases, the size of the second feature map increases.

[0176] In practical applications, a corresponding second feature extraction algorithm can be executed through a neural network model to extract second features from the depth image. It should be noted that, for ease of description, the neural network model used for second feature extraction in this specification can be called an encoder, and the parameters used to construct the encoder can be called encoder parameters.

[0177] The specific content of the encoder parameters is related to the functions required by the second feature extraction algorithm. For example, if the second feature extraction algorithm needs to implement convolution, the encoder parameters may include planar filter parameters, stride parameters, padding parameters, and the number of feature dimensions used in the convolution, so that the constructed encoder can perform second feature extraction with convolution functionality on the depth image.

[0178] It is understood that the second feature extraction algorithm and the corresponding encoder parameters can be set according to the actual application scenario and requirements, and this manual does not impose specific restrictions on them.

[0179] As can be seen from the description of the target detector section above, the neural network in the encoder can be divided into different neural network layers according to the functions it performs, and the output of the encoder is the second feature map.

[0180] Depending on the specific application scenario and requirements, the neural network layers in the encoder can be modified, replaced, or added. For details, please refer to the description of the relevant part of the target detector, which will not be repeated here.

[0181] Optionally, to reduce the complexity of the algorithm architecture and improve its tunability, the encoder and the target detector may include the same neural network architecture for their respective feature extraction. For example, the encoder may be divided into stages; or, for example, the encoder may employ a residual network structure. Specific details can be found in the relevant sections above and will not be repeated here.

[0182] It should be noted that although the encoder and the target detector may have at least partially the same neural network architecture, for the same neural network architecture, the parameter types and parameter values ​​used by the encoder and the target detector may be different depending on the actual application scenario and requirements. This specification does not impose specific restrictions on this.

[0183] In practice, because depth images use a forward-looking perspective, external objects exhibit perspective relationships, and the size of each object varies. For example, a truck at a distance may be smaller than a car at a close distance, and the actual relative size relationship between them cannot be directly obtained from the depth image. Furthermore, depth images suffer from perspective distortion, meaning the shape of an object is deformed. Correspondingly, the second feature map obtained from the depth image also suffers from scale inconsistencies and perspective distortion.

[0184] Therefore, a perspective transformation can be performed on the second feature map to convert the forward view to a top view, making the objects in the image consistent in size and correcting perspective distortion. This is beneficial for more accurate feature extraction in the future. Moreover, since it is a perspective transformation between two-dimensional images, the computational load is small, effectively ensuring data processing efficiency.

[0185] Specifically, based on the correspondence between each point in the point cloud and each pixel in the depth image, and the correspondence between each pixel in the depth image and each pixel in the second feature image, each pixel in the second feature image can be projected onto the coordinate system of the top-view perspective to obtain a two-dimensional bird's-eye view.

[0186] For example, continue to refer to Figure 23 Based on the correspondence between each point in the point cloud and each pixel in the depth image, and the relationship between each pixel in the depth image and the second feature map ( Figure 23 By establishing the correspondence between pixels in the point cloud (not shown), the correspondence between point pt and pixels in the second feature map can be determined. Based on the angle and distance information of point pt in the point cloud, the pixels in the second feature map can be projected onto the x”o”y” coordinate system to obtain the pixels of the two-dimensional bird's-eye view. The grayscale values ​​of the pixels in the second feature map correspond to the grayscale values ​​of the two-dimensional bird's-eye view. This process is repeated to project all pixels in the second feature map onto the x”o”y” coordinate system, thus obtaining the two-dimensional bird's-eye view.

[0187] In practice, to improve the efficiency of perspective transformation, the second feature map can be upsampled so that the size of the second feature map is consistent with the size of the depth image. This allows for the direct acquisition of the correspondence between the point cloud and the second feature map, thereby improving the processing speed.

[0188] Furthermore, upsampling can be achieved using interpolation algorithms, which may include bilinear interpolation, nearest neighbor interpolation, etc.

[0189] For example, taking the xOy coordinate system as an example, if a new point P3 is to be inserted between points P1(x1, y1) and P2(x2, y2), the interpolation values ​​on the x and y axes can be calculated respectively. Specifically, the value of x3 is calculated based on the distance of point Px on the x-axis relative to x1 and x2. Similarly, the value of y3 is calculated based on the distance of point Py on the y-axis relative to y1 and y2, thus obtaining the coordinates (x3, y3) of point P3.

[0190] It is understood that the above examples are for illustrative purposes only. In specific applications, appropriate methods can be selected to implement upsampling according to the application scenario and requirements. This manual does not impose specific restrictions on this.

[0191] Figure 8 A flowchart illustrating a method for obtaining a two-dimensional bird's-eye view corresponding to the depth image in another specific embodiment of this specification is shown. Figure 8 The method shown includes the following steps:

[0192] S81, Obtain the depth image corresponding to the point cloud.

[0193] The method for acquiring depth images can be referred to the descriptions in the relevant sections above, and will not be repeated here.

[0194] S82, perform second feature extraction on the depth image in multiple stages, wherein each stage contains different feature dimensions.

[0195] S83, in the plurality of stages, at least one intermediate result of the second feature of a specified stage is obtained and subjected to perspective transformation to obtain the corresponding stage perspective view, so as to determine the two-dimensional bird's-eye view corresponding to the depth image.

[0196] Specifically, in the multiple stages, a second feature intermediate result of a specified stage can be obtained and subjected to perspective transformation to obtain a corresponding stage perspective view, which serves as the two-dimensional bird's-eye view.

[0197] For example, Figure 9 It indicates Figure 8 A preferred embodiment of the method. In this preferred embodiment, a second feature extraction comprising three stages can be performed on the depth image, namely... Figure 9 The process involves stages 4 through 6, where the size of the first feature intermediate results in stages 4 through 6 gradually decreases while the feature dimension gradually increases. From these three stages, the second feature intermediate result of the last stage, i.e., stage 6, is selected and subjected to perspective transformation to obtain a corresponding stage perspective view, which serves as the two-dimensional bird's-eye view. The implementation method of the perspective transformation can be found in the aforementioned related content and will not be elaborated upon here.

[0198] For example, Figure 10It indicates Figure 8 Another preferred embodiment of the method. In this preferred embodiment, the depth image can be subjected to a three-stage second feature extraction, namely... Figure 10 The process involves stages 4 through 6, where the size of the intermediate first feature results in stages 4 through 6 gradually decreases while the feature dimension gradually increases. From these three stages, the intermediate second feature result of stage 5 is selected and subjected to perspective transformation to obtain a corresponding stage perspective view, which serves as the two-dimensional bird's-eye view. Alternatively, multiple intermediate second feature results can be obtained from the multiple stages, and perspective transformation can be performed on each of these intermediate results to obtain multiple stage perspective views. Then, based on these multiple stage perspective views, the two-dimensional bird's-eye view is determined, and based on this view, the first feature is extracted to obtain the first feature map.

[0199] The multiple stage perspective views and multiple two-dimensional bird's-eye view views can have a one-to-one correspondence; or they can have a many-to-one relationship, meaning that at least two stage perspective views are integrated to form a two-dimensional bird's-eye view. The integration method can include overlay, fusion, etc. Furthermore, the implementation method of perspective transformation can be found in the above-mentioned related content and will not be repeated here.

[0200] For example, multiple perspective views of the aforementioned stages can be used as two-dimensional bird's-eye views. Specifically, such as... Figure 11 The diagram shown is a flowchart of one type of depth image processing, combined with... Figure 8 The second feature extraction can be performed on the depth image in three stages, namely... Figure 11 The process involves stages 4 through 6, where the size of the first feature intermediate results gradually decreases while the feature dimension gradually increases. The second feature intermediate results of stages 4 and 5 are sequentially acquired within these three stages and subjected to perspective transformation to obtain two corresponding stage perspective views, namely stage perspective view P1 and stage perspective view P2, which are then used as two-dimensional bird's-eye view Q1 and two-dimensional bird's-eye view Q2, respectively.

[0201] For example, multiple perspective views of the aforementioned stages can be merged to obtain a two-dimensional bird's-eye view. Specifically, such as... Figure 12 The diagram shown is another flowchart of the depth image processing described above, combined with... Figure 8 The second feature extraction can be performed on the depth image in three stages, namely... Figure 12The diagram is divided into stages 4 to 6, where the size of the first feature intermediate results in stages 4 to 6 gradually decreases while the feature dimension gradually increases. The second feature intermediate results of stages 4 and 5 are obtained and subjected to perspective transformation to obtain two corresponding stage perspective views, namely stage perspective view P3 and stage perspective view P4. These are then integrated to obtain a two-dimensional bird's-eye view Q3.

[0202] In a specific implementation, if multiple two-dimensional bird's-eye view images are obtained based on multiple stage perspective views, then a first feature extraction can be performed on each of the two-dimensional bird's-eye view images to obtain a corresponding first sub-feature image; and the first feature image is determined based on the multiple first sub-feature images.

[0203] In this embodiment, the first feature can be extracted sequentially according to a specified order, or the first feature can be extracted separately. This specification does not impose any restrictions on this.

[0204] For example, continue to refer to Figure 11 ,according to Figure 11 The execution order shown in ① and ② is as follows: First, the first feature is extracted from the two-dimensional bird's-eye view Q1 to obtain the first sub-feature map R1. Then, the first feature is extracted from the stage perspective view Q2 to obtain the first sub-feature map R2. Finally, the first sub-feature maps R1 and R2 are integrated to obtain the first feature map T1. The integration method may include overlay, fusion, etc.

[0205] For example, refer to Figure 13 This is another flowchart of the depth image processing procedure, which is related to... Figure 11 The similarity lies in the fact that, in the three stages, the intermediate results of the second feature in stages 4 and 5 are obtained sequentially, and perspective transformation is performed to obtain two corresponding stage perspective views, namely stage perspective view P5 and stage perspective view P6, which are respectively used as two-dimensional bird's-eye view Q4 and two-dimensional bird's-eye view Q5; and Figure 11 The difference lies in that, after obtaining the two-dimensional bird's-eye view Q4 and the two-dimensional bird's-eye view Q5, feature transformations are performed respectively to obtain the first sub-feature map R3 and the first sub-feature map R4. Then, the first sub-feature maps R3 and R4 are integrated to obtain the first feature map T3.

[0206] Furthermore, the implementation method of the first feature extraction can be referred to the above-mentioned content, and will not be repeated here.

[0207] In specific implementation, if a two-dimensional bird's-eye view is determined based on multiple stage perspective views, then a first feature extraction is performed on the two-dimensional bird's-eye view, for example, such as... Figure 12As shown, the first feature is extracted from the two-dimensional bird's-eye view Q3 to obtain the first feature map T2. The process of the first feature extraction can be referred to the relevant content above, and will not be repeated here.

[0208] In existing applications, target detection is usually performed only on the output of the neural network model (i.e., feature map). Since a lot of useful information may be lost during feature extraction, it will increase the detection difficulty for objects that are far away or small in size, i.e., the detection rate will decrease. By obtaining intermediate results of second features at multiple specified stages, information from different stages can be fused to compensate for useful information, thereby improving the detection rate.

[0209] In specific implementation, in order to further improve the target detection rate, if there are multiple two-dimensional bird's-eye view images, at least one stage of first feature extraction can be performed on each of the two-dimensional bird's-eye view images; then, according to the set selection stage, the corresponding first feature intermediate result is obtained, wherein the feature dimension of the selection stage is determined, and the feature dimension of the first feature intermediate result corresponding to the same selection stage is the same; then, the first feature intermediate results corresponding to the same selection stage are integrated to obtain stage sub-feature result, and the first feature map is determined according to the stage sub-feature result.

[0210] For example, such as Figure 14 The diagram shown is a flowchart of a multi-stage first feature extraction process using multiple two-dimensional bird's-eye views, combined with... Figure 8 Three two-dimensional bird's-eye view images were obtained, namely two-dimensional bird's-eye view image Q6, two-dimensional bird's-eye view image Q7 and two-dimensional bird's-eye view image Q8, and the first feature extraction was performed in three stages respectively.

[0211] Furthermore, to reduce structural complexity and development workload, multiple two-dimensional bird's-eye view images undergo a unified first feature extraction. For example, the two-dimensional bird's-eye view images Q6, Q7, and Q8 correspond to the first feature extraction in three stages, namely stages 11 to 13.

[0212] In stages 11 through 13, the feature dimensions gradually increase, and the size of the intermediate result of the first feature gradually decreases in response to the increasing feature dimensions. Stage 12 is the designated selection stage.

[0213] When the first feature extraction of the two-dimensional bird's-eye view Q6 is performed in stages 11 to 13, the first sub-feature map R5 is obtained.

[0214] When the two-dimensional bird's-eye view Q7 is subjected to the first feature extraction in stages 11 to 13, it proceeds to stage 12, obtains the intermediate result of the first feature of the two-dimensional bird's-eye view Q6 in stage 12, integrates it with the intermediate result of the first feature of the two-dimensional bird's-eye view Q7 in stage 12 to obtain the stage sub-feature result of the two-dimensional bird's-eye view Q7, and performs the first feature extraction in stage 13 to obtain the first sub-feature map R6.

[0215] When the 2D bird's-eye view Q8 is subjected to the first feature extraction in stages 11-13, the process proceeds to stage 12. The intermediate first feature result of 2D bird's-eye view Q7 in stage 12 is obtained and integrated with the intermediate first feature result of 2D bird's-eye view Q8 in stage 12 to obtain the stage sub-feature result of 2D bird's-eye view Q8. Then, the first feature extraction in stage 13 is performed to obtain the first sub-feature map R7. Finally, the first feature map T4 is determined based on the first sub-feature maps R5 to R7.

[0216] For example, such as Figure 15 The diagram shown is a flowchart of a multi-stage first feature extraction process using multiple two-dimensional bird's-eye views, combined with... Figure 8 Three two-dimensional bird's-eye view images are obtained, namely two-dimensional bird's-eye view image Q9, two-dimensional bird's-eye view image Q10 and two-dimensional bird's-eye view image Q11, and the first feature extraction is performed in three stages respectively. Specifically, the two-dimensional bird's-eye view images Q9, Q10 and Q11 correspond to the first feature extraction in the three stages, namely stages 21 to 23.

[0217] In stages 21-23, the feature dimensions gradually increase, and the size of the intermediate result of the first feature gradually decreases in response to the increasing feature dimensions. Stage 22 is the designated selection stage.

[0218] The first feature extraction in stages 21 to 23 is performed on the two-dimensional bird's-eye view Q9 to obtain the first sub-feature map R8.

[0219] When the two-dimensional bird's-eye view Q10 is performing the first feature extraction in stages 21 to 23, it proceeds to stage 22, obtains the intermediate result of the first feature of the two-dimensional bird's-eye view Q9 in stage 22, integrates it with the intermediate result of the first feature of the two-dimensional bird's-eye view Q10 in stage 22 to obtain the stage sub-feature result of the two-dimensional bird's-eye view Q10, and performs the first feature extraction in stage 13 to obtain the first sub-feature map R9.

[0220] When the two-dimensional bird's-eye view Q11 is performing the first feature extraction in stages 21 to 23, it proceeds to stage 22, where the intermediate first feature results of the two-dimensional bird's-eye view Q9 and the intermediate first feature results of the two-dimensional bird's-eye view Q10 in stage 22 are obtained. These results are then integrated with the intermediate first feature results of the two-dimensional bird's-eye view Q11 in stage 22 to obtain the stage sub-feature results of the two-dimensional bird's-eye view Q11. The first feature extraction in stage 23 is then performed to obtain the first sub-feature map R10.

[0221] Then, the first feature map T5 is determined based on the first sub-feature map R8, the first sub-feature map R9, and the first sub-feature map R10.

[0222] It is understood that the above embodiments are merely illustrative. In practical applications, one or more selection stages can be set according to actual needs, the number of stages, and the obtained two-dimensional bird's-eye view. The number of prior first feature intermediate results corresponding to the same selection stage can also be set. This specification does not impose any limitations on these aspects.

[0223] Furthermore, the first feature can be extracted from multiple two-dimensional bird's-eye views in the order of their generation, for example, as... Figure 14 As shown, if two-dimensional bird's-eye view Q6, two-dimensional bird's-eye view Q7 and two-dimensional bird's-eye view Q8 are generated in sequence, then two-dimensional bird's-eye view Q6 will undergo the first feature extraction first, two-dimensional bird's-eye view Q7 will undergo the first feature extraction second, and two-dimensional bird's-eye view Q8 will undergo the first feature extraction last.

[0224] In specific implementation, to quickly obtain the correspondence between the intermediate result of the second feature and the points in the point cloud, at least one specified intermediate result of the second feature is upsampled in the multiple stages to make the size of the intermediate result of the second feature consistent with the size of the depth image; then, based on the correspondence between the upsampled intermediate result of the second feature and the point cloud, perspective transformation is performed on the upsampled intermediate result of the second feature to obtain the two-dimensional bird's-eye view. The implementation method of upsampling can be referred to the description in the relevant sections above, and will not be repeated here.

[0225] In practice, the perspective transformation process can be optimized to filter the data, retaining more useful information and effectively reducing the amount of data. Specifically, for example... Figure 16 The diagram shown is a flowchart for another method of obtaining a two-dimensional bird's-eye view corresponding to the depth image. The two-dimensional bird's-eye view corresponding to the depth image can be obtained in the following ways:

[0226] S161, determine the image to be processed based on the depth image. The image to be processed can be any one of the following: a depth image, a second feature map, an upsampled second feature map, an intermediate result of the second feature map, and an upsampled intermediate result of the second feature map.

[0227] S162, perform perspective transformation on the pixels in the image to be processed to obtain corresponding perspective transformation points. The pixels include values ​​in multiple dimensions. The dimensions of the pixels are the feature dimensions.

[0228] S163, obtain the perspective transformation points of the perspective transformation to the preset grid, and select the values ​​of the perspective transformation points in the grid that are in the same dimension to obtain the characterization points to characterize all perspective transformation points in the grid.

[0229] The grid refers to the area used for perspective transformation optimization and generating a 2D bird's-eye view. The size of the grid used for perspective transformation can be set according to actual needs. By removing perspective transformation points outside the grid area, effective data filtering can be achieved.

[0230] In practice, a grid may contain multiple perspective transformation points. For example, there may be W perspective transformation points, each of which includes values ​​in v dimensions. For each grid, the values ​​of the perspective transformation points in the same dimension can be processed by maximizing or averaging, thereby optimizing the perspective transformation.

[0231] For example, each perspective transformation point can include values ​​in four feature dimensions, namely the x-coordinate, y-coordinate, z-coordinate, and reflectance r. If a grid contains multiple perspective transformation points, the maximum value of each perspective transformation point in the x-coordinate, y-coordinate, z-coordinate, and reflectance r can be selected to obtain the representation point of the grid.

[0232] S164, Generate the two-dimensional bird's-eye view based on the representation points corresponding to the grid.

[0233] By adopting the above scheme, the perspective transformation points are effectively filtered using a grid, thereby reducing the amount of data and ensuring uniform pixel distribution in the two-dimensional bird's-eye view, thus optimizing the perspective transformation.

[0234] In a specific implementation, before performing multi-stage second feature extraction on the depth image through the encoder, the encoder can be trained and backpropagation algorithms can be used to adjust the encoder parameters to ensure that the encoder has converged and can accurately extract the second features.

[0235] For example, the encoder's output is subjected to feature reconstruction and supervised learning to adjust the encoder parameters. For ease of description, the neural network model used for feature reconstruction and supervised learning can be called a decoder, and the parameters used to construct the decoder network can be called decoder parameters.

[0236] Feature reconstruction can be performed according to the set feature reconstruction algorithm. The specific content of the decoder parameters is related to the functions that the feature reconstruction algorithm needs to achieve. For example, if the feature reconstruction algorithm needs to implement deconvolution, the decoder parameters may include planar filter parameters, stride parameters, padding parameters, the number of feature dimensions used in convolution, etc., so that the constructed target detector can perform feature reconstruction with deconvolution function on the two-dimensional bird's-eye view.

[0237] Based on the above description of the target detector, the neural network in the decoder can be divided into different neural network layers according to their functions, and the output of the decoder is the third feature map.

[0238] In practice, the second feature map can be reconstructed in multiple stages to obtain a third feature map, which is then used to adjust parameters. Specifically, the training image is input into the encoder; the encoder performs multi-stage second feature extraction on the training image; the decoder performs multi-stage feature reconstruction on the result output by the encoder (i.e., the second feature map) to obtain a third feature map; target classification and recognition are performed on the third feature map, such as classifying each position in the third feature map, where the category may include vehicle, person, non-motorized vehicle, etc.

[0239] Then, the identified target classification result is compared with the actual target classification result of the training image, and the loss value is calculated by the set loss function. If the loss value meets the adjustment conditions, the encoder parameters are adjusted by the backpropagation algorithm.

[0240] In existing model training, neural network models may contain cross-layer connections, meaning that a certain neural network layer might skip the subsequent p neural network layers and connect directly to the (p+1)th neural network layer, where p is a positive integer. Due to these cross-layer connections, the neural network layers used for training also need to be used during application, increasing computational overhead. However, the encoder output and decoder input connections provided in the embodiments of this specification do not affect the encoder's feature extraction performance after the encoder has completed training. Therefore, the decoder can be used only during training and not in actual application, thereby reducing computational load and improving processing speed.

[0241] In practice, the decoder can use the same neural network structure as the encoder.

[0242] For example, such as Figure 17 The diagram shows a flowchart of an encoder training process. The training image is input into the encoder, which performs multi-stage second feature extraction through stages 21-23. The decoder then performs multi-stage feature reconstruction through stages 35-37, which have the same structure but opposite trends in feature dimension changes, to obtain a third feature map. In other words, if the feature dimension of stages 21-23 gradually increases, the feature dimension of stages 35-37 gradually decreases; conversely, if the feature dimension of stages 21-23 gradually decreases, the feature dimension of stages 35-37 gradually increases. Figure 17 The solid arrows in the diagram indicate the positive direction, which is the data flow direction during actual encoder application. Figure 17 The dashed arrows in the diagram indicate the reverse direction, which is the direction of data flow used by the backpropagation algorithm to adjust the encoder parameters.

[0243] In a specific implementation, if the encoder uses a convolutional neural network for the second feature extraction, the decoder can use a deconvolutional neural network for the third feature extraction. Specifically, the decoder upsamples the size of the encoder's second feature map using interpolation, then performs convolution, and then repeats the upsampling and convolution until the feature dimension and size of the intermediate result of the third feature map meet the requirements, thereby obtaining the third feature map.

[0244] In specific implementations, the second feature map obtained by extracting the second feature from the depth image can also be used for target detection results. Based on preset target categories, such as cars, trucks, buses, bicycles, pedestrians, and non-motorized vehicles, target classification results can be obtained through algorithm processing; and based on preset anchor points, target bounding box recognition results can be obtained through algorithm processing. However, the second feature map is an image without perspective transformation, and its viewpoint suffers from multi-scale object problems, such as near objects appearing larger than distant objects and object distortion, which interferes with target detection. Although classification information for each point can be obtained, the recognition results for target bounding boxes are poor.

[0245] To improve the accuracy of bounding box recognition results, such as Figure 18 As shown, another point cloud processing method is used. Steps S11 to S13 can be referred to Figure 1 The relevant content will not be repeated here. Figure 1 In comparison, it can also include:

[0246] S181, Obtain the second feature map corresponding to the depth image, wherein the second feature map is obtained by extracting the second feature from the depth image.

[0247] S182, Perform target classification and recognition on the second feature map to obtain the target classification result.

[0248] S183, Based on the target classification result and the target detection result, determine the classification information of each point in the point cloud.

[0249] There is no necessary order between steps S181 and steps S12 to S13. They can be executed in a preset order or simultaneously. Similarly, there is no necessary order between steps S182 and steps S12 to S13. They can be executed in a preset order or simultaneously. This specification does not impose any restrictions on this.

[0250] Therefore, by combining the target classification results and target detection results from the second feature map, the accuracy of the target bounding box recognition can be improved. In specific implementation, the classification information of each point in the point cloud is actually a probability vector of the category. The category corresponding to the highest probability value is the category that the point is most likely to correspond to in the environment. For example, for a point with a probability value of 0.9, corresponding to the category of vehicle, then this point is most likely to correspond to a vehicle in the actual environment; while for another point with a probability value of 0.67, corresponding to the category of pedestrian, then this point is most likely to correspond to a pedestrian in the actual environment.

[0251] In practice, categories can be displayed in different ways and shown at corresponding xyz coordinate points to form a classification infographic. For example, different categories can be assigned different grayscale values; see reference for details. Figure 20 The infographic shown depicts cars in white. Another example is assigning different numbers to different categories.

[0252] Furthermore, by setting display thresholds for different categories, some useless categories can be excluded, and only the required categories can be displayed. For example, if the display threshold for trees that are not desired to be displayed is set to 0.99, then tree points will not be displayed when displaying the point cloud. Thus, the point cloud can be selectively displayed according to needs, thereby highlighting the spatial information of the required categories.

[0253] In practical implementation, the target bounding box recognition results can be normalized. Different normalization methods can be used for the values ​​of each feature. For example, for 3D coordinates, logarithmic or proportional methods can be chosen to make the corresponding feature values ​​fall within the range [-1, 1]. For corners, sine or cosine functions can be chosen to make the corresponding feature values ​​fall within the range [-1, 1].

[0254] In practice, before performing perspective transformation, the image to be processed is upsampled to improve the resolution of the image, but this also adds useless information. In order to determine the importance of each perspective transformation point in the two-dimensional bird's-eye view, the two-dimensional bird's-eye view is calculated through a preset attention learning mechanism to obtain the attention distribution weight, which represents the importance of each perspective transformation point in the two-dimensional bird's-eye view.

[0255] Therefore, when performing object detection on the first feature map, the attention distribution weights corresponding to the first feature map can be obtained, where the weight values ​​can range from [0,1]. Then, based on the attention distribution weights corresponding to the first feature map, object detection is performed on the first feature map to obtain the object detection result, thereby improving the accuracy of the object detection result.

[0256] It is understood that the above description provides multiple embodiments, and the optional methods described in each embodiment can be combined and cross-referenced without conflict, thereby extending to a variety of possible embodiments. These can all be considered as embodiments disclosed in this specification.

[0257] This specification also provides a point cloud processing device corresponding to the above-described point cloud processing method. The following detailed description, with reference to the accompanying drawings, uses specific embodiments. It should be noted that the point cloud processing device described below can be considered as a functional module required to implement the point cloud processing method provided in this specification; the content of the point cloud processing device described below can be referred to in correspondence with the content of the point cloud processing method described above.

[0258] In specific implementation, such as Figure 20 The diagram shown is a structural block diagram of a point cloud processing device according to an embodiment of this specification. The point cloud processing device M0 may include:

[0259] Image acquisition unit M1 is suitable for acquiring a two-dimensional bird's-eye view corresponding to a point cloud;

[0260] The feature processing unit M2 is adapted to perform a first feature extraction based on the two-dimensional bird's-eye view to obtain a first feature map;

[0261] The target detection unit M3 is adapted to perform target detection on the first feature map to obtain the target detection result.

[0262] As can be seen from the above scheme, on the one hand, by performing feature extraction and target detection on the two-dimensional bird's-eye view, the accuracy of the target detection results is ensured, thus eliminating the need for rasterization of the point cloud, avoiding quantization errors, and improving information completeness and accuracy; on the other hand, converting the three-dimensional point cloud into a two-dimensional image can reduce the complexity of feature processing and the amount of computation, resulting in faster calculation speed, less time consumption, reduced computer memory usage, and thus improved computing performance; in addition, the data processing method provided in the embodiments of this specification can be adapted to more existing hardware platforms, saving hardware development costs and time, and has a wider range of application prospects.

[0263] like Figure 21 The diagram shown is a structural block diagram of an image acquisition unit according to an embodiment of this specification. The image acquisition unit M1 may include a feature processing subunit M11 and a perspective transformation subunit M12.

[0264] In an optional example, the feature processing subunit M11 is adapted to acquire the depth image corresponding to the point cloud and perform a second feature extraction on the depth image to obtain a second feature map; the perspective transformation subunit M12 is adapted to perform a perspective transformation on the second feature map to obtain the two-dimensional bird's-eye view.

[0265] In another optional example, such as Figure 21 As shown, the feature processing subunit M11 is adapted to acquire the depth image corresponding to the point cloud and perform second feature extraction on the depth image in multiple stages, wherein each stage contains different feature dimensions; the perspective transformation subunit M12 is adapted to acquire at least one intermediate result of the second feature in the multiple stages and perform perspective transformation to obtain the corresponding stage perspective view, so as to determine the two-dimensional bird's-eye view corresponding to the depth image.

[0266] In specific implementation, such as Figure 21 As shown, the perspective transformation subunit M12 is adapted to obtain the intermediate result of the second feature of a specified stage in the multiple stages, perform perspective transformation, and obtain a corresponding stage perspective view as the two-dimensional bird's-eye view.

[0267] In specific implementation, such as Figure 20 and 21 As shown, the perspective transformation subunit M12 is adapted to acquire intermediate results of second features for multiple specified stages in the multiple stages, and to perform perspective transformation on each intermediate result of the second features to obtain corresponding multiple stage perspective views; based on the multiple stage perspective views, multiple two-dimensional bird's-eye views are determined. The feature processing unit M2 is adapted to perform first feature extraction based on the multiple two-dimensional bird's-eye views to obtain the first feature map.

[0268] In one possible embodiment, the feature processing unit M2 is adapted to perform a first feature extraction on each of the two-dimensional bird's-eye views to obtain a corresponding first sub-feature map, and to determine the first feature map based on the plurality of first sub-feature maps.

[0269] In another feasible embodiment, such as Figure 20 As shown, the feature processing unit M2 is adapted to perform at least one stage of first feature extraction on each of the two-dimensional bird's-eye views; obtain corresponding first feature intermediate results according to a set selection stage; integrate the first feature intermediate results corresponding to the same selection stage to obtain stage sub-feature results; and determine the first feature map according to the stage sub-feature results. Wherein, the feature dimension of the selection stage is determined; and the feature dimensions of the first feature intermediate results corresponding to the same selection stage are the same.

[0270] In specific implementation, such as Figure 21 As shown, the perspective transformation subunit M12 may include:

[0271] Upsampling module M121 is adapted to upsample at least one specified stage of the intermediate result of the second feature in the plurality of stages, so that the size of the intermediate result of the second feature is consistent with the size of the depth image;

[0272] The perspective transformation module M122 is adapted to perform perspective transformation on the upsampled second feature intermediate result based on the correspondence between the upsampled second feature intermediate result and the point cloud to obtain the two-dimensional bird's-eye view.

[0273] In specific implementation, such as Figure 21 As shown, the feature processing subunit M11 may include:

[0274] Encoder M111 is adapted to perform multi-stage second feature extraction on the depth image, wherein encoder M111 is constructed based on encoder parameters;

[0275] The encoder parameter adjustment module M112 performs feature reconstruction and supervised learning on the output of the encoder M111 in order to adjust the encoder parameters.

[0276] In specific implementation, such as Figure 22 The diagram shown is a structural block diagram of another image acquisition unit in an embodiment of this specification. The image acquisition unit M1 may include:

[0277] The image acquisition subunit MA1 is adapted to determine the image to be processed based on the depth image;

[0278] The perspective transformation subunit MA2 is adapted to perform perspective transformation on the pixels in the image to be processed to obtain corresponding perspective transformation points, wherein the pixels include values ​​of multiple dimensions.

[0279] The characterization point acquisition subunit MA3 is adapted to acquire the perspective transformation points of the perspective transformation to the preset grid, and select the values ​​of the perspective transformation points in the grid that are in the same dimension to obtain characterization points, so as to characterize all perspective transformation points in the grid.

[0280] The two-dimensional bird's-eye view generation sub-unit MA4 is adapted to generate the two-dimensional bird's-eye view based on the representation points corresponding to the grid.

[0281] In specific implementation, such as Figure 20 As shown, the image acquisition unit M1 is also adapted to acquire a second feature map corresponding to the depth image, wherein the second feature map is obtained by extracting the second feature from the depth image;

[0282] The target detection unit M3 is also adapted to perform target classification and recognition on the second feature map, obtain target classification results, and determine the classification information of each point in the point cloud based on the target classification results and the target detection results.

[0283] In specific implementation, such as Figure 20 As shown, the target detection unit M3 is also adapted to obtain the attention distribution weights corresponding to the first feature map, wherein the attention distribution weights are calculated by the two-dimensional bird's-eye view through a preset attention learning mechanism to characterize the importance of each perspective transformation point in the two-dimensional bird's-eye view; based on the attention distribution weights corresponding to the first feature map, target detection is performed on the first feature map to obtain the target detection result.

[0284] This specification also provides an automatic identification system, including: a point cloud acquisition device and a data processing device, wherein the data processing device is connected to the point cloud acquisition device, wherein:

[0285] The point cloud acquisition device is suitable for acquiring point clouds;

[0286] The data processing device is adapted to perform the steps of any of the point cloud processing methods described above, to process the point cloud acquired by the point cloud acquisition device and obtain the target detection result. For details, please refer to the relevant descriptions, which will not be repeated here.

[0287] It is understood that in practical applications, the automatic identification system may also include other devices, such as image acquisition devices, positioning devices, and display devices. Furthermore, the data processing device's processing objects are not limited to point cloud acquisition devices. Therefore, based on actual data processing needs and communication configurations, the data processing device can establish communication connections with multiple devices in the automatic identification system to process the data transmitted from these devices. This specification does not impose specific limitations on the types of devices included in the automatic identification system, the processing objects of the data processing device, or the processing methods.

[0288] In a specific implementation, the data processing device may include a memory and a processor. The memory may store one or more computer-executable instructions, and the processor may invoke the one or more computer-executable instructions to execute the steps of the method provided in the embodiments of this specification.

[0289] In specific implementations, the point cloud acquisition device can be a lidar, millimeter-wave radar, or the like. The automatic identification system can be mounted on a device carrier to provide the data required for fully automated, semi-automated, or assisted control tasks, such as target detection results. Furthermore, the device carrier can be a vehicle (such as a motor vehicle or non-motor vehicle) or a portable platform (such as a handheld device), and this specification does not limit this.

[0290] This specification also provides an embodiment of a lidar, including multiple transmitters, multiple detectors, and a data processing device, wherein:

[0291] The transmitter and the detector are corresponding; the transmitter is suitable for transmitting detection signals from multiple angles; the detector is suitable for acquiring echo signals from multiple angles.

[0292] The data processing device is adapted to perform data processing based on the detection signal from the transmitter and the echo signal from the corresponding detector to obtain a point cloud, and to execute the steps of any of the point cloud processing methods described above. For details, please refer to the relevant descriptions; further elaboration is not provided here.

[0293] It is understandable that in practical applications, the data processing device's processing objects are not limited to signal processing. Therefore, depending on the actual data processing needs and communication configuration, the data processing device can also perform other processing on the transmitted data, as well as process other transmitted data. This specification does not impose specific restrictions on the data types of communication interactions in LiDAR or the processing methods of the data processing device.

[0294] This specification also provides a computer-readable storage medium storing computer instructions that, when executed, can perform the steps of the methods described in any of the above embodiments. The computer-readable storage medium can be any suitable readable storage medium, such as an optical disc, a hard disk drive, or a solid-state drive. The instructions stored on the computer-readable storage medium execute the methods described in any of the above embodiments; specific details can be found in the above embodiments and will not be repeated here.

[0295] The computer-readable storage medium may include, for example, any suitable type of memory cell, memory device, memory article, memory medium, storage device, storage article, storage medium and / or storage cell, such as memory, removable or non-removable medium, erasable or non-erasable medium, writable or rewritable medium, digital or analog medium, hard disk, floppy disk, optical disc read-only memory (CD-ROM), recordable optical disc (CD-R), rewritable optical disc (CD-RW), optical disc, magnetic medium, magneto-optical medium, removable memory card or disk, various types of digital universal optical disc (DVD), magnetic tape, cassette tape, etc.

[0296] Computer instructions may include any suitable type of code implemented using any appropriate high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, etc.

[0297] It should be noted that the term "an embodiment" or "embodiment" as used in this specification refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this specification.

[0298] While the embodiments disclosed in this specification are as described above, this specification is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this specification; therefore, the scope of protection of this specification should be determined by the scope defined in the claims.

Claims

1. A point cloud processing method, characterized in that, include: Obtaining a two-dimensional bird's-eye view corresponding to a point cloud includes: obtaining a depth image corresponding to the point cloud; performing second feature extraction on the depth image in multiple stages, wherein the feature dimensions of the intermediate results of the second features in each stage are different; in the multiple stages, obtaining the intermediate results of the second features in at least one specified stage and performing perspective transformation to obtain the corresponding stage perspective view, so as to determine the two-dimensional bird's-eye view corresponding to the depth image. Based on the two-dimensional bird's-eye view, the first feature is extracted to obtain the first feature map; Target detection is performed on the first feature map to obtain the target detection result.

2. The point cloud processing method according to claim 1, characterized in that, In the multiple stages, obtaining at least one intermediate result of the second feature of a specified stage and performing perspective transformation to obtain the corresponding stage perspective view, thereby determining the two-dimensional bird's-eye view corresponding to the depth image, includes: In the multiple stages, the intermediate result of the second feature of a specified stage is obtained and subjected to perspective transformation to obtain a corresponding stage perspective view, which serves as the two-dimensional bird's-eye view.

3. The point cloud processing method according to claim 1, characterized in that, In the multiple stages, obtaining at least one intermediate result of the second feature of a specified stage and performing perspective transformation to obtain the corresponding stage perspective view, thereby determining the two-dimensional bird's-eye view corresponding to the depth image, includes: In the multiple stages, intermediate results of the second feature of multiple specified stages are obtained, and perspective transformation is performed on each intermediate result of the second feature to obtain the corresponding multiple stage perspective views; Based on the multiple stage perspective views, multiple two-dimensional bird's-eye views are determined; The step of extracting the first feature based on the two-dimensional bird's-eye view to obtain the first feature map includes: Based on multiple two-dimensional bird's-eye view images, a first feature is extracted to obtain the first feature map.

4. The point cloud processing method according to claim 3, characterized in that, The first feature map is obtained by extracting first features based on multiple two-dimensional bird's-eye view images, including: Perform first feature extraction on each of the two-dimensional bird's-eye views to obtain the corresponding first sub-feature map; The first feature map is determined based on multiple first sub-feature maps.

5. The point cloud processing method according to claim 3, characterized in that, The first feature map is obtained by extracting first features based on multiple two-dimensional bird's-eye view images, including: At least one stage of first feature extraction is performed on each of the two-dimensional bird's-eye views; According to the set selection stage, the corresponding first feature intermediate result is obtained; wherein, the feature dimension of the first feature intermediate result of the selection stage is determined; and the feature dimension of the first feature intermediate result corresponding to the same selection stage is the same. The intermediate results of the first feature corresponding to the same selection stage are integrated to obtain the stage sub-feature results; The first feature map is determined based on the stage sub-feature results.

6. The point cloud processing method according to claim 1, characterized in that, In the plurality of stages, obtaining at least one intermediate result of the second feature of a specified stage and performing perspective transformation to obtain the corresponding stage perspective view includes: In the plurality of stages, at least one intermediate result of the second feature in a specified stage is obtained and upsampled so that the size of the intermediate result of the second feature is consistent with the size of the depth image; Based on the correspondence between the upsampled second feature intermediate result and the point cloud, a perspective transformation is performed on the upsampled second feature intermediate result to obtain the two-dimensional bird's-eye view.

7. The point cloud processing method according to claim 1, characterized in that, The two-dimensional bird's-eye view corresponding to the depth image is obtained through the following method: Based on the depth image, determine the image to be processed; The pixels in the image to be processed are subjected to perspective transformation to obtain the corresponding perspective transformation points. The pixels include values ​​of multiple feature dimensions. Obtain the perspective transformation points to the preset grid, and select the values ​​of the perspective transformation points in the grid that are in the same feature dimension to obtain the characterization points, so as to characterize all perspective transformation points in the grid. The two-dimensional bird's-eye view is generated based on the representation points corresponding to the grid.

8. The point cloud processing method according to claim 1, characterized in that, The step of obtaining the two-dimensional bird's-eye view corresponding to the depth image includes: The depth image is subjected to multi-stage second feature extraction using an encoder; The encoder is constructed based on encoder parameters, and the output of the encoder is subjected to feature reconstruction and supervised learning to adjust the encoder parameters.

9. The point cloud processing method according to claim 1, characterized in that, Also includes: Obtain a second feature map corresponding to the depth image, wherein the second feature map is obtained by extracting a second feature from the depth image; The second feature map is used for target classification and recognition to obtain the target classification result; Based on the target classification results and the target detection results, the classification information of each point in the point cloud is determined.

10. The point cloud processing method according to claim 1, characterized in that, The step of performing target detection on the first feature map to obtain target detection results includes: Obtain the attention distribution weights corresponding to the first feature map, wherein the attention distribution weights are calculated on the two-dimensional bird's-eye view through a preset attention learning mechanism to characterize the importance of each perspective transformation point in the two-dimensional bird's-eye view; Based on the attention distribution weights corresponding to the first feature map, target detection is performed on the first feature map to obtain the target detection result.

11. A point cloud processing device, characterized in that, include: An image acquisition unit is adapted to acquire a two-dimensional bird's-eye view corresponding to a point cloud, comprising: acquiring a depth image corresponding to the point cloud; performing second feature extraction on the depth image in multiple stages, wherein the feature dimensions of the intermediate results of the second features in each stage are different; and, in the multiple stages, acquiring the intermediate results of the second features in at least one specified stage and performing perspective transformation to obtain a corresponding stage perspective view, so as to determine the two-dimensional bird's-eye view corresponding to the depth image. The feature processing unit is adapted to perform a first feature extraction based on the two-dimensional bird's-eye view to obtain a first feature map; The target detection unit is adapted to perform target detection on the first feature map to obtain the target detection result.

12. An automatic identification system, characterized in that, include: A point cloud acquisition device and a data processing device, wherein the data processing device is connected to the point cloud acquisition device, wherein: The point cloud acquisition device is suitable for acquiring point clouds; The data processing device is adapted to perform the steps of the point cloud processing method according to any one of claims 1-10, so as to process the point cloud acquired by the point cloud acquisition device to obtain the target detection result.

13. A lidar, characterized in that, It includes multiple transmitters, multiple detectors, and data processing equipment, among which: The transmitter and the detector are corresponding; the transmitter is suitable for transmitting detection signals from multiple angles; the detector is suitable for acquiring echo signals from multiple angles. The data processing device is adapted to perform data processing based on the detection signal of the transmitter and the echo signal of the corresponding detector to obtain a point cloud, and to perform the steps of the point cloud processing method according to any one of claims 1-10.

Images