Target object prediction method and device and computer readable storage medium

Abstract

The present disclosure provides a target object prediction method, comprising: performing first voxelization processing on a plurality of first points included in a point cloud to obtain a plurality of first voxels; predicting a plurality of second points according to the plurality of first voxels, where each of the plurality of second points represents a prediction of a center of the target object by a first point; performing second voxelization processing on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels; and predicting a target object according to the one or more third voxels. In addition, the invention also provides a device capable of executing the method and a computer readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a method, device and computer-readable storage medium for predicting a target object. Background Art

[0002] At present, lidar is a scanning sensor that uses non-contact laser ranging technology. Its working principle is similar to that of a general radar system. It detects targets by emitting laser beams and forms point clouds (point cloud data) by collecting reflected beams. These point clouds can sense the spatial range of the target object after certain processing and have been widely used in the field of intelligent driving.

[0003] The existing technology mainly converts the point cloud into dense voxels (voxel data), then extracts the voxel features of the dense voxels, and performs calculations on the dense voxel feature map. This calculation method consumes a lot of computing cost for data in a large spatial range, which is not conducive to the rapid perception of distant target objects. At present, there is an existing technology that improves the above method, that is, only processes sparse point clouds, such as converting point clouds into fully sparse voxels to predict target objects. Among them, it is necessary to perform instance segmentation on the object, that is, cluster the point cloud and then extract the object features and predict its bounding box. However, the process of this calculation method is very complicated. At the same time, the accuracy of object instance segmentation will greatly affect the accuracy of the final prediction of the target object. Summary of the invention

[0004] In view of this, the present disclosure proposes a method, device and computer-readable storage medium for predicting a target object, which can predict the target object by performing a second voxelization process on multiple point data features to obtain real voxels and virtual voxels. This is because there will be many important features of the target object itself at the prediction center point to which the target object belongs and its surroundings, so the accuracy of the prediction can be improved. Therefore, the method proposed in the present disclosure can not only simplify the instance segmentation steps with complex processes, but also increase the accuracy of the target object prediction.

[0005] A first aspect of the present disclosure proposes a method for predicting a target object, comprising performing a first voxelization process on a plurality of first points included in a point cloud to obtain a plurality of first voxels; predicting a plurality of second points based on the plurality of first voxels, wherein each of the plurality of second points represents a prediction of a first point for the center of the target object; performing a second voxelization process on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels; and predicting the target object based on the one or more third voxels, wherein the one or more second voxels correspond to the plurality of first points, and the one or more third voxels correspond to the plurality of second points.

[0006] The second aspect of the present disclosure proposes a prediction system for a target object, comprising a memory storing a computer program; a processor coupled to the memory, wherein when the processor executes the computer program, the prediction system implements a method including performing a first voxelization process on a plurality of first points included in a point cloud to obtain a plurality of first voxels; predicting a plurality of second points based on the plurality of first voxels, wherein each of the plurality of second points represents a prediction of a first point for the center of the target object; performing a second voxelization process on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels; and predicting the target object based on the one or more third voxels, wherein the one or more second voxels correspond to the plurality of first points, and the one or more third voxels correspond to the plurality of second points.

[0007] The third aspect of the present disclosure proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method comprising performing a first voxelization process on a plurality of first points included in a point cloud to obtain a plurality of first voxels; predicting a plurality of second points based on the plurality of first voxels, wherein each of the plurality of second points represents a prediction of a first point for the center of the target object; performing a second voxelization process on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels; and predicting the target object based on the one or more third voxels, wherein the one or more second voxels correspond to the plurality of first points, and the one or more third voxels correspond to the plurality of second points. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In all drawings, the same reference numerals refer to similar but not necessarily identical elements. Figure 1 It is a schematic diagram showing the process of a method for predicting a target object according to an embodiment of the present disclosure. Figure 2 FIG. 4 is a flowchart illustrating a target object prediction system according to an embodiment of the present disclosure. Figure 3 It is a schematic diagram showing the process of a method for predicting a target object according to an embodiment of the present disclosure. Figure 4A is a schematic diagram showing a virtual feature mixing module according to an embodiment of the present disclosure. Figure 4B is a schematic diagram showing a virtual feature mixing module according to an embodiment of the present disclosure. Figure 5 is a schematic diagram illustrating a method for training a target object according to an embodiment of the present disclosure. Figure 6 is a block diagram showing a target object prediction device according to an embodiment of the present disclosure. Figure 7 is a schematic diagram showing a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0009] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] In this disclosure, the term "plurality" means two or more, unless otherwise specified. In this disclosure, the term "and / or" describes the association relationship of associated objects, covering any one of the listed objects and all possible combinations. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0011] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects without intending to limit their positional relationship, temporal relationship, or importance relationship. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in ways other than those illustrated or described herein.

[0012] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, system, product, or apparatus.

[0013] Figure 1 It is a schematic diagram showing the process of a method for predicting a target object according to an embodiment of the present disclosure.

[0014] The present disclosure provides an example of an application scenario of a target object prediction system. During the driving of a vehicle, a vehicle-mounted laser radar installed on the vehicle can obtain a point cloud (point cloud data) of the vehicle's surrounding environment at a certain time interval. Of course, it is also possible to obtain a point cloud of the vehicle's surrounding environment at a certain distance during the driving of the vehicle. After the vehicle-mounted laser radar obtains the point cloud, the vehicle-mounted laser radar can send the point cloud to a server via the network. After obtaining the point cloud, the server can divide the point cloud into l x × y × z Multiple squares. Each square can be a voxel after the first voxelization process. Then you can refer to Figure 1 In the step from inputting point cloud to extracting sparse voxel features, for one or more voxels, a deep learning model can be used based on the point cloud, such as a PointNet point cloud feature extractor, to extract point cloud features, thereby obtaining multiple first voxels with features.

[0015] Please refer to Figure 1 After obtaining a plurality of first voxels, the first voxels may be input into a sparse voxel encoder, such as a sparse convolutional network or a sparse attention network, to obtain a first voxel feature. After obtaining the first voxel feature, the first voxel feature may be mapped to one or more first points A1 included in the first voxel corresponding to the first voxel feature to obtain one or more initial point data features, and one or more point data features may be formed by combining one or more geometric features from the first point A1 to the center of the first voxel to which the first point A1 belongs.

[0016] See also Figure 1 From the step of sparse voxel feature extraction to the step of point cloud classification and the step of center prediction. After obtaining one or more point data features, the server can input the one or more point data features into a multilayer perceptron (MLP) to predict multiple offsets between multiple first points A1 and the center of the target object to obtain multiple second points A2.

[0017] Please continue to refer to Figure 1, a second voxelization process is performed on the multiple first points A1 and the multiple second points A2, and one or more second voxels B1 and one or more third voxels B2 can be obtained. In some embodiments, the second voxel corresponds to the real first point A1 in the point cloud, so it can be called a real voxel, and the third voxel corresponds to the second point A2 generated by prediction, so it can be called a virtual voxel. Among them, one or more second voxels correspond to multiple first points, and one or more third voxels correspond to multiple second points. Then, after obtaining the multiple third voxels, the multiple third voxels can be input into a sparse voxel encoder, such as a sparse convolutional network or a sparse attention network, to obtain multiple third voxel features. Next, the multiple third voxel features can be mixed by a first mixing module to obtain multiple first mixed voxel features. Specifically, each of the multiple third voxels corresponds to one of the multiple first mixed voxel features.

[0018] In some embodiments, the plurality of second voxels and the plurality of third voxels may be input into a sparse voxel encoder, such as a sparse convolutional network or a sparse attention network, to obtain a plurality of second voxel features and a plurality of third voxel features. Then, the plurality of second voxel features and the plurality of third voxel features are mixed to obtain a plurality of second mixed voxel features. Specifically, each of the plurality of second voxels corresponds to one of the plurality of second mixed voxel features, and each of the plurality of third voxels corresponds to one of the plurality of second mixed voxel features.

[0019] Next, a prediction module of a neural network is provided (not shown in the figure), and the prediction module of the neural network generates a target prediction label corresponding to the third voxel according to the second mixed voxel feature corresponding to each of the plurality of third voxels. In some embodiments, a prediction module of a neural network can also generate a target prediction label corresponding to the third voxel according to the first mixed voxel feature corresponding to each of the plurality of third voxels. The target prediction label can be a prediction box used to predict the target object. Therefore, the target prediction label output by the prediction model of the neural network can be used to predict the target object.

[0020] Among them, since the target prediction labels will overlap to a considerable extent, non-maximum suppression (NMS) can be used to select the best target prediction label from the overlapping target prediction boxes to correct the result of the target object prediction in the previous step.

[0021] It should be noted that the mixing module used to mix multiple second voxel features and multiple third voxel features can use the above-mentioned first mixing module to mix, or provide a second mixing module to mix. In addition, the mixing module used to mix multiple second voxel features and multiple third voxel features can be mixed by a mixing module including a convolutional network model. The first mixed voxel features mixed by the first mixing module, and the second mixed voxel features mixed by the first mixing module or the second mixing module can both more effectively extract the features of the predicted target, and obtain a target prediction mark for predicting the target object, that is, a prediction box for the target object, thereby improving the accuracy of the prediction of the target object. Finally, the server can send the prediction box used to predict the target object to the control system of the vehicle in the intelligent driving process, so that the vehicle can promptly determine whether it is necessary to take braking measures, honk the horn, etc.

[0022] Please refer to Figure 2 , the embodiment of this specification provides a target object prediction system. The target object prediction system may include a data acquisition device 110 and a server 120. The data acquisition device 110 may be a laser radar with network access capability and installed on a vehicle that can perform intelligent driving. Specifically, for example, the data acquisition device 110 may be a scanning device such as a laser radar, which can obtain a point cloud by scanning. Among them, the laser radar may be a vehicle-mounted laser radar, an airborne laser radar, a satellite-mounted laser radar, a station-type laser radar, a mobile laser radar, etc. The server 120 may be an electronic device with a certain computing and processing capability. It may have a network communication module, a processor, and a memory, etc. Of course, the server may also refer to software running in the electronic device. The server 120 may also be a distributed server, which may be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server 120 may also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server 120 may also be a new technical means that can realize the corresponding functions of the embodiment of the specification. For example, it may be a new form of "server" based on quantum computing.

[0023] Please refer to Figure 3 The present disclosure provides a method for predicting a target object. The method for predicting a target object can be applied to a server. The method for predicting a target object can include the following steps.

[0024] Step S210: performing first voxelization processing on the plurality of first points included in the point cloud to obtain a plurality of first voxels.

[0025] In some embodiments, when using point clouds to identify target objects, the point clouds need to be converted into grid data of a certain size. Therefore, the point clouds may first be converted into volume data that can represent the range of the object.

[0026] The point cloud may be a set of vectors in a three-dimensional coordinate system. The point cloud may be obtained by laser radar scanning, where the scanned data is recorded in the form of points, each of which contains three-dimensional coordinates. Of course, the point cloud may also include information such as color, reflectivity, intensity, etc., which are used to represent the properties of the scanned object. Of course, the point cloud may also be a set of vectors in a two-dimensional coordinate system.

[0027] In some implementations, the step of performing a first voxelization process on the point cloud to obtain a plurality of first voxels may include: dividing the point cloud into the plurality of first voxels at a specified resolution according to a spatial coordinate range of the point cloud.

[0028] The first voxelization method can divide the point cloud into grids according to the spatial range, so that the point cloud can be stored in memory in an orderly manner, which is conducive to reducing random memory access and increasing data computing efficiency. After the multiple first points included in the point cloud are subjected to the first voxelization process to obtain multiple first voxels, the first voxels can be efficiently subjected to spatial convolution operations, which is conducive to extracting multi-scale and multi-level local features. On the other hand, after the first voxelization process, the point cloud can be effectively downsampled, which is suitable for processing point clouds of a larger order of magnitude.

[0029] The first voxel may be a pixel in a 3D space. Specifically, the size of the voxel may be set in advance, such as a 5 cm*5 cm*5 cm cube, or a 30 cm*30 cm*30 cm cube, or any cube with a side length between 5 cm and 30 cm, or a cube of other sizes. That is, each of the first voxels may include a plurality of first points in the point cloud.

[0030] In some embodiments, the categories of multiple first points can also be predicted based on a classifier (such as a multi-layer perceptron, MLP) and multiple point data features. That is, the category of the target object to which the first point belongs, or the background category. In some cases, the first point is point cloud data representing the target object. Specifically, for example, the point cloud obtained by scanning the on-board laser radar includes buildings, trees, ground, pedestrians, obstacles, etc., because the vehicle will be affected by obstacles, pedestrians, etc. during driving, and it is necessary to honk or avoid them. Of course, in some cases, the first point can also be point cloud data representing ground data, so as to identify the lane line, thereby reminding the vehicle to change lanes immediately during driving.

[0031] The method for performing first voxelization processing on a point cloud may include: determining a minimum cuboid containing all points according to the minimum and maximum values ​​of all points in the point cloud in the three coordinate directions of X, Y, and Z; setting the size of the first voxel according to the size of the minimum cuboid and the resolution requirement, and dividing the minimum cuboid into first voxels according to the size of the first voxel; for each point in the point cloud, determining the first voxel in which it is located; and after traversing all points in the point cloud, displaying only the first voxel containing the point.

[0032] Step S220: predicting a plurality of second points based on the plurality of first voxels, wherein each of the plurality of second points represents a prediction of a first point for the center of the target object.

[0033] In some embodiments, step S220 includes extracting a plurality of first voxel features of the plurality of first voxels; obtaining a plurality of point data features of the plurality of first points based on the plurality of first voxel features; and predicting a plurality of second points based on the plurality of point data features.

[0034] In some embodiments, the first voxel features of the first voxels can be extracted by a sparse voxel encoder (such as a sparse convolution-based network, a sparse attention network, etc.). In some cases, if the features of the points in the point cloud are directly extracted, the features of the extracted point data may be relatively poor, which directly affects the classification of the point cloud or the result of the point cloud segmentation. Since the features of the point data included in the first voxel are similar to the first voxel features, for each of the first voxels: after each of the first voxels in the first voxels is obtained after the first voxelization process, the first voxel features of the first voxel can be mapped to one or more first points contained in the first voxel to obtain one or more initial point data features. Then, one or more geometric features between the one or more first points and the corresponding first voxels are calculated. Then, based on the one or more initial point data features and one or more geometric features (for example, the offset of each first point to the center of the first voxel to which it belongs), one or more point data features of one or more first points of the first voxel are generated.

[0035] The first voxel feature can be extracted using a point cloud feature extractor. Specifically, for example, PointNet can be used to extract the initial point cloud feature. PointNet is a neural network architecture that can process point clouds end-to-end. PointNet has two alignment networks, namely T-Net for aligning input data (input transform) and features (feature transform), which can be recorded as T-Net-1 and T-Net-2. T-Net-1 outputs a 3×3 transformation matrix for transforming the input (xyz). T-Net-2 outputs a 64×64 matrix for aligning the feature space so that the transformation matrix A of the feature space is as close to an orthogonal matrix as possible. Of course, the method for extracting the first voxel feature in the disclosed embodiment is not limited, and the method for extracting the initial voxel feature can also be PointNet++, KPConv, PointConv, Point Transformer, etc.

[0036] In some cases, in order to make the extracted features richer, a sparse voxel encoder may be used to extract the first voxel feature.

[0037] The method of inputting the first voxel into a preset sparse voxel encoder to extract the first voxel feature can be implemented by using a commonly used sparse voxel encoder. Specifically, the first voxel can be represented in the form of a sparse matrix, and then the sparse matrix can be input into a 3D convolutional network, such as a network based on sparse convolution, to extract the feature.

[0038] In some cases, since each of the multiple first voxels may include multiple first points, if the first voxel feature is directly mapped to the first point included in the first voxel, the point data feature of the obtained first point may be repeated. Therefore, in order to distinguish the points included in each first voxel, the geometric dimension or geometric feature of the point (point data) can be increased so that the acquired point data features can be distinguished, thereby obtaining the point data feature of the first point. The geometric feature can be a relationship feature between the first point and the first voxel to which the first point belongs. Specifically, for example, the distance relationship between the first point and the first voxel can be obtained based on the difference between the coordinate vector of the first point and the coordinate vector of the center of the first voxel to which the first point belongs. Of course, the geometric feature can also include the angle between the straight line connecting the first point and the center of the first voxel and the horizontal direction or the numerical direction or a specified direction.

[0039] In some cases, the point data feature of the first point may be obtained by directly mapping the first voxel feature to the first point.

[0040] The first point may be point data in the initial point cloud. The first point may also be a smaller voxel further divided on the basis of the first voxel currently divided. Since the smaller voxel represents a smaller spatial range, the smaller voxel can be regarded as a point, and the difference between the center vector (center of mass) of the smaller voxel and the vector of the point data included in the voxel is very small.

[0041] The initial point data feature is the initial point data feature obtained by mapping the first voxel feature to the first point included in the first voxel corresponding to the first voxel feature. Specifically, for example, if a first voxel includes four first points, then these four first points have the same initial point data feature as the first voxel feature. Of course, in order to distinguish these four first points, the relative coordinates of the four first points relative to the center of the first voxel of the first voxel to which the four first points belong can also be added to the initial point data feature, thereby generating the four point data features of the four first points.

[0042] Of course, one or more point data features of one or more first points can also be obtained by a certain interpolation algorithm, such as an inverse distance weighted interpolation algorithm, a linear interpolation method, a bilinear interpolation method, a Kriging interpolation method, etc. Specifically, for example, the steps of obtaining point data features using the inverse distance weighted (IDW) interpolation method can be: determining the reference pixel (i.e., the first voxel) to be used for the point, calculating the distance between the first point vector and the center vector of the first voxel, and then using the formula:

[0043] in, is the feature of the data of the point to be interpolated, Z(S i ) is the attribute value of the reference pixel, N is the number of selected reference pixels, λ i is the weight coefficient calculated according to the inverse of the distance.

[0044] In some embodiments, multiple offsets between multiple first points and the center of the target object are predicted based on a classifier (such as a multi-layer perceptron, MLP) and multiple point data features, and multiple second points are obtained based on the multiple first points and the multiple offsets.

[0045] The so-called offset refers to the position difference of the first point relative to the center of the target object. Through multiple first points and multiple offsets, multiple center positions corresponding to the target object to be predicted can be predicted, that is, multiple second points can be obtained.

[0046] Step S230: performing a second voxelization process on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels.

[0047] In this embodiment, the first voxel feature can be mapped to the first point to obtain the initial point data feature, and then the multiple offsets between the multiple first points and the center of the target object are calculated, and then the multiple second points are obtained based on the multiple first points and the multiple offsets. Then, the multiple first points are subjected to the second voxelization process to obtain one or more second voxels; at the same time, the multiple second points are subjected to the second voxelization process to obtain one or more third voxels. Since the second voxel is obtained by voxelizing the first point (the point in the original point cloud), it can also be called a real voxel. Since the third voxel is obtained by voxelizing the second point (the point generated by prediction), it can also be called a virtual voxel. In some embodiments, the size of the second voxel or the third voxel obtained by the second voxelization process is larger than the size of the first voxel obtained by the first voxelization process. For example, the size of the second voxel or the third voxel can be a cube of 0.5 meters * 0.5 meters * 0.5 meters, or a cube of other sizes. Most of the third voxels are located near the center of the target object, so the problem of missing central features in the sparse point cloud can be solved by obtaining the third voxels. In addition, since the third voxels are close to the center of the target object, the difficulty of the network in predicting the position will be reduced. In other words, compared with the method of segmenting objects in traditional methods, such as the method of clustering point clouds, the target object prediction method in this embodiment helps to improve the overall computing performance.

[0048] Specifically, a first voxel can be defined as a first point contained in each first voxel having a number greater than or equal to a first predetermined value, wherein the first predetermined value is, for example, 1. That is, when the number of first points in a certain voxel is greater than or equal to 1, this voxel is defined as a first voxel. A third voxel is defined as a second point contained in each third voxel having a number greater than or equal to a second predetermined value, wherein the second predetermined value is, for example, 1. That is, when the number of second points in a certain voxel is greater than or equal to 1, this voxel is defined as a third voxel. However, the first predetermined value and the second predetermined value can also be set to other values ​​according to actual needs.

[0049] Step S240: predicting a target object according to one or more third voxels.

[0050] In some embodiments, one or more third voxel features of the one or more third voxels may be extracted to predict the target object. Alternatively, the target object may be predicted based on the one or more second voxels and the one or more third voxels. Specifically, the multiple third voxel features of the multiple third voxels may be extracted by a sparse voxel encoder (such as a sparse convolution-based network, a sparse attention network, etc.), and the multiple second voxel features of the multiple second voxels may be extracted.

[0051] In some embodiments, after extracting the plurality of third voxel features of the plurality of third voxels, the plurality of third voxel features may not be mixed by a mixing module, but the plurality of third voxel features may be directly used to predict the target object.

[0052] In some cases, each of the multiple third voxel features cannot completely cover the entire target object. For example, one of the multiple third voxel features may only contain the hood of the vehicle, not the entire vehicle. This is because the center prediction of the target object in reality is not completely accurate, that is, the center prediction of the target object may not be located at the true center position of the target object. Therefore, a mixing module is needed to mix one or more nearby third voxel features. This mixing module can be a sparse feature extraction encoder (such as a sparse convolution-based network, a sparse attention network, etc.) to mix multiple voxels.

[0053] Please refer to Figure 4A and Figure 4B , a schematic diagram of a hybrid module in an embodiment of the present disclosure.

[0054] Figure 4A and Figure 4B They all predict the same target object. Figure 4A The center prediction of the target object (the point in the box) is relatively accurate, and the features of the entire target object are aggregated to voxel C. T In which voxel C T is a virtual voxel. Please continue to refer to Figure 4B In one embodiment, it is assumed that the center prediction of the target object will be divided into three center predictions (voxel C1, voxel C2 and voxel C3 respectively), that is, the features of the target object are split into features provided by voxel C1, voxel C2 and voxel C3. For example: when the feature provided by voxel C1 is only the roof of the vehicle, the target object will be predicted directly by the voxel feature represented by voxel C1. At this time, this is equivalent to predicting a car only by the features of the roof. Among them, voxel C1, voxel C2 and voxel C3 are virtual voxels. Therefore, in some embodiments, voxel C1, voxel C2 and voxel C3 can be feature extracted by a sparse feature extraction encoder to mix multiple virtual voxel features, and the target object is predicted by mixing multiple virtual voxel features. This can make better use of the features of the entire target object as a prediction, and can accurately predict the target object without using instance segmentation (equivalent to segmenting the object, that is, clustering the point cloud). The above method is simpler than the instance segmentation method and helps to improve the overall computing performance. By blending multiple virtual voxel features, the target object can be predicted more accurately.

[0055] In some embodiments, a plurality of third voxel features of the plurality of third voxels may be extracted by a sparse voxel encoder (such as a sparse convolution-based network, a sparse attention network, etc.). Then, the plurality of third voxel features are mixed by one or more mixing modules to obtain a plurality of first mixed voxel features. Each of the plurality of third voxels corresponds to one of the plurality of first mixed voxel features.

[0056] In some embodiments, a plurality of second voxel features of the plurality of second voxels may be extracted by a sparse voxel encoder (such as a sparse convolution-based network, a sparse attention network, etc.). The plurality of second voxel features are then mixed by one or more mixing modules to finally obtain a plurality of first mixed voxel features. Each of the plurality of second voxels corresponds to one of the plurality of first mixed voxel features.

[0057] In some embodiments, a plurality of second voxel features of the plurality of second voxels and a plurality of third voxel features of the plurality of third voxels can be extracted by a sparse voxel encoder (such as a sparse convolution-based network, a sparse attention network, etc.). Then, the plurality of second voxels and the plurality of third voxel features are simultaneously mixed by one or more mixing modules to obtain a plurality of second mixed voxel features. Each of the one or more second voxels corresponds to one of the plurality of second mixed voxel features, and each of the one or more third voxels corresponds to one of the plurality of second mixed voxel features. In other words, each second voxel and each third voxel obtains a mixed second voxel feature.

[0058] The modules of mixed voxel features in the above embodiments can be implemented by mixing modules of one or more convolutional network models, or by sharing the same or more mixing modules. The input of the mixing module can be the second voxel feature, the third voxel feature, or a mixture of the two voxel features.

[0059] In addition to obtaining the mixed voxel features, the prediction of the target object also requires passing the mixed voxel features corresponding to each voxel through a prediction module to generate a target prediction label corresponding to the voxel, wherein one voxel can correspond to one target prediction label.

[0060] In some embodiments, the target object can be predicted based on the multiple first mixed voxel features. Alternatively, the target object can be predicted based on the multiple second mixed voxel features. Specifically, the target object can be predicted based on the second mixed voxel features corresponding to each of the one or more third voxels to generate a target prediction label corresponding to the third voxel. In more detail, a target prediction label corresponding to the third voxel can be generated through a prediction module based on the second mixed voxel features corresponding to each third voxel. Among them, the target prediction label is a prediction of the target object. Further, the target prediction label is a prediction box (bounding box) for the target object.

[0061] In some embodiments, one or more third voxels may be input to the neural network to output a target prediction label through the prediction model of the neural network. Then, the target object is predicted using the target prediction label output by the prediction model of the neural network.

[0062] In some embodiments, please refer to Figure 5 For each of the one or more third voxels, the centroid position of the one or more second points corresponding to the third voxel can be calculated. The one or more second points corresponding to the third voxel include one or more foreground points D2 and one or more non-foreground points D1. The foreground points D2 are, for example, points belonging to the target object in the point cloud, and the non-foreground points D1 are, for example, points not belonging to the target object in the point cloud.

[0063] Please refer to Figure 5 In order to save computational effort, the size of the third voxel can be designed to be larger than some smaller objects. For example, the luminous area of ​​a traffic sign is not large when viewed from a distance, so the size of the third voxel may be larger than the luminous area of ​​the traffic sign. However, this makes it difficult to determine whether a voxel contains a smaller object frame, and thus it is difficult to determine whether to assign the smaller object frame to the points in the voxel for training when training the neural network. Therefore, it is necessary to provide a method for training a prediction model for a neural network to avoid the above problems as much as possible.

[0064] In some embodiments, in order to train the prediction model of the neural network, the prediction method of the target object further includes obtaining a sample prediction label, which represents the true prediction of the target object, that is, the ground truth (GT), wherein the sample prediction label is also represented as a sample prediction box. Specifically, the neural network can be trained with the sample prediction label according to the relationship between the centroid position (the centroid position of one or more second points corresponding to the third voxel (virtual voxel)) and the sample prediction label.

[0065] Please refer to Figure 5 , Figure 5 When assigning sample prediction labels to virtual voxels V x When the geometric center G1 of the virtual voxel size is used as the standard, it may be that the center G1 does not fall into the sample prediction mark G d , which results in the sample prediction label G d is ignored. That is, the sample prediction label G is not d As a positive sample, assigned to the virtual voxel V x In the multiple second points. Even, the sample prediction label G d As a representation that it does not belong to the virtual voxel V x Negative samples of multiple second points in; when assigning sample prediction labels to virtual voxels V x When the center G2 of multiple points in the virtual voxel is used as the standard, it may also be because the center G2 does not fall into the sample prediction mark G d , which results in the sample prediction label G d is ignored. That is, the sample prediction label G is not d As a positive sample, assigned to the virtual voxel V x In the multiple second points. Even, the sample prediction label G d As a representation that it does not belong to the virtual voxel V x Negative samples of multiple second points in; when assigning sample prediction labels to virtual voxels V x When the weighted centroid G3 of one or more points corresponding to the virtual voxel is used as the standard, the weighted centroid G3 has a greater probability of falling on the sample prediction mark G d Within, and avoid the sample prediction label G that should have been used as a positive sample d is ignored, and the sample prediction label G is avoided d are treated as negative samples.

[0066] Therefore, when training the prediction model of the neural network, it is necessary to provide a method to assign labels to virtual voxels so that the weighted centroid of one or more points corresponding to the virtual voxels is more likely to fall on the sample prediction label G d The method proposed in the present disclosure includes using the weighted centroid of one or more points corresponding to the third voxel (virtual voxel) as a reference standard point for assigning training samples to the virtual voxel. Specifically, the foreground point of the target object can better reflect the real position of the target object. Therefore, when the weighted centroid of the virtual voxel is closer to the foreground, it has a greater probability of falling within the sample prediction label, so that the sample prediction label can be smoothly assigned to the virtual voxel.

[0067] In some embodiments, calculating the centroid position of the one or more second points corresponding to the third voxel further includes: weighted averaging the positions of the one or more foreground points having a first weight and the positions of the one or more non-foreground points having a second weight to obtain the centroid position of the one or more second points corresponding to the third voxel, wherein the first weight is greater than the second weight. In some embodiments, the first weight can be set to 1, and the second weight can be set to less than 1.

[0068] In some embodiments, for each of the one or more third voxels, the centroid position of the one or more second points corresponding to the third voxel can also be calculated by the following formula (2) and formula (3). Specifically, the step of obtaining the centroid position of the one or more second points corresponding to the third voxel can be: determining the third voxel to be used, wherein the one or more second points corresponding to the third voxel include one or more foreground points and one or more non-foreground points. in, is the foreground point, x is the coordinate of one or more second points corresponding to the third voxel, and I(x) is the weight, where 0<α<1.

[0069] Next, the weighted average coordinates of the third voxel can be calculated, that is, the positions of one or more foreground points with the first weight and the positions of one or more non-foreground points with the second weight are weighted averaged to obtain the centroid position of the one or more second points corresponding to the third voxel. Use formula (3): in, is the weighted average coordinate, x i is the coordinate of the second point.

[0070] Specifically, in response to the centroid position falling within the sample prediction frame, the neural network is trained with the sample prediction mark. When the centroid position (referring to the weighted centroid point of one or more second points corresponding to the virtual voxel) falls within the sample prediction frame (referring to the real prediction frame of the target object), it is identified as a positive sample, and the sample prediction frame is assigned to the virtual voxel; if the weighted centroid falls outside the sample prediction frame, it is identified as a negative sample. Then, the information of the positive and negative samples obtained above can be used to train and update the neural network.

[0071] Furthermore, the updating of the neural network further includes: comparing the target prediction label and the sample target prediction label to generate a comparison result. According to the comparison result, the neural network will update its own prediction model to output a new target prediction label. The updated prediction model can be used to make the prediction of the target object represented by the new target prediction label closer to the true prediction of the target object. That is to say, if the centroid position falls within the sample prediction box, then the sample prediction label can be assigned to the virtual voxel, the virtual voxel is marked as the sample prediction label, and the sample prediction label is a positive sample of the virtual voxel; if the centroid position falls outside the sample prediction box, then the sample prediction label is a negative sample of the virtual voxel. When the sample prediction label is determined to be a positive sample or a negative sample of the virtual voxel, the comparison result generated according to the sample prediction label can be used to update the neural network.

[0072] Specifically, the updating process is to adjust the neural network so that it can output a true prediction that is closer to the target object. In other words, the neural network will improve its prediction model based on the comparison results to improve the prediction accuracy of the target object. The updated prediction model will be used to generate a new target prediction tag. This new target prediction tag will be closer to the true prediction of the target object. By continuously performing this updating process, the neural network will gradually improve its prediction ability for the target object. In short, during the updating process, the neural network will compare the actual target with its own target prediction, and then improve the prediction model based on the comparison results to output a prediction that is closer to the actual target. This process will be repeated to improve the prediction accuracy of the neural network.

[0073] See also Figure 6 One embodiment of the present specification further provides a target object prediction device 600. The target object prediction device 600 may include a point cloud voxelization processing module 610, a voxel feature extraction module 620, a voxel feature mapping module 630, a voxel feature mixing module 640 and a target object prediction module 650.

[0074] The point cloud voxelization processing module 610 is used to voxelize the point cloud to obtain a plurality of voxels; wherein the plurality of voxels correspond to a plurality of points in the point cloud.

[0075] The voxel feature extraction module 620 is used to extract voxel features of multiple voxels to obtain multiple voxel features.

[0076] The voxel feature mapping module 630 is used to map the multiple voxel features to the multiple points included in the multiple voxels respectively, so as to obtain the multiple point data features of the multiple points.

[0077] The voxel feature mixing module 640 is used to mix multiple voxel features to obtain multiple mixed voxel features.

[0078] The target object prediction module 650 is used to predict the target object according to a plurality of mixed voxel features.

[0079] The specific definition of the prediction device for the target object can be found in the definition of the prediction method for the target object above, which will not be repeated here. Each module in the above-mentioned prediction device for the target object can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] See also Figure 7 The embodiment of this specification also provides a computer device 700, including a memory 710 and a processor 720, wherein the memory 710 stores a computer program 712, and is characterized in that when the processor 720 executes the computer program 712, the target object prediction method in the above embodiment is implemented.

[0081] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuit systems, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as an electrical signal, an optical signal, or an electromagnetic signal generated by a machine, which is generated to encode information to be transmitted to a suitable receiver device.

[0082] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and the computer program can be deployed in any form, including being deployed as an independent program or as a module, component, subroutine, or another unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program under discussion, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or partial codes). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed in multiple sites and interconnected by a communication network.

[0083] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)).

[0084] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The necessary elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer will also be operatively connected to receive data from the one or more mass storage devices or to transfer data to the one or more mass storage devices or to perform both. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by a dedicated logic circuit system or may be incorporated into the dedicated logic circuit system.

[0085] Some embodiments described herein are described in the general context of a method or process, which in one embodiment can be implemented by a computer program product included in a computer-readable medium, which may include computer executable instructions (such as program code), which can be executed by a computer in a networked environment, for example. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. Therefore, computer-readable media may include non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer or processor executable instructions, associated data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. Such specific sequences of executable instructions or associated data structures represent examples of corresponding actions for implementing the functions described in these steps or processes.

[0086] Some embodiments of the disclosed embodiments can be implemented as devices or modules using hardware circuits, software or a combination thereof. For example, a hardware circuit implementation may include discrete analog and / or digital components, which may be integrated as parts of a printed circuit board, for example. Alternatively or additionally, the disclosed components or modules may be implemented as application specific integrated circuits (ASICs) and / or field programmable gate arrays (FPGAs) devices. Additionally or alternatively, some implementations may include a digital signal processor (DSP), which is a dedicated microprocessor with an architecture optimized for the operational needs of digital signal processing associated with the disclosed functionality of the present application. Similarly, various components or subassemblies within each module may be implemented with software, hardware or firmware. Any connection method and medium known in the art may be used to provide a connection between a module and / or a component within a module, including but not limited to communication via the Internet, a wired network or a wireless network using an appropriate protocol.

[0087] Although this paper includes many details, these details should not be interpreted as limitations to the scope of the claimed invention, but are interpreted as descriptions of features peculiar to specific embodiments. Some features described in the context of different embodiments herein may also be combined in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be separated or implemented in multiple embodiments with any suitable sub-combination. In addition, although features may be described as acting with some combinations and even initially claimed above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a variation of a sub-combination or a sub-combination. Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring to perform such operations in the particular order shown or in a sequential order, or to perform all described operations to achieve desired results.

[0088] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

Claims

1. A method for predicting a target object, characterized in that: The prediction method comprises: Performing a first voxelization process on a plurality of first points included in the point cloud to obtain a plurality of first voxels; Predicting a plurality of second points based on the plurality of first voxels, wherein each of the plurality of second points represents a prediction of a first point for a center of the target object; Performing a second voxelization process on the plurality of first points and the plurality of second points to obtain one or more second voxels and one or more third voxels; and predicting the target object according to the one or more third voxels, The one or more second voxels correspond to the plurality of first points, and the one or more third voxels correspond to the plurality of second points.

2. The prediction method according to claim 1, characterized in that: Predicting the plurality of second points according to the plurality of first voxels comprises: extracting a plurality of first voxel features of the plurality of first voxels; Based on the plurality of first voxel features, obtaining a plurality of point data features of the plurality of first points; and Based on the plurality of point data features, the plurality of second points are predicted.

3. The prediction method according to claim 2, characterized in that: Extracting the plurality of first voxel features of the plurality of first voxels comprises: Based on a sparse voxel encoder, the plurality of first voxel features of the plurality of first voxels are extracted.

4. The prediction method according to claim 2, characterized in that: Based on the plurality of first voxel features, obtaining the plurality of point data features of the plurality of first points comprises: For each of the plurality of first voxels: Mapping the first voxel feature of the first voxel to one or more first points included in the first voxel to obtain one or more initial point data features; calculating one or more geometric features between the one or more first points and the corresponding first voxels; and The one or more point data features of the one or more first points of the first voxel are generated based on the one or more initial point data features and the one or more geometric features.

5. The prediction method according to claim 2, characterized in that: Predicting the plurality of second points based on the plurality of point data features comprises: Based on the classifier and the plurality of point data features, predicting a plurality of offsets between the plurality of first points and the center of the target object; and The plurality of second points are obtained based on the plurality of first points and the plurality of offsets.

6. The prediction method according to claim 1, characterized in that: Predicting the target object according to the one or more third voxels comprises: One or more third voxel features of the one or more third voxels are extracted to predict the target object.

7. The prediction method according to claim 1, characterized in that: The one or more third voxels are a plurality of third voxels, wherein predicting the target object according to the one or more third voxels comprises: extracting a plurality of third voxel features of the plurality of third voxels; blending the plurality of third voxel features to obtain a plurality of first blended voxel features, wherein each of the plurality of third voxels corresponds to one of the plurality of first blended voxel features; and The target object is predicted according to the plurality of first mixed voxel features.

8. The prediction method according to claim 7, characterized in that: Extracting the plurality of third voxel features of the plurality of third voxels comprises: Based on a sparse voxel encoder, the plurality of third voxel features of the plurality of third voxels are extracted.

9. The prediction method according to claim 1, characterized in that: Predicting the target object according to the one or more third voxels comprises: The target object is predicted based on the one or more second voxels and the one or more third voxels.

10. The prediction method according to claim 9, characterized in that: Predicting the target object according to the one or more second voxels and the one or more third voxels comprises: extracting one or more second voxel features of the one or more second voxels; extracting one or more third voxel features of the one or more third voxels; Mixing the one or more second voxel features and the one or more third voxel features to obtain a plurality of second mixed voxel features, wherein each of the one or more second voxels corresponds to one of the plurality of second mixed voxel features, and each of the one or more third voxels corresponds to one of the plurality of second mixed voxel features; and The target object is predicted according to the plurality of second mixed voxel features.

11. The prediction method according to claim 10, characterized in that: The one or more second voxel features and the one or more third voxel features are mixed by a mixing module including a convolutional network model.

12. The prediction method according to claim 10, characterized in that: Predicting the target object according to the plurality of second mixed voxel features comprises: The target object is predicted by the prediction module according to the second mixed voxel feature corresponding to each of the one or more third voxels to generate a target prediction label corresponding to the third voxel.

13. The prediction method according to claim 1, characterized in that: The size of the voxel obtained by the second voxelization process is greater than the size of the voxel obtained by the first voxelization process.

14. The prediction method according to claim 1, characterized in that: The number of first points included in each of the plurality of first voxels is greater than or equal to a first predetermined value, and the number of second points included in each of the one or more third voxels is greater than or equal to a second predetermined value.

15. The prediction method according to claim 1, characterized in that: Predicting the target object according to the one or more third voxels comprises: Inputting the one or more third voxels into a neural network to output a target prediction label through a prediction model of the neural network, wherein the target prediction label represents a prediction of the target object, wherein the target object prediction method further includes: A sample target prediction label is obtained, wherein the sample target prediction label represents a true prediction of the target object; wherein, for each of the one or more third voxels: calculating a centroid position of one or more second points corresponding to the third voxel; and According to the relationship between the centroid position and the sample target prediction label, the neural network is trained with the sample target prediction label.

16. The prediction method according to claim 15, characterized in that: The one or more second points corresponding to the third voxel include one or more foreground points and one or more non-foreground points, wherein calculating the centroid position representing the one or more second points corresponding to the third voxel comprises: The positions of the one or more foreground points having a first weight and the positions of the one or more non-foreground points having a second weight are weighted averaged to obtain the centroid position of one or more second points corresponding to the third voxel, wherein the first weight is greater than the second weight.

17. The prediction method according to claim 15, characterized in that: The sample target prediction mark is a sample prediction box, wherein, according to the relationship between the centroid position and the sample target prediction mark, the neural network is trained with the sample target prediction mark, including: In response to the centroid position falling within the sample prediction frame, the neural network is trained with the sample target prediction label.

18. The prediction method according to claim 17, characterized in that: In response to the centroid position falling within the sample prediction frame, training the neural network with the sample target prediction label includes: comparing the target prediction marker and the sample target prediction marker to generate a comparison result; and The neural network updates the prediction model according to the comparison result to output a new target prediction label, and the updated prediction model is used to make the prediction of the target object represented by the new target prediction label closer to the actual prediction of the target object.

19. A prediction device for a target object, characterized in that: include: a memory storing a computer program; A processor is coupled to the memory, and when the processor executes the computer program, the prediction system implements the prediction method according to any one of claims 1 to 18.

20. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the prediction method according to any one of claims 1 to 18.

Images