Methods for semantic segmentation of point clouds

Abstract

This paper presents a novel method for semantic segmentation. [Solution] A method for semantic segmentation of a point cloud using a neural network in a driver assistance system, wherein a neighborhood is defined for each point in the point cloud, and this neighborhood is a set of other points in the point cloud placed near that point, and the features of a single point are convolved together with the features of points in the neighborhood of this single point according to a weight matrix, the points of the point cloud are organized into a sequence, an ordinal number indicating the position of the point in the sequence is assigned, an algorithm is used to create this sequence, and this algorithm ensures that the difference between the ordinal numbers of two points correlates with the spatial distance between these points in the point cloud, and that the neighborhood of a single point is defined as a set of points, and the ordinal numbers of these points are a sequence of numbers that contain the ordinal number of the single point.

Description

[Technical Field]

[0001] The present invention relates to a method for semantic segmentation of a point cloud by a neural network in a driver assistance system for an automobile, wherein a neighborhood is defined for each individual point in the point cloud, and this neighborhood is a set of other points in the point cloud placed near that point, and the features of a single point are convolved together with the features of points in the neighborhood of this single point according to a learned weight matrix. [Background technology]

[0002] In driver assistance systems for automobiles, a model of the vehicle's surroundings is generated based on on-board sensors, usually including radar and / or lidar sensors, and this model then serves as the basis for decisions regarding the driver assistance system's actions. Positioning data from sensors, such as radar sensors, can be displayed as a two- or three-dimensional point cloud, where each received radar reflection is represented as a point, and the coordinates of this point in Cartesian coordinates (or polar coordinates) indicate the position of the radar target in space. If the radar sensor has angular resolution in both azimuth and elevation, a three-dimensional point cloud with distance, azimuth, and elevation as coordinates is obtained, which can be converted to Cartesian coordinates x, y, z. Furthermore, each point is assigned one or more features that characterize the point in more detail. In the case of a radar sensor, these features could be, for example, the radial velocity of the radar target (with adjustments for the vehicle's own motion) or the radar's scattering cross-section. In addition to these local features, points can also be assigned non-local features that characterize specific relationships between that point and other points in the point cloud.

[0003] In a generalized sense, the coordinates of a point can also be interpreted as "features." For example, in a three-dimensional point cloud, each point has a set of features, the first three features of this set being three spatial coordinates, followed by further features such as radial velocity, scattering cross-section, and so on. In claim 1, the term "features" should be understood in this generalized sense.

[0004] In this context, "semantic segmentation" broadly refers to the process of generating a model of a vehicle's surroundings from a point cloud. In its simplest form, semantic segmentation may involve positioning and classifying a single object (e.g., a vehicle or traffic sign). For objects consisting of multiple parts, semantic segmentation may also involve identifying the individual parts and their relationships to each other. At its most complex stage, semantic segmentation provides a more detailed description, to varying degrees, of the entire traffic environment, including the objects present.

[0005] To enable the use of neural networks in semantic segmentation, a point cloud representation is often chosen that allows for some type of convolution operation. Such convolution operations can recognize specific structures in the locations and features of a limited number of adjacent points. Since these structures do not depend on the location of this group of adjacent points in space, a degree of translational invariance is achieved. That is, if a network learns to recognize and correctly classify an object at the left edge of the field of view, for example, it requires little additional learning cost to recognize the same object at the right edge of the field of view. In "PointPillars: Fast Encoders for Object Detection from Point Clouds" by Lang et al. (https: / / openaccess.thecvf.com / content_CVPR_2019 / papers / Lang_PointPillars_Fast_Encoders_for_Object_Detection_From_Point_Clouds_CVPR_2019_paper.pdf), a grid-based method is described that enables convolution operations on two-dimensional point clouds. In this case, a regular two-dimensional grid is placed across the entire field of view, and a convolutional neural network (CNN) is used to recognize the structure of the point cloud in a manner similar to that used in digital image recognition.

[0006] Swanningson et al., "Radar Point GNN: Graph Based Object Recognition for Unstructured Radar Point-cloud Data" (https: / / ieeexplore.ieee.org / document / 9455172), describes a neural network that performs convolution operations on a graph that displays the neighbor relationships between points in a point cloud.

[0007] These known convolutional networks differ from one another in their computational cost and memory requirements. In the case of grid-based approaches, the computational cost and memory requirements increase or decrease by the square (for 2D point clouds) or cube (for 3D point clouds) of the number of grid cells, and do not depend on the number of points in the point cloud. Therefore, this method is inefficient, especially for low-density point clouds equivalent to those obtained by radar sensors.

[0008] In graph-based approaches, computational and memory costs increase or decrease only with the number of points in the point cloud, thus achieving relatively high efficiency in many applications. However, generating a graph that meaningfully represents the point cloud requires a relatively complex algorithm. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Lang et al.: "PointPillars: Fast Encoders for Object Detection from Point Clouds" (https: / / openaccess.thecvf.com / content_CVPR_2019 / papers / Lang_PointPillars_Fast_Encoders_for_Object_Detection_From_Point_Clouds_CVPR_2019_paper.pdf) [Non-Patent Document 2] Swanningson et al.: "Radar Point GNN: Graph Based Object Recognition for Unstructured Radar Point-cloud Data" (https: / / ieeexplore.ieee.org / document / 9455172) [Non-Patent Document 3] Qi et al.: "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" - https: / / openaccess.thecvf.com / content_cvpr_2017 / papers / Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf [Non-Patent Document 4] Cuthill-McKee algorithm (https: / / en.wikipedia.org / wiki / Cuthill%E2%80%93McKee_algorithm) [Non-Patent Document 5] Chollet: "Xception: Deep Learning with Depth Side Separable Convolutions" - https: / / arxiv.org / pdf / 1610.02357.pdf [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The object of the present invention is to present a method for semantic segmentation that features low memory and computation costs and can be efficiently executed with limited hardware resources. [Means for solving the problem]

[0011] This problem is solved by the present invention by organizing the points of a point cloud into a sequence, for which each point is assigned an ordinal number indicating its position within the sequence, and an algorithm is used to create this sequence, which ensures that the difference between the ordinal numbers of any two points is positively correlated with the spatial distance between those points within the point cloud, and that the neighborhood of a single point is defined as a set of points, and the ordinal numbers of these points are a sequence of successive numbers that contain the ordinal number of the single point.

[0012] In this method, the point cloud is reorganized into a one-dimensional sequence, so the subsequent convolution operation is reduced to a one-dimensional convolution, that is, an operation similar to point-by-point convolution on a discrete one-dimensional grid. Therefore, the memory and computational costs for the convolution operation, both during network training and during the actual semantic segmentation, increase or decrease only by the relatively limited number of points in the (one-dimensional) neighborhood. This facilitates the execution of this operation by hardware incorporated into automotive driver assistance systems. Moreover, the abundant literature on one-dimensional convolution and the experience gained therein can be utilized.

[0013] The most advantageous form is presented in the dependent claims. Unlike other known point-based architectures (e.g., Qi et al.: "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" - https: / / openacess.thecvf.com / content_cvpr_2017 / papers / Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf), the method proposed here does not have invariance under permutations of the sequence of points. Instead, the order in which the points are arranged in the sequence plays a central role in approximately representing the spatial relationships of the points within a 2D or 3D point cloud. The essence of this lies in the fact that the distance between two points within the sequence is positively correlated with the spatial separation between these points within the point cloud. In other words, if two points are close to each other within the sequence, these points are usually also relatively close to each other within the point cloud. This property of the sequence can be achieved by a number of known algorithms. In the case of a two-dimensional point cloud, for example, it can be achieved by arranging the points in ascending order based on the x coordinate and, in the case of the same x coordinate, in ascending order based on the y coordinate, or vice versa. Another possibility is to fill the sensor's field of view with a space-filling path, i.e., a continuous path that fills each point of the plane or space, excluding intervals smaller than a given limit, almost equally. In this case, each point of the point cloud is assigned to the position on the path closest to this point, and the order of these positions on the path determines the order of the points within the sequence. Another conceivable algorithm is the Cuthill-McKee algorithm (https: / / en.wikipedia.org / wiki / Cuthill%E2%80%93McKee_algorithm). All of these known algorithms can be generalized in three dimensions in an easily conceivable way.

[0014] Unlike the grid-based convolution algorithm, the method proposed here is not inherently translation invariant. However, translation invariance can be achieved approximately at least by appropriately restricting the space of admissible weight matrices. Furthermore, the weight matrix can be masked so that the weights are forced to be set to 0 for point pairs that are too far apart from each other.

[0015] In the literature, many variants of the method for the original discrete convolution operation are known (see, for example, Chollet: "Xception: Deep Learning with Depth Side Separable Convolutions" - https: / / arxiv.org / pdf / 1610.02357.pdf -). These variants are also possible with the method proposed here.

[0016] Exemplary embodiments will be explained in detail below based on the drawings.

Brief Explanation of Drawings

[0017] [Figure 1] It is a block diagram of an essential part of a driver assistance system adapted to the method according to the present invention. [Figure 2] It is a schematic diagram for explaining some basic principles of the method for discrete convolution. [Figure 3] It is a diagram showing a simplified example of a two-dimensional point group in which the points are arranged in a sequence. [Figure 4] It is a diagram showing the true distances between consecutive points in a sequence based on FIG. 3. [Figure 5] It is a diagram showing the same as FIG. 3, but a point group sequenced using a plane-filling type path. [Figure 6] It is a diagram showing the true distances between consecutive points in a sequence based on FIG. 5. [Figure 7]This diagram shows the true distance between each of the first points in the sequence and other points in the sequence in Figure 5. [Figure 8] This figure shows a more realistic example of a point cloud, such as one that can be produced by an automotive radar system. [Figure 9] This figure shows the point cloud based on Figure 8, along with a space-filling path for sequencing this point cloud. [Figure 10] This figure shows the sequencing results of the point cloud in Figure 9. [Figure 11] This graph shows three examples of extended features of points in a point cloud, as functions of the point's position within the sequence. [Figure 12] This figure shows a neighborhood graph of the point cloud sequenced based on Figure 10. [Figure 13] This figure shows a comparison of neighboring property graphs created based on different criteria. [Figure 14] This is a flowchart illustrating one of the various variations of the method according to the present invention. [Figure 15] This is a flowchart illustrating one of the various variations of the method according to the present invention. [Figure 16] This is a flowchart illustrating one of the various variations of the method according to the present invention. [Figure 17] This is a flowchart illustrating one of the various variations of the method according to the present invention. [Modes for carrying out the invention]

[0018] Figure 1 shows a simplified block diagram of the driver assistance system, specifically the part that works to create a surrounding environment model. The radar sensor 10 measures the distance, azimuth, and relative velocity of objects within a specific field of view of the radar sensor. The processor 12 converts the distance and azimuth into Cartesian coordinates (x,y) to create a two-dimensional point cloud 14, where each point represents the location of the positioned radar target. The sequencer 16 organizes the points of the point cloud into a sequence 18, within the sequencer 16 assigning an ordinal number i (i=1, ..., N) to each of the N points in the point cloud. To do this, the sequencer 16 uses an algorithm, which will be described in more detail later, that ensures that the true distance between any two points in the point cloud 14 is positively correlated with the difference in the ordinal numbers of those points. In other words, the sequence 18 is formed such that two points that are close together in the point cloud 14 are generally also close together in the sequence 18.

[0019] Subsequently, within sequence 18, one neighborhood 20 is defined for each point with ordinal number i, and neighborhood 20 consists of 2j+1 points with ordinal numbers ij, ..., i-1, i, i+1, ..., i+j (in the example shown, j=3).

[0020] In the (highly simplified) example shown here, each point in sequence 18 has three generalized features, specifically the point's coordinates x,y and its radial velocity calculated based on its Doppler shift (as a "real" feature). These generalized features of the points in sequence 18 are the input data for the first layer 22 of a convolutional neural network 24 (CNN). The first layer 22 has several units 26 (also called nodes or neurons), and there is a one-to-one relationship between the points in sequence 18 and the units 26. This assumes that the number of points N in the point cloud is the same as the number of units in layer 22. This assumption does not always have to be met in practice. Generally, the network 24 is designed so that the number of units in the first layer is greater than the maximum expected number of points in the point cloud. If the number of points in the point cloud is smaller, the point cloud is filled with virtual points, and some appropriate values ​​are assumed for the features of these virtual points (a procedure called "padding"). To prevent this padding from causing significant defects during subsequent data processing, each point is further stored with information indicating whether it is a virtual or real point.

[0021] The second layer 28 of the neural network 24 has the same number of units as the first layer 22, and the units of the second layer are also assigned in a 1:1 ratio to the ordinal number i of the points in sequence 18. Each unit in the second layer 28 is associated with a unit 26 in its neighborhood 20 in the first layer by a weight (symbolized by a line) that is not zero, meaning that each unit in the first layer is associated with a unit whose ordinal number differs by up to 3 from the ordinal number of the unit in layer 28.

[0022] To simplify the diagram, let's assume for now that each point in sequence 18 has only one feature. In this case, the value of this feature is input to the assigned unit 26 of the first layer 22. Thus, the convolution operation is to form a weighted sum within each unit of the second layer 28 from the values ​​of the units in the first layer 22 that are within its surroundings 20, and the weights indicating the strength of the connection between the units of the first and second layers are determined by the weight matrix. That is, within the second layer 28, for each position in sequence 18, a value is calculated that depends on the feature values ​​of the neighboring points in the neighborhood 20, in the manner given by the weight matrix. Usually, a so-called bias is added to these calculated values, and this bias does not depend on the values ​​of the units in the first layer 22.

[0023] However, since each point in sequence 18 is assigned four features (coordinates x, y, radial velocity, and a feature characterizing the virtual point), each unit 26 in the first layer shown in Figure 1 must have four data channels. In other words, instead of just one unit 26 (with only one channel), there are four such units, all of which are assigned to the same position in sequence 18. Figuratively speaking, in this case, the units 26 of the first layer form four layers stacked on top of each other in a direction perpendicular to the diagram plane in Figure 1.

[0024] The second layer 28 may also have multiple layers, each representing a new feature. Generally speaking, the number and meaning of features in the second layer may differ from the number and meaning of features in the first layer, and one unit for one feature in the second layer 28 may be linked by a weight matrix to units in multiple different layers in the first layer 22. In other words, the original features of a point in the second layer can be consolidated into fewer features, or conversely, more "artificial" features can be generated from the original features.

[0025] The weighted sum calculated within the units of the second layer 28 is then further subjected to what is usually a nonlinear activation function, and the result is then passed to a further layer 30. In the example shown, this further layer 30 is a fully networked layer, meaning that no further convolution operations are performed, and each unit in layer 30 is linked to each unit in layer 28. The number of units in layer 30 may be different from the number of units in the first layer 22, and the units in layer 30 do not need to be assigned to specific positions in sequence 18.

[0026] Within the third layer 30, a weighted sum is calculated using a weight matrix from the activation function values ​​calculated in layer 28. This weighted sum calculated in layer 30 is then passed to the next layer 32 (usually without the application of a nonlinear activation function). In the example shown, this layer 32 is also a fully networked layer and is the output layer of the neural network 24. The values ​​calculated here together form a description of the characteristics of the point cloud 14, and based on this description, for example, it may be possible to derive which objects are located at which points within the field of view of the radar sensor 10.

[0027] In reality, the number of layers in a neural network and the number of units in each layer are significantly greater than in the example shown here. In particular, multiple stacked convolutional layers can be provided, where the units of these layers are assigned to the positions of points in sequence 18, such as the units of layer 28, and the convolution operation is performed on the solution of the previous convolutional layer.

[0028] To illustrate the general functionalities of convolutional neural networks (CNNs), Figure 2 shows a simple example of a conventional grid-based CNN. The input layer records the luminance values ​​of “pixels” or square grid cells 34 arranged in a rectangular grid with 11 rows and 15 columns. For the cell in the fourth row of the fourth column, a neighborhood 36 is shown, consisting of the cell itself and its eight direct neighboring cells.

[0029] Within a neighborhood, a so-called kernel is defined, assigning a weight to each point in that neighborhood. In the example shown, all points in the central column of the kernel have a weight of 2, and all remaining kernel cells have a weight of -1. For a convolution operation on the grid cell at column and row index (4,4), a weighted sum is formed from the luminance values ​​of the cells in neighborhood 36 and the weights presented by the kernel. The same convolution operation is performed for the neighborhoods of all other grid cells. Figuratively speaking, this is equivalent to stepping the kernel horizontally and vertically along the grid until the kernel has traversed the entire grid. The step size can be the width of a single grid cell or, for example, twice this width.

[0030] The luminance pattern of the grid cells, in the example shown, consists of 38 vertical line elements and 40 horizontal line elements. When the kernel is shifted to the right by a step size of 1 from the position shown in Figure 2, after 5 steps, three dark grid cells of line element 38 reach a position with a weight of 2. The weighted sum with respect to the middle cell of these cells has a maximum value of 6 in this case. For the topmost and bottommost cells of line element 38, the weighted sum is 4, respectively.

[0031] Shifting the kernel further down will eventually reach positions where the cells of horizontal line element 40 have weights of -1, 2, and -1. In this case, the weighted sums with respect to grid cells (9,7), (9,8), and (9,9) are each 0, meaning the whole is the same as being in the white neighborhood. Therefore, the kernel shown here is well-suited to recognizing vertical line segments like segment 38, but is blind to horizontal line segments.

[0032] The weight matrix that connects the CNN's input layer to the first convolutional layer displays all kernel positions on the grid, i.e., causes a convolution operation for all neighborhoods of all grid cells (at least for a step size of 1). However, the weight matrix can be decimated by selecting a step size of 2 in the vertical direction. In this case, vertical line segments 38 can still be reliably found.

[0033] Grid cells at the edges of the grid present a slight challenge because part of the kernel lies outside the grid. However, this can be resolved by "padding," which adds a virtual grid cell with a brightness value of 0.

[0034] The weighted sum obtained within the second layer of the network here has high values ​​for units that precisely correspond to the positions of vertical line segments. Therefore, the values ​​passed from the second layer to the next higher layer are, so to speak, a map that presents the positions of all vertical line segments. Correspondingly, the second (90° rotated) kernel can selectively search for horizontal line segments such as segment 40.

[0035] Once a CNN learns the correct kernel weights, it can find the structure being searched for, regardless of its location on the grid. In this sense, grid-based CNNs are translationally invariant, meaning they are unaffected by the spatial displacement of line segments 38 and 40.

[0036] The grid cells colored darkly in Figure 2 can also represent points in a point cloud. Accordingly, the CNN shown in Figure 2 can be similarly used for point cloud segmentation. However, in this case, the number of arithmetic operations required and the memory space required are determined only by the grid resolution and do not depend on the number of points in the point cloud, so it will be less efficient for sparse point clouds.

[0037] To reduce computational cost, a method is proposed here in which the convolution operation is point-based rather than grid-based, and the kernel has only one dimension instead of two or three. For this purpose, the points of the point cloud are organized into a one-dimensional sequence, and then one neighborhood is defined for each point within the sequence.

[0038] One possible method for forming a sequence is illustrated in Figure 3, which shows a two-dimensional point cloud consisting of only eight points. The sequence is defined by assigning an ordinal number between 1 and 8 to each point. These points are first sorted in ascending order based on their x-coordinate values. Only if two points have the same x-coordinate are they sorted in ascending order based on their y-coordinate values. It can be seen that two points that are close together in the point cloud generally have a small distance from each other in the sequence as well, meaning their ordinal numbers are only slightly different. For example, the distance between point 1 and point 8 is clearly greater than the distance between point 1 and point 2.

[0039] In Figure 4, points 1 through 8 are arranged on a straight line such that the distance between any two consecutive points in a sequence on the line corresponds to the actual distance between those points in the x,y plane. In the ideal case where the distance in the sequence matches the spatial distance within the point cloud, the points in Figure 4 must be arranged on a regular raster. In reality, the raster of the points in Figure 4 is not regular, but it is already relatively close to a regular raster.

[0040] Of course, as an alternative, these points could be sorted in ascending order based on their y-coordinate, and then in ascending order based on their x-coordinate if they match. Finally, it is conceivable to combine both methods, for which a sequence could be generated first based on each of the methods, then for each point, the arithmetic mean of its ordinal numbers in both sequences could be calculated, and then the sequence could be sorted in ascending order based on the resulting mean value.

[0041] Following this, in the case of a three-dimensional point cloud, a sequence can be formed by organizing points in some order based on three coordinates. Here again, different orders can be combined with each other. Furthermore, it is conceivable to project the points of the point cloud onto multiple lines pointing in different directions in space. In this method, a different sequence of the point cloud is obtained for each line, and the final sequence can be generated by averaging these again.

[0042] Another method for forming a sequence is shown in Figure 5. It shows the same point cloud as in Figure 3, but with a space-filling path 42 in the xy-plane. This path 42 is defined as a continuous mapping of real numbers to the xy-plane, and this continuous mapping has the property that, within the portion of the plane under consideration, no point is further than a certain positive (as small as possible) limit ε from the nearest point on path 42. In the example shown, path 42 consists of vertical and horizontal path segments that form a kind of fractal, where two adjacent parallel path segments are always the same distance 2ε from each other. Consequently, it is guaranteed that no point in the plane is further than ε from the nearest point on path 42. In this case, a sequence is formed by moving along path 42 from a starting point A and an ending point E, "collecting" points from the point cloud along the way, and arranging these points in the order they were collected. Here again, we can see that two points that are close together in the point cloud are also close together in the sequence, at least approximately.

[0043] In Figure 6, similar to Figure 4, points 1 through 8 are arranged on a straight line such that the distance between consecutive points on the line corresponds to the actual distance in the plane. Here again, we can see that a nearly regular raster is generated.

[0044] In Figure 7, points 1 through 8 are arranged on a straight line such that the distances between point 1 and points 2 through 8 correspond to their true distances within the point cloud. Here again, some correlation can be seen between the distances within the sequence and the true distances in the plane, although some outliers exist. For example, the true distance between point 1 and point 4 is clearly smaller than the distance between point 1 and point 2, and the distance between point 1 and point 7 is clearly larger than the distance between point 1 and point 8. Nevertheless, this sequencing method also reproduces the neighborhood relationships between points almost correctly.

[0045] Figure 8 shows a somewhat more factual point cloud 44, which may be recorded, for example, by a radar sensor 10 in an automobile. Point cloud 44 contains points 46, 48, 50, and 52 with different plane filling densities. Points with the same plane filling density correspond to radar targets with the same radial velocity. Thus, the plane filling density represents the characteristics of the points and makes it possible to identify objects within the point cloud consisting of radar targets that are spatially close together and have (almost) the same radial velocity.

[0046] Figure 9 shows the same point cloud as Figure 8, except that a tiling path 42 is added to the xy-plane, which allows the points to be organized into a sequence. Figure 10 shows a neighborhood graph for the sequence formed based on Figure 9. Points 46 to 52 are shown in their true positions within the point cloud 44, except that adjacent points in the sequence are connected by edges 54. This method is similar to teaching children to draw, where characteristic points of the drawing are pre-printed, and the child is instructed to connect these points with straight lines in a predetermined order ("Connect the Dots" or "Connect the Dots"). Thus, this method will be abbreviated as the CTD method below. Here again, we can see that the neighborhood relationships within the sequence correlate well with the true neighborhood relationships. For example, three points 48 that are close together in the point cloud are also connected by edges in the graph. However, in the case of point 46, this correlation is not perfect. Here, only four points are connected to each other by edges, while the fifth point is not connected to the other four. Therefore, the object represented by these 46 points is, so to speak, "torn apart." This error can, and must, be corrected in a later processing step.

[0047] Figure 11 shows three functions f that represent the x-coordinate, y-coordinate, and radial velocity values ​​as features of the sequence shown by the neighboring graph in Figure 10, as functions of the ordinal number i (i.e., the position of the point within the sequence). mThe graph of (i) is shown. The curves labeled x represent the x-coordinate, the curves labeled y represent the y-coordinate, and the thickly drawn curve vr represents the radial velocity. The situation in which neighborhood relationships within the point cloud correlate well with neighborhood relationships within the sequence is expressed here by the fact that the curves for x and y are relatively "smooth" and do not irregularly jump back and forth between very large and very small values. The curve vr for radial velocity has a clearly negative value only in the ordinal numbers near i=20, which represents point 48 in the neighborhood graph (Figure 10). Thus these points 48 represent objects approaching the vehicle. Furthermore, the radial velocity curve has three positive swings. One of them is at i=33, which represents an object moving away, corresponding to point 50 in the point cloud. Two other positive swings at i<10 and i=42 belong to the "torn" object represented by point 46.

[0048] Figure 12 shows a neighborhood graph of the point cloud 44, where each point (except the first and last points in the sequence) is connected to four immediate neighbors by four edges 54. This corresponds to neighborhood 20 in Figure 1 at j=2. For comparison, Figure 13 shows a neighborhood graph of the point cloud 44, formed based on real-world neighborhood relationships, independent of the sequence. Here, each point is connected to neighbors within a certain threshold distance. It can be seen that the neighborhood graph in Figure 12, formed using the sequence, provides a better approximation to the "real-world" neighborhood graph in Figure 13.

[0049] Padding at the beginning and end of the sequence allows the first and last points of the sequence to also have all (partially virtual) neighbors, so that the convolution operation has the same mathematical form for all points.

[0050] When the point cloud is arranged into a sequence based on one of the methods described above, a convolutional operation can be performed by a neural network, in which each unit of the input layer is assigned to exactly one point in the sequence, and each unit of the next (convolutional) layer is also assigned to exactly one point in the sequence.

[0051] If the point cloud has dimension D (typically 2 or 3) and each point of the point cloud has C features (in a relatively narrow sense), a single point x of the point cloud i can be fully described by a vector f with D + C components. i The convolutional operation (hereinafter referred to as the CTD operation) exists in the following operation in this case.

[0052] CTD w (x i ) = Σ k=-j k=j w k f (i+k) + b where CTD w (x i ) is the result of the convolutional operation for a single output channel of the unit belonging to point x i , w k is the row vector constituting the k-th row of the weight matrix W, f (i+k) is the vector indicating the extended feature of the point with ordinal number i + k, and b is the bias already mentioned. The index k continues from -j to +j. That is, the feature vectors of 2j + 1 points are summed, and these points constitute the neighborhood of point x i , and are weighted by the learned weight coefficients respectively, and these weight coefficients are the components W k of the row vector w kl . These components W kl define the kernel of the convolutional operation (for one output channel).

[0053] This CTD operation is invariant under positional displacement within the point sequence. That is, if a weight matrix is ​​trained to recognize a specific structure within the point cloud, the weight matrix will recognize this structure regardless of its location within the sequence.

[0054] However, CTD operations do not easily have translational invariance with respect to the spatial position of points in a point cloud. However, this problem is solved by the weight matrix W kl This can be mitigated by imposing certain restrictions. In this regard, if d is an index that presents the spatial coordinates of the components of vector f, then the components W that make up column d of the weight matrix kd However, it is useful to require that it be added to 0. If the vector f has components (x, y, vr), Σ k W k1 =0 and Σ k W k2 =0 This is required.

[0055] In this case, if we assume that spatial translation in the x-direction is performed by replacing the x-coordinate of all points with x+Δx, then in the CTD operation, the terms that depend on Δx are added to zero, and therefore the solution is not changed.

[0056] Below, as a simple example, we consider semantic segmentation, which involves classifying each point in a point cloud into one of the following four classes. • Vehicle • Vulnerable objects (pedestrians, cyclists, motorcyclists) • Static objects • Padding Furthermore, we assume that the point cloud contains N=256 points, and each point has two relatively narrow-sense features (vr and radar cross-section), including both point coordinates, meaning it has four extended features.

[0057] Figure 14 shows one possible network architecture. The input data 56 consists of 256 × 4 variables. The encoder 58 sorts the points represented by the input data, first in ascending order based on the x-coordinate, then in ascending order based on the y-coordinate, performs padding at the end of the sequence, and combines the coordinates and features into an extended feature vector (including information about virtual points). The CTD layer 60 performs a convolution operation on a neighborhood of 25 points (j=12) using a weight matrix restricted to achieve translational invariance. Each unit of this CTD layer has 16 output channels, and therefore the result consists of 256 × 16 variables. A nonlinear activation function 62 (ReLu; Rectified Linear Unit) is applied to this result. In the second CTD layer 64, the CTD operation is again applied to the solution of this activation function with the same neighborhood and the same number of output channels as the first layer 60. Finally, a known softmax activation function (a smoothed MAX function applied to each of the 16 output channels of layer 64) yields four output variables, which, for each of the four classes mentioned above, indicate the probability that a point belongs to that class.

[0058] Figure 15 shows a flowchart of the essential steps of the training procedure for the network based on Figure 14. In the first step S1, the points of the point cloud are arranged and placed into a sequence based on one of the methods described above. In the freely chosen step S2, additional features may be added as needed, for example, an index that further characterizes the space-filling paths 42. Finally, the network is trained in the usual way in step S3, for which a CTD operation with an arbitrarily chosen weight matrix is ​​applied to the training data, the result is compared to the correct classification of the points to calculate the loss function, which in this case is minimized by fitting the weights in the weight matrix, for example, by backpropagation, and this process is then repeated with further training data. In this case, the weight matrix may be constrained to provide at least approximate translational invariance.

[0059] Figure 16 is a flowchart illustrating the network's operation during actual semantic segmentation. The first two steps, S4 and S5, correspond to steps S1 and S2 during the training run based on Figure 15. Step S6 performs the actual segmentation and evaluates the network. Step S7 performs post-processing, such as NMS (Non Maximum Suppression), to remove errors as needed.

[0060] Figure 17 shows an alternative neural network architecture in which two different sequencing methods are combined. The input data 56 is sorted in the first sequencer 68, first in ascending order based on the x-coordinate and second in ascending order based on the y-coordinate. In parallel, the input data 56 is sorted in the second sequencer 70, first in ascending order based on the y-coordinate and second in ascending order based on the x-coordinate. These sequencers are followed by either a CTD layer 72 or 74, followed by either an activation stage 76 or 78. The activation stage 78 is followed by a permutation stage, in which the points in the sequence are rearranged so that the point sequence corresponds to the sequence generated in sequencer 68. The solutions from the activation stage 76 and the permutation stage 80 are combined in the synthesis stage 82, thus yielding 32 output variables instead of 16 per point. Following the synthesis stage is a second CTD layer 84, in which the number of output variables is again reduced to 16, followed by the same softmax activation 66 as in Figure 14. This different sequencing allows the method to better adapt to various geometries of the point cloud. Generally, it is useful to sort based on point coordinates first, as point coordinates result in a relatively large spread of the point cloud.

[0061] In a further variation, networks with more than two edges can also be used, within which points are sequenced in different ways, for example, by using polar coordinates, by a first sort based on azimuth in ascending order and a second sort based on distance, or vice versa. Permutations of points can also be considered before passing them to a second CTD layer. [Explanation of Symbols]

[0062] 10 Radar sensors 12 processors 14 2D point cloud 16 Sequencer 18 Sequences 20 Neighborhoods 22. First Layer 24 Convolutional Neural Networks 26 units 28. Second Layer 42 Space-filling paths, Plane-filling paths 44 point cloud 46, 48, 50, 52 points 54 Edge i (ordinal number)

Claims

1. A method for semantic segmentation of a point cloud (44) by a neural network (24) in a driver assistance system for an automobile, wherein a neighborhood (20) is defined for each individual point (46, 48, 50, 52) of the point cloud (44), the neighborhood (20) being a set of other points of the point cloud located near the point, and the features of a single point are convolved together with the features of points in the neighborhood of the single point according to a learned weight matrix, wherein the points of the point cloud are A method characterized in that the points are organized into a sequence (18), for which each point is assigned an ordinal number (i) indicating the position of the point within the sequence (18), an algorithm is used to create the sequence, the algorithm ensures that the difference between the ordinal numbers of each pair of points is positively correlated with the spatial distance between the points within the point group (44), and that the neighborhood (20) of a single point is defined as a set of points, and the ordinal numbers of the points are a sequence of successive numbers that encompass the ordinal number of the single point.

2. The method according to claim 1, wherein the point coordinates of the point are treated as a generalized feature of the point.

3. The method according to claim 1 or 2, wherein the sequence (18) is created by sorting the points based on the coordinates of at least one point.

4. The method according to claim 3, wherein the points are sorted firstly based on a first point coordinate and secondly based on a second point coordinate, and the sorting order is selected depending on the geometry of the point cloud.

5. The method according to claim 1 or 2, wherein the space occupied by the point cloud is filled with space-filling paths (42), and the sequence (18) is created by sorting the points in the order in which they meet along the space-filling paths (42).

6. The method according to any one of claims 1 to 5, wherein a plurality of sequencing methods are combined with each other to create the sequence (18).

7. The method according to any one of claims 1 to 5, wherein multiple sequences (18) are created by different sequencing methods, the sequences are then further processed in parallel, and finally the results are merged.

8. The method according to any one of claims 1 to 7, wherein sufficient translational invariance with respect to the translation of points in the point cloud in the space is produced by setting constraints on the components of the weight matrix.

9. The method according to any one of claims 1 to 8, wherein the solution of the aforementioned convolution operation is subjected to a further convolution operation using the same neighborhood (20).

10. A driver assistance system for an automobile, comprising a data processing system configured to perform the method according to any one of claims 1 to 9.

Images