Method for determining a three dimensional envelope

Abstract

To provide a method for determining a three dimensional envelope of an object, a data processing device, a program product and a computer-readable data carrier.SOLUTION: A computer-implemented method for determining a three dimensional envelope (16), preferably a cube, for an object represented on a first data (12), the first data (12) comprising at least three spatial dimensions, the method comprising receiving information data of a two dimensional minimum enclosing polygon for the object represented on a second data, receiving a projection matrix defining a mapping of three dimensional data points (14) of the first data (12) to two dimensional data points in the second data, A three dimensional envelope (16) of the object is found by using a mathematical optimizing method and target functions (Z1, Z2, Z3). It contains terms below 100000 (T1, T2, T3, T1 ', T2 ', T1 ", T2 ").SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for determining a three-dimensional envelope.

[0002] The invention also relates to an apparatus for processing data, including means for carrying out the method described above.

[0003] The invention further relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the above method.

[0004] The invention further relates to a computer-readable data carrier having stored thereon the computer program product described above. [Background technology]

[0005] Autonomous and semi-autonomous driving have the potential to transform mobility, for example, by reducing journey times, energy consumption, and / or emissions. 3D object recognition technology is attracting attention as a key element for autonomous driving, and in recent years, efforts to develop 3D object recognition technology based on machine learning have been gaining attention.

[0006] Existing approaches to 3D object recognition technology can basically be divided into two groups depending on whether the output data is 2D image data or a 3D point cloud, usually generated by a LiDAR sensor. For methods based on image data captured by a camera, estimating the 3D envelope of an object from 2D image data is a major challenge. However, based on rapid developments in the field of machine learning, especially deep learning techniques, image-based 3D recognition has made remarkable progress.

[0007] However, one drawback of image-based 3D recognition techniques using machine learning is that statistical algorithms, such as artificial neural networks, must be trained using a large amount of training data, which is time-consuming. During training, the learning algorithm learns a function that exhibits the desired behavior based on the features of the training data. However, such a trained function is a black box because the relationship between the individual terms of the trained function and the individual features of the image data is unknown. Essentially, the trained function describes behavior in the form of an equation, but does not take into account the complexity of the problem. Therefore, such black box models are not suitable for obtaining evolving knowledge.

[0008] In other words, how dynamically and nonlinearly programmed systems such as artificial neural networks, deep learning systems, and / or genetic algorithms arrive at their results is unclear to users. This black-box system aspect is particularly problematic in the field of (semi-)autonomous vehicles, where the ever-increasing use of AI systems shifts accountability more and more to the software and its results. For example, in the field of liability law, there is a starting point for the need for interpretive rules for the use of AI.

[0009] Improvements in 3D object recognition are also needed for algorithms such as neural networks, which must be trained on large datasets in a supervised learning framework before they can be used. The data used for this training is annotated in supervised learning, meaning that the classification results that the algorithm needs to learn are already predicted. In other words, the annotated data includes bounding boxes around 3D objects and labels that indicate that the 3D object is a vehicle. Providing such annotated data has traditionally been labor-intensive, sometimes manual, resulting in significant time and cost. Summary of the Invention [Problem to be solved by the invention]

[0010] Starting from this, the object of the present invention is to enable 3D object recognition in a simple, fast and / or cost-effective manner. [Means for solving the problem]

[0011] This problem is solved by the subject matter of the independent claims. Preferred developments emerge from the dependent claims.

[0012] Thus, according to the present invention, there is provided a computer-implemented method for determining a three-dimensional envelope, preferably a cube, for an object represented on first datum, wherein the first datum includes at least three spatial dimensions, the method comprising: - receiving information data of a two-dimensional minimum enclosing polygon, preferably a two-dimensional minimum enclosing rectangle, for an object represented on second data, wherein the second data includes at least two spatial dimensions, and wherein the received information data defines the size and position of the minimum enclosing polygon of the object in the second data; - receiving a projection matrix defining a mapping of three-dimensional data points of the first data onto two-dimensional data points in the second data; - determining the three-dimensional envelope of the object using a mathematical optimization method and an objective function, where the objective function contains less than 100,000 terms, said step comprising: - initiating information data of a three-dimensional envelope of the object, the information data defining the size, position and orientation of the envelope of the object in the first data; - generating further information data of a two-dimensional minimal enclosing polygon by projection of the enveloping body initiated using the received projection matrix; - comparing the generated information data with the received information data; A method is provided which includes:

[0013] In other words, the method according to the present invention is a mathematical optimization method by which a three-dimensional envelope, preferably a cube, is determined based on a two-dimensional minimum enclosing polygon, preferably a rectangle. The mathematical optimization method is the minimization or maximization of an objective function, where the starting, generating, and comparing steps are preferably performed multiple times until an interruption criterion is reached. In this example, the objective function has fewer than 100,000 terms, more preferably fewer than 10,000 terms, even more preferably fewer than 1,000 terms, and particularly preferably fewer than 100 terms.

[0014] That is, in contrast to the procedures known in the prior art using deep learning techniques, in which an objective function with a very large number of terms (typically more than 100,000 terms) is learned during training using training data, in this example an objective function containing a comparatively small number of terms is optimized. This has the advantage that the objective function is interpretable and explainable and can be used for further knowledge acquisition. Particularly preferably, in this example the objective function is not learned from training data and the determination of the three-dimensional envelope is preferably not performed using machine learning.

[0015] Or, in other words, this is not a black box approach, but rather a white box approach, where the reference of the objective function to each individual parameter can be understood by the user.

[0016] According to this method, a three-dimensional envelope, preferably a convex envelope, particularly preferably a cube, for the object represented on the first data is determined. Preferably, this envelope is an axis-aligned cube. Since the first data has at least three spatial dimensions, the object can basically be visualized by plotting the first data on a three-dimensional grid. The respective values ​​of the data points for the three spatial dimensions are also referred to as spatial coordinates (x, y, z).

[0017] In the first step of the method, information data of a two-dimensional minimum bounding polygon, preferably a two-dimensional minimum bounding rectangle, is received. Preferably, the two-dimensional minimum bounding rectangle is an axis-aligned bounding box (AABB) of an object on the second data. The second data includes at least two spatial dimensions, which means that the object represented on the second data can be represented, preferably graphically, by plotting the second data in a two-dimensional grid. The respective values ​​of the data points for the two spatial dimensions are also referred to as image coordinates (x, y).

[0018] An AABB is a special minimal bounding polygon for an object represented in the second data, where the object preferably touches all four sides of the AABB. A compact, i.e., confined and bounded, 2D object has exactly one AABB. This is the smallest axis-aligned rectangle that encloses the object. The AABB can be found, for example, via a minimum and maximum search on the coordinates of all corner points of the object in the second data.

[0019] That is, the two-dimensional minimum bounding polygon, preferably the AABB, is defined by its information data specifying the size, particularly preferably the lengths of two sides of the rectangle, and the position, particularly preferably the image coordinates of a point in the second data, for example a corner point or the center of the rectangle.

[0020] Preferably, the first data and the second data represent the same object. Preferably, the first data and the second data represent the same object at substantially the same time. That is, the first data and the second data are preferably temporally correlated with each other. In this example, substantially the same time or temporally correlated preferably means that the recording time of the second data, i.e., data having at least two spatial dimensions, is between the start and end of recording of the first data, i.e., data having at least three spatial dimensions.

[0021] In a further step of the method, a projection matrix is ​​received, which defines a mathematical mapping of three-dimensional data points of the first data to two-dimensional data points in the second data, i.e., applying the projection matrix to spatial coordinates generates image coordinates.

[0022] In a further step of the method, a three-dimensional envelope, particularly preferably a cube, is determined by carrying out a mathematical optimization method with an objective function.

[0023] In this case, the starting point is information data of the three-dimensional envelope, which includes the size, position, and orientation of the envelope in the first data. The three-dimensional envelope, i.e., preferably a cube, is defined by its information data, which specify the size, particularly preferably the lengths of the three sides of the cube, the position, particularly preferably the spatial coordinates of points of the cube, such as corner points or the center, and the orientation, particularly preferably three angles, such as the roll-pitch-yaw angles.

[0024] In a further step, information data of a further 2D minimal enclosing polygon, preferably an AABB, is generated by projecting the initiated envelope using the received projection matrix, i.e. preferably the initiated envelope is projected into 2D image coordinates using the projection matrix.

[0025] A consistency check is then preferably performed by comparing the generated information data with the received information data, i.e. the result of the projection is preferably compared with the original received AABB. Preferably, the initiating, generating and comparing steps are performed multiple times until an interruption criterion is reached, in particular with respect to the consistency check.

[0026] According to a preferred development of the invention, it is provided that the first data is a point cloud recorded by means of a lidar or radar sensor, i.e. the first data is preferably sparse data, since a lidar or radar sensor records values ​​only at a few discrete spatial coordinates.

[0027] According to a preferred embodiment of the invention, the second data is image data recorded by a camera, and the projection matrix is ​​particularly preferably defined by the orientation and position of the sensor used to record the first data relative to the camera.

[0028] The received information data of the two-dimensional minimum enclosing polygon can basically be generated by manually annotating the image data recorded by the camera. However, it is preferably assumed that the image data recorded by the camera is evaluated using machine learning to determine the two-dimensional minimum enclosing polygon, preferably the AABB. Particularly preferably, the received information data not only includes the size and position of the two-dimensional minimum enclosing polygon, but also includes substantially vertically oriented line segments identifying the substantially vertically extending edge of the object represented on the second data that is closest to the camera to be recorded. These line segments make it possible to define a meaningful initial value for the orientation of the envelope in the first data at the start of the information data of the three-dimensional envelope of the object, especially when solving a mathematical optimization method. Within the meaning of the present invention, substantially vertical preferably includes a perpendicular line ±45 degrees.

[0029] According to a further preferred embodiment of the invention, the method includes a step of receiving first data, wherein the received data is taken into account when determining the three-dimensional envelope of the object using a mathematical optimization method. Particularly good results can be obtained when determining the three-dimensional envelope if the first data is also taken into account during the optimization. This can be done, for example, by prioritizing three-dimensional envelopes that contain a particularly large number of three-dimensional data points of the first data during the optimization, or by minimizing the number of three-dimensional data points outside the envelope.

[0030] In connection with the mathematical optimization method, it is basically possible to jointly optimize all nine existing variables of the object's three-dimensional envelope (preferably three spatial coordinates for position, three lengths for size, and three angles for orientation) using an objective function. However, according to a preferred embodiment of the present invention, the mathematical optimization method includes optimizing three objective functions, each having less than 100,000 terms, where the first objective function is used exclusively to optimize the position of the three-dimensional envelope, the second objective function is used exclusively to optimize the size of the three-dimensional envelope, and the third objective function is used exclusively to optimize the orientation of the three-dimensional envelope. It has been found that optimizing the position, size, and orientation separately has the advantage that the objective functions are less complex and the terms of the objective functions are related to empirical observations. Preferably, each of the three objective functions has less than 10,000 terms, more preferably less than 1,000 terms, and particularly preferably less than 100 terms. This procedure results in mutual equalization of the optimizations.

[0031] Particularly preferably, according to a further development in this context, it is envisaged that the mathematical optimization method comprises a step of iteratively optimizing, preferably minimizing, three objective functions, starting with a first objective function. It has been shown that the optimization leads to particularly good and particularly fast results if it starts with an optimization of the position of the three-dimensional envelope, followed by the other two optimizations (size and orientation). Preferably, the orientation is optimized after the size optimization. And even more preferably, after the first optimization of the position, size and orientation, respectively, it starts again with an optimization of the position.

[0032] For position optimization using the first objective function, it is assumed that two of the three spatial coordinates are transferred to depth parameters based on their linear dependence. In a clockwise spatial coordinate system in which the z-axis is defined by the depth detection of a lidar or radar sensor and the y-axis is defined by the vertical direction, the spatial coordinates x and z can be transferred to depth parameters. This is possible because, with respect to the existing line of sight of the sensor used to record the first data, these spatial coordinates are linearly dependent on each other by transferring them to image coordinates.

[0033] With regard to the first objective function, according to a further preferred development of the invention, it is provided that the first objective function comprises at least one term, preferably a linear combination of at least three terms, and even with just three terms it is possible to find a three-dimensional envelope that matches very well.

[0034] Regarding the first term, it is preferably assumed in this context that the first term is a 2D-3D consistency term determined by comparing the generated information data with the received information data. Particularly preferably, the first term includes a Jaccard coefficient, also known as intersection over union, for determining the similarity between the backprojected 2D minimum surrounding polygon and the original received 2D minimum surrounding polygon. In this regard, it is particularly advantageous to determine the 2D-3D consistency term by placing the center of the backprojected 2D minimum surrounding polygon at the center of the original received 2D minimum surrounding polygon, since such smaller projection errors have no effect. Also advantageously, the 2D-3D consistency term includes a tolerance term, so that the 2D-3D consistency term has its minimum value (or alternatively its maximum value) even in the case of polygons that do not ideally match.

[0035] Alternatively or additionally, it is preferably envisaged that the second term is a 3D point population term determined by determining the number of three-dimensional data points of the first data within the three-dimensional envelope of the initiated and / or determined object.

[0036] In this context, it is preferred that the method includes a step of receiving first data, i.e., determining how many three-dimensional data points are present inside the three-dimensional envelope. Preferably, when determining the number of three-dimensional data points inside the envelope, and optimizing the position of the three-dimensional envelope, preferably the depth parameter, the envelope is not defined as a cube, but as a cylinder with its axis of rotation parallel to the y-axis, or as a sphere. This has the advantage that, due to the rotational symmetry, errors related to the starting and / or determined orientation of the envelope do not have an effect. It is further preferred that the extension of the cylinder along the axis of rotation is not included in determining the number of three-dimensional data points, since this increases the robustness of the second term to errors related to the contact patch calculation.

[0037] It may be further preferred that the point population terms are normalized by determining the number of three-dimensional data points of the first data that lie within a two-dimensional minimum bounding polygon defined by the received information data after projection with the projection matrix. Furthermore, these point population terms may be normalized by determining the number of three-dimensional data points of the first data that lie within a truncated pyramid or truncated cone having the same degree of divergence as the two-dimensional minimum bounding polygon defined by the received information data.

[0038] Alternatively or additionally, more preferably, the third term is an envelope distance term, which is assumed to be determined by determining the distance from one face of the envelope to a side, preferably to the side of the object facing the lidar or radar sensor. With respect to the first data, the object is scanned by the sensor only on a face aligned with the sensor due to its self-occlusion. Accordingly, advantageously, local information, in this example, preferably the local point density of the first data, is used to identify the face of the object in the first data facing the sensor, and is further used to approximate the three-dimensional envelope as closely as possible to the three-dimensional data points counted in the face. Preferably, during optimization, the envelope distance term prioritizes positions of the three-dimensional envelope where the distance between the three-dimensional envelope and the face of the object facing the sensor is as small as possible.

[0039] It is further preferred that the 2D-3D consistency term of the first objective function includes a distance term, which preferably prioritizes positions of the three-dimensional envelope that have a long separation distance to the lidar or radar sensor. The distance term is preferably linearly related to the distance to the lidar or radar sensor, thus canceling the symmetry of the consistency calculation using the Jaccard coefficient. In the case of first data with particularly low local point density, multiple positions of the three-dimensional envelope may have similarly high values ​​in the objective function including the above three terms, and accordingly, one position cannot be prioritized during optimization. This can be avoided by having the 2D-3D consistency term also include a distance term, which preferably prioritizes positions of the three-dimensional envelope that have a long separation distance to the lidar or radar sensor during optimization, by moving the three-dimensional envelope closer to the surface of the object facing the sensor.

[0040] Particularly preferably, the first objective function comprises a 2D-3D consistency term, a 3D point population term and an envelope distance term. In addition, according to a preferred embodiment of the invention, it is provided that the first term has a higher weighting than the second term, preferably a weighting of 2 times ±20%, and the second term has a higher weighting than the third term, preferably a weighting of at least 50 times. It has been found that these weightings allow a particularly good determination of the three-dimensional envelope using mathematical optimization methods.

[0041] As already mentioned, the size of the three-dimensional envelope is preferably exclusively optimized using the second objective function, in this respect, according to a further preferred development of the invention, it is provided that the second objective function comprises at least one term, preferably a linear combination of at least two terms.

[0042] Preferably, the first term is a 2D-3D consistency term determined by comparing the generated information data with the received information data.Similar to the first objective function, preferably the second objective function also includes a 2D-3D consistency term.

[0043] Alternatively or additionally, it is further preferred that the second term of the second objective function is a prior knowledge term, and the method preferably includes a step of receiving a classification result of the object. That is, for the size of the three-dimensional envelope, a predefined size distribution is preferably used that depends on the object class, and particularly preferably a Gaussian size distribution is used. For example, a first predefined size distribution is used for compact cars and / or a second predefined size distribution is used for trucks. Essentially, the size of objects, particularly vehicles, can be described by a Gaussian mixture model. A mixture model is a model for representing the existence of subpopulations within an overall population. Preferably, the prior knowledge term during optimization prioritizes object sizes that are near the mean and / or median of the corresponding subpopulation, taking into account the classification result. This can be mathematically formulated, for example, via a conditional probability distribution.

[0044] It is furthermore provided that the second objective function comprises a first and a second term. In this regard, according to a preferred development of the invention, it is provided that the first and second terms are weighted equally by ±20%, which leads to particularly good results when determining the three-dimensional envelope.

[0045] The second objective function may further include a 3D point population term determined by determining the number of three-dimensional data points of the first data within the initiated and / or determined three-dimensional envelope of the object. In this regard, it is particularly preferred that, in contrast to the position optimization, the elongation of the envelope along the z-axis is also considered during the size optimization. Furthermore, in this context, it is preferably preferred that the method comprises a step of receiving the first data.

[0046] Similarly, the second objective function, similar to the first objective function, can include an envelope distance term that prioritizes a 3D envelope size where the distance between the 3D envelope and the surface of the object facing the sensor is as small as possible during optimization.

[0047] As already mentioned, the third objective function is preferably used to optimize the orientation of the three-dimensional envelope exclusively. In this regard, according to a further preferred development of the invention, it is provided that the third objective function comprises at least one term, preferably a linear combination of at least two terms.

[0048] Preferably, the first term is a 2D-3D consistency term determined by comparing the generated information data with the received information data. Similar to the first and second objective functions, the third objective function also preferably includes a 2D-3D consistency term. In this context, it is preferably assumed that the orientation-related 2D-3D consistency term takes into account the aspect ratio of the backprojected 2D minimum bounding polygon as well as the aspect ratio of the original received 2D minimum bounding polygon when comparing. In particular, the yaw angle, rotation around the y-axis (vertical axis), affects the aspect ratio of the backprojected AABB.

[0049] Alternatively or additionally, and more preferably, the second term is a 3D point population term determined by determining the number of 3D data points of the first data within the 3D envelope of the initiated and / or determined object. Similar to the first and second objective functions, the third objective function preferably also includes a 3D point population term. However, particularly preferred in this regard is that, in contrast to the first objective function, the envelope is not directly defined as a cylinder or a sphere in the case of the third objective function. Furthermore, it is also possible to avoid including the extension of the 3D envelope along the y-axis (vertical axis) in determining the number of 3D data points, since this increases the robustness of the second term to errors related to determining the ground plane.

[0050] Similarly, the third objective function, similar to the first objective function, can include an envelope distance term that prioritizes an orientation of the three-dimensional envelope during optimization in which the distance between the three-dimensional envelope and the surface of the object facing the sensor is as small as possible.

[0051] Preferably, the third objective function comprises a first and a second term. Furthermore, according to a preferred development of the invention, it is provided that the first term has a higher weighting than the second term, preferably a weighting twice as high ±20%. Such a weighting has been shown to lead to particularly good results when determining the three-dimensional envelope.

[0052] In a further preferred embodiment of the present invention, the mathematical optimization method is implemented using an algorithm that does not use gradients or finite differences to determine the search direction. This means that algorithms from the group of so-called random search algorithms are particularly preferred. These algorithms are particularly suitable for optimizing discontinuous and / or non-derivable objective functions, since they do not rely on gradients or finite differences to find minimum or maximum values.

[0053] In principle, the method can also be used with static first and second data sets, i.e., without a time dimension. In this case, information data, preferably a two-dimensional minimum surrounding polygon, preferably a two-dimensional minimum surrounding rectangle, for an object depicted in a camera image is received, along with first data sets of a lidar or radar sensor recorded in a time-correlated manner. However, the first and second data sets preferably have a time dimension. That is, for the first data sets, a plurality of temporally consecutive lidar or radar sensor data sets are received, and for the second data sets, a plurality of temporally consecutive image data sets are received. Preferably, for the second data sets having a time dimension, object tracking is performed, and information obtained by this object tracking is considered in a mathematical optimization method. Preferably, the temporally consecutive image data sets are compared with each other to identify moving and / or stationary objects and to determine whether the depicted object is also depicted in earlier and / or later image data sets. In the case of a moving object, the trajectory of the object is preferably captured. That is, according to a preferred development of the invention, the first data and the second data each have a time dimension; a) It is assumed that during optimization the determined three-dimensional envelope is taken into account for objects whose shape does not change over time beyond predefined limits. That is, preferably, in the optimization, preference is given to envelopes whose shape does not change over time, and in particular whose size does not change over time, i.e., the optimization utilizes the fact that, graphically, the size of, for example, a vehicle, as represented on successive image data in time, cannot change suddenly.

[0054] More preferably, the first data and the second data each have a time dimension; b) In optimization, - the jumps in the trajectory of the determined three-dimensional envelope do not exceed a predefined size, and / or - the determined three-dimensional envelope does not exceed and / or fall below a predefined instantaneous velocity, and / or It is assumed that it is taken into account that the determined three-dimensional envelope does not exceed and / or fall below a predefined instantaneous acceleration. In other words, the consistency of the trajectory with respect to the position, velocity and acceleration of the envelope is also used in the optimization to prioritize results that are as consistent as possible, i.e., graphically, the optimization makes use of the fact that, for example, vehicles represented on successive image data move through space in a continuous manner.

[0055] The object of the present invention is also achieved by an apparatus for processing data, which comprises means for implementing the method described above.

[0056] Furthermore, the invention relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method described above.

[0057] Furthermore, according to the present invention there is provided a computer readable data carrier having stored thereon the computer program product described above.

[0058] The technical advantages of the apparatus for data processing, the computer program product and the computer readable data carrier will be apparent to those skilled in the art from the description of the determination method for determining the three-dimensional envelope as well as from the examples described below. [Brief explanation of the drawings]

[0059] [Figure 1] FIG. 1 is a schematic diagram illustrating a flowchart of a computer-implemented method for determining a three-dimensional envelope, according to a preferred embodiment of the present invention. [Figure 2]2 is a schematic diagram illustrating the 2D-3D consistency term of the objective function used in the position optimization of the method for determining the 3D envelope from FIG. 1; FIG. [Figure 3] 2 is a schematic diagram illustrating a procedure for determining the 3D point population terms of the objective function used in the position optimization of the method for determining the three-dimensional envelope from FIG. 1; FIG. [Figure 4] FIG. 4 is a schematic diagram illustrating the sum of the function shown in FIG. 2 and the 3D point population term from FIG. 3. [Figure 5] 2 is a schematic diagram illustrating a procedure for determining the envelope distance term of the objective function used in the position optimization of the method for determining the three-dimensional envelope from FIG. 1; FIG. [Figure 6] 2 is a schematic diagram illustrating the 2D-3D consistency term of the objective function used in the orientation optimization of the method for determining the three-dimensional envelope from FIG. 1; FIG. DETAILED DESCRIPTION OF THE INVENTION

[0060] FIG. 1 generally illustrates an exemplary flowchart 10 of a computer-implemented method for determining a three-dimensional envelope, in accordance with a preferred embodiment of the present invention.

[0061] 1, in this embodiment of the method, in a first step S100, information data of a two-dimensional minimum bounding axis-aligned bounding box (AABB) is received from an object represented on image data, which image data has two spatial dimensions and was recorded using a camera.

[0062] The received information data, in this example, defines the size and position of the AABB within the image data. The information data also includes a vertically oriented line segment identifying the vertically extending edge of the object represented in the image data that is closest to the camera to record. Additionally, a classification result for the object represented in the image data is received. In this example, the object represented in the image data is assigned to the class of a passenger car vehicle.

[0063] Further, in this embodiment, in a first step S100, lidar sensor data 12 is received that is temporally correlated with the image data recorded by the camera. This lidar sensor data 12 has three spatial dimensions. Further, a projection matrix is ​​received that defines a mapping of three-dimensional data points 14 of the lidar sensor data 12 to two-dimensional data points in the image data. Exemplary lidar sensor data 12 is shown in FIGS. 3 and 6.

[0064] Subsequently, in step S200, the three-dimensional envelope of the object, in this example in the form of a cube 16 (also shown in FIG. 6), is determined using a mathematical optimization method. As a result of this optimization method, i.e., in step S300, information data for the cube 16 exists, which in this case defines the size, position, and orientation of the cube 16 within the lidar sensor data 12.

[0065] The mathematical optimization method in step S200 involves optimizing three objective functions Z1, Z2, and Z3, each having less than 100 terms in this embodiment, where step S210 optimizes the position of cube 16 using a first objective function Z1, step S220 optimizes the size of cube 16 using a second objective function Z2, and step S230 optimizes the orientation of cube 16 using a third objective function Z3. These three objective functions Z1, Z2, and Z3 are iteratively minimized, where it is apparent from FIG. 1 that first objective function Z1 is minimized, then second objective function Z2, and then third objective function Z3.

[0066] In the optimization, information data for each cube 16 is initiated, back-projected AABB' information data is generated by projecting the initiated cube 16 using a projection matrix, and these generated AABB' information data are compared with the original AABB' information data received in step S100, where this process is repeated until an abort criterion is reached. The received lidar sensor data 12 is also taken into account in the optimization by prioritizing cubes 16 that match the lidar sensor data 12 as closely as possible.

[0067] With further reference to FIGS. 2-6, the three objective functions Z1, Z2, Z3 and the optimization process are described in more detail below.

[0068] The first objective function Z1 for optimizing the position of cube 16 is, in this example, a linear combination of three terms T1, T2, and T3. The first term T1 is, in this example, a 2D-3D consistency term T1, where determining the 2D-3D consistency term involves comparing generated information data with received information data. Figure 2 shows a schematic diagram of the 2D-3D consistency term T1, which can take values ​​between 0 and 1, as seen on the y-axis 18 of Figure 2.

[0069] In this example, the 2D-3D consistency term T1 is essentially defined by the following equation, where IoU stands for Intersection over Union, and the original AABB and the back-projected AABB' have centers that coincide with each other:

number

[0070] Furthermore, it is clear from Figure 2 that the 2D-3D consistency term T1 includes a tolerance term, so that the 2D-3D consistency term T1 has its minimum value of 0 even if the AABBs do not ideally match. The x-axis 20 in Figure 2 corresponds to the position of cube 16 on the z-axis 20 of a right-handed spatial coordinate system, where the z-axis 20 is defined by the depth detection of the lidar sensor and the y-axis is defined by the vertical direction. Thus, graphically, Figure 2 illustrates that cube 16 is not properly positioned at its direct assumed location at the lidar sensor at 0 m, because the back-projected AABB' does not properly match the original AABB. In contrast, at a distance of approximately 75 m to 100 m from the lidar sensor on the z-axis 20, there is good agreement between the back-projected AABB' and the original AABB. Similarly, it can be seen from Figure 2 that the 2D-3D consistency term T1 includes a distance term that slightly favors long distances to the lidar sensor, eliminating the symmetry of the intersection over union.

[0071] The second term T2 of the first objective function Z1 is, in this example, the 3D point population term T2. Figure 3 shows a schematic procedure for determining the 3D point population term T2, while Figure 4 shows the weighted sum of the 2D-3D consistency term T1 from Figure 2 and the 3D point population term T2.

[0072] The 3D point population term T2 is found in this example by first determining the number of three-dimensional data points 14 of the lidar sensor data 12 within a volume co-determined by the cube 16. This volume corresponds in this example to the volume of a cylinder whose axis of rotation is parallel to the y-axis of the spatial coordinate system and extends through the center of the cube 16. We call the subgroup of these points P in Furthermore, P in is P norm It is normalized by dividing by, in this case, P normcorresponds to the number of three-dimensional data points 14 of the lidar sensor data 12 that lie within a truncated cone having the same divergence as the AABB defined by the received information data. Therefore, T2 is determined in this example as follows: T2=1-(P in / P norm )

[0073] That is, diagrammatically, in FIG. 3, cube 16 is shifted along z-axis 14 of the spatial coordinate system to determine how many three-dimensional data points 14 lie within a cylinder extending through cube 16. FIG. 3 also shows all other three-dimensional data points 14 that lie within a truncated cone, and these numbers are used for normalization. FIG. 4 shows the weighted sum of T1 and T2, where T1 is weighted by a factor of 2 and T2 is weighted by a factor of 1, and it is clear that particularly many data points 14 lie within the cylinder at approximately 25 m, 50 m, and 80 m. The weighted sum from T1 and T2 has a minimum value at approximately 80 m.

[0074] The third term T3 of the first objective function Z1 is, in this example, an envelope distance term T3, which is determined by determining the distance between a face of the cube 16 and the side of the object facing the lidar sensor. FIG. 5 schematically illustrates the procedure for determining the envelope distance term T3. In FIG. 5, because only the face of the object (in this example, a passenger vehicle) facing the lidar sensor is scanned, it can be clearly seen that there are no data points 14 on the rear side of the passenger vehicle opposite the lidar sensor. Accordingly, local information in the lidar sensor data 12 (in this example, the local point density of the three-dimensional data points 14) can be used to identify the object face facing the lidar sensor in the lidar sensor data 12. Based on the identified object face, the envelope distance term is determined, and during optimization, a position of the cube 16 is prioritized such that the distance 22 of the cube 16 relative to the object face facing the sensor is as small as possible.

[0075] FIG. 5a) shows a position of cube 16 in lidar sensor data 12 where distance 22 is long and the envelope distance term has a correspondingly high value, while FIG. 5b) shows a position of cube 16 in lidar sensor data 12 where distance 22 is short and the envelope distance term has a correspondingly small value. The position of cube 16 shown in FIG. 5b) is accordingly prioritized during position optimization using first objective function Z1. The third term T3 is weighted least in first objective function Z1.

[0076] That is, in this embodiment, the first objective function Z1 is a linear combination of the following terms T1, T2, and T3: Z1=2T1+1T2+0.01T3

[0077] That is, in step S210, the position is optimized using estimates for the orientation and size of the cube 16. In this example, it is assumed that the received perpendicular line is used for this purpose with respect to the estimate of the orientation of the cube 16. The received classification result is used with respect to the estimate of the size of the cube 16.

[0078] After the optimization using the first objective function Z1 is completed in step S210, the size of the cube 16 is optimized in step S220 using a second objective function Z2 and the optimized value for the position of the cube 16 obtained in step S210. For this purpose, in this embodiment, a second objective function Z2 including two terms T1' and T2' is used.

[0079] The first term T1′ is a 2D-3D consistency term similar to the term T1 of the first objective function Z1, where determining the 2D-3D consistency term includes comparing the generated information data with the received information data.

[0080] The second term T2' is a prior knowledge term T2', which takes into account the classification result of the object. In this example, the prior knowledge term T2' is defined as follows: T2'=1-P(M│K) In this case, M is a Gaussian mixture model of the size distribution of possible class K, and P(M|K) represents the conditional probability, describing the probability for the size of an object under the conditions of the classification result of the object. In this way, during optimization with the prior knowledge term T2′, preference is given to cube 16 sizes that are close to the mean and / or median of the corresponding subpopulation of class K.

[0081] That is, in this example, the second objective function Z2 is the following linear combination of terms T1′ and T2′: Z2=T1'+T2'

[0082] After the optimization using the second objective function Z2 is completed in step S220, the orientation of the cube 16 is optimized in step S230 using a third objective function Z3 and the optimized values ​​for the position and size of the cube 16 obtained in steps S210 and S220. For this purpose, in this embodiment, a third objective function Z3 including two terms T1'' and T2'' is used.

[0083] The first term T1″ is a 2D-3D consistency term similar to the term T1 of the first objective function Z1, where determining the 2D-3D consistency term T1″ includes comparing information data generated by backprojection of the AABB with information data of the received AABB.

[0084] FIG. 6 shows a schematic diagram of the 2D-3D consistency term T1″, which can take values ​​between 0 and 1.4, as seen on the y-axis 18 of FIG. 6. In this example, the 2D-3D consistency term T1″ is defined by the following equation, where SV represents the aspect ratio of the original AABB and SV′ represents the aspect ratio of the backprojected AABB′: T1''=|SV-SV'|+|[(1 / SV)-(1 / SV')] / 2|

[0085] In particular, the yaw angle, or rotation around the y-axis (vertical axis) of the spatial coordinate system, affects the aspect ratio of the backprojected AABB'. The x-axis 24 in FIG. 6 corresponds to the yaw angle of the cube 16, where 0 degrees corresponds to an orientation where the z-axis 20 of the spatial coordinate system defined by the lidar sensor's depth detection corresponds to the normal vector of one side of the cube 16. Thus, graphically, FIG. 6 shows that the cube 16 is well oriented at an assumed yaw angle of 100 degrees or 80 degrees, because the backprojected AABB' closely matches the original AABB.

[0086] The second term T2″ of the third objective function Z3 is a 3D point population term similar to term T2 of the first objective function Z1, except that here the volume of the cube 16 is used, not the volume of the cylinder.

[0087] That is, in this example, the third objective function Z3 is the following linear combination of terms T1″ and T2″: Z3=2T1''+1T2''

[0088] In this embodiment, after the optimization using the third objective function Z3 is completed in step S230, the position of the cube 16 is again optimized using the first objective function Z1 and the optimized values ​​for the size and orientation of the cube 16 obtained in steps S220 and S230. The already optimized position is preferably used as the starting value for the position. Thereafter, as already explained, subsequent optimizations are performed in an iterative process until an interruption criterion is reached. [Explanation of symbols]

[0089] 10 Flowchart 12 Lidar sensor data 14 3D data points 16 3D envelope, cube 18 y-axis of Figures 2, 4 and 6 20 The x-axis in Figures 2 and 4 corresponds to the z-axis of the spatial coordinate system 22 distance 24 The x-axis in Figure 6, yaw angle S100 Input S200 Optimization S210 Position Optimization S220 Size Optimization Optimization of S230 orientation S300 Output Z1 First objective function Z2 second objective function Z3 Third objective function T1 The first term of the first objective function, the 2D-3D consistency term T2: The second term in the first objective function, the 3D point population term T3 The third term of the first objective function, the envelope distance term T1': The first term of the second objective function, the 2D-3D consistency term T2' Second term of the second objective function, prior knowledge term T1'' is the first term of the third objective function, the 2D-3D consistency term T2'' is the second term of the third objective function, the 3D point population term

Claims

1. 1. A computer-implemented method for determining a three-dimensional envelope (16), preferably a cube, for an object represented on first data (12), said first data (12) including at least three spatial dimensions, said method comprising: - receiving information data of a two-dimensional minimum enclosing polygon, preferably a two-dimensional minimum enclosing rectangle, for said object represented on second data, said second data including at least two spatial dimensions, said received information data defining the size and position of the minimum enclosing polygon of said object within said second data; - receiving a projection matrix defining a mapping of three-dimensional data points (14) of said first data (12) onto two-dimensional data points in said second data; - Mathematical optimization method and objective function (Z １ , Z ２ , Z ３ ) to determine the three-dimensional envelope (16) of the object, wherein the objective function has less than 100,000 terms, preferably less than 10,000 terms, and even more preferably less than 1,000 terms (T １ , T ２ , T ３ , T １ ', T ２ ', T １ '', T ２ ''), wherein the determining step comprises: - initiating information data of the three-dimensional envelope (16) of said object, defining the size, position and orientation of said three-dimensional envelope (16) of said object within said first data (12); - generating information data of a further two-dimensional minimal enclosing polygon by projecting the initiated three-dimensional envelope (16) using the received projection matrix; - comparing said generated information data with said received information data; A method comprising:

2. the first data (12) is a point cloud recorded using a lidar or radar sensor and / or the second data is image data recorded using a camera, The method of claim 1.

3. The method includes receiving the first data (12), and taking the received data (12) into account when determining the three-dimensional envelope (16) of the object using the mathematical optimization method.

3. The method according to claim 1 or 2.

4. The mathematical optimization method involves three objective functions (Z １ , Z ２ , Z ３ ) and optimizing a first objective function (Z １ ) is used to optimize the position of the three-dimensional envelope (16) exclusively, and a second objective function (Z ２ ) is used to optimize the size of the three-dimensional envelope (16), and a third objective function (Z ３ ) is used exclusively to optimize the orientation of the three-dimensional envelope (16), and / or The mathematical optimization method is based on the three objective functions (Z １ , Z ２ , Z ３ ) and iteratively optimizing the first objective function (Z １ ), 4. The method according to any one of claims 1 to 3.

5. The first objective function (Z1) includes at least one term (T １ ), preferably including at least three terms (T １ , T ２ , T ３ ) is a linear combination of a) The first term (T １ ) is a 2D-3D consistency term determined by comparing the generated information data with the received information data; and / or b) the method includes receiving the first data (12), and determining a second term (T ２ ) is a 3D point population term determined by determining the number of three-dimensional data points (14) of the first data (12) within the initiated and / or determined three-dimensional envelope (16) of the object; and / or c) The third term (T ３ ) is an envelope distance term determined by determining the distance (22) from the face of the three-dimensional envelope (16) to the side, preferably the distance (22) to the side of the object facing the lidar or radar sensor; The method of claim 4.

6. The first term (T １ ) is the second term (T ２ ), preferably with a weighting of 2 times ±20%, and the second term (T ２ ) is the third term (T ３ ) have a higher weighting, preferably at least 50 times higher, than The method of claim 5.

7. The second objective function (Z ２ ) has at least one term (T １ '), preferably containing at least two terms (T １ ', T ２ '), and a) The first term (T １ ') is a 2D-3D consistency term determined by comparing the generated information data with the received information data; and / or b) The second term (T ２ ') is a prior knowledge term, and the method preferably includes receiving a classification result of the object.

7. The method according to any one of claims 4 to 6.

8. The first term (T １ ') and the second term (T ２ ') are weighted with equal strength ±20%, The method of claim 7.

9. The third objective function (Z ３ ) has at least one term (T １ ''), preferably containing at least two terms (T １ '', T ２ ''), and a) The first term (T １ '') is a 2D-3D consistency term determined by comparing the generated information data with the received information data; and / or b) the method includes receiving the first data (12), and determining a second term (T ２ ''') is a 3D point population term determined by determining the number of three-dimensional data points (14) of the first data (12) within the initiated and / or determined three-dimensional envelope (16) of the object; 9. The method according to any one of claims 4 to 8.

10. The first term (T １ '') is the second term (T ２ '') have a higher weighting, preferably twice as high, ±20%; 10. The method of claim 9.

11. To implement the mathematical optimization method, an algorithm is used that does not use gradients or finite differences to determine the search direction.

11. The method according to any one of claims 1 to 10.

12. the first data (12) and the second data each have a time dimension; a) in said optimization, said determined three-dimensional envelope (16) takes into account that the shape of said object does not change over time beyond a predefined limit value; and / or b) in said optimization, - the jumps in the trajectory of the determined three-dimensional envelope (16) do not exceed a predefined size, and / or - the determined three-dimensional envelope (16) does not exceed and / or fall below a predefined instantaneous velocity, and / or - the determined three-dimensional envelope (16) does not exceed and / or fall below a predefined instantaneous acceleration; is taken into consideration, 12. The method according to any one of claims 1 to 11.

13. 1. A computing environment for data processing, comprising: comprising means for carrying out the method of any one of claims 1 to 12, computing environment.

14. 1. A computer program product comprising: A program comprising instructions that, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 12. Computer program products.

15. A computer-readable data carrier comprising: A computer program product according to claim 14, A computer-readable data carrier.

Images