Method for generating training data for training a machine learning algorithm

Abstract

The invention relates to a method for generating training data for training a machine learning algorithm, said algorithm being designed to determine at least one stixel in image data captured by an optical sensor and, for each of the at least one stixel, the respective distance to an object represented in the at least one stixel from the captured image data. The method (1) has the following step: ascertaining the distance to the object represented in the corresponding stixel on the basis of the image data and information detected using the at least one distance sensor, wherein the step of ascertaining the distance includes a respective process of transforming coordinates in a coordinate system which represents one of the at least one distance sensors into coordinates in a coordinate system which represents a base plane, and the training data is based (5) on the respective image data, a stixel generated therein, and the distance to the object represented in the corresponding stixel.

Description

[0001] Description

[0002] title

[0003] Method for generating training data for training a machine learning algorithm

[0004] The invention relates to a method for generating training data for training a machine learning algorithm and, in particular, to an improved method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0005] Machine learning algorithms are based on the use of statistical methods to train a computer to perform a specific task without having been explicitly programmed to do so. The goal of machine learning is to construct algorithms that can learn from data and make predictions. These algorithms create mathematical models that can be used, for example, to classify data.

[0006] The training of a machine learning algorithm is usually based on training data, for example, using a deep learning method. The training data is labeled data or data provided with corresponding information that represents the actual behavior or conditions of a given application for which the machine learning algorithm is being trained, i.e., a system to be modeled or a corresponding downstream process. Such machine learning algorithms are also used, for example, in the control of driver assistance systems of a motor vehicle and / or functions of an autonomously driving vehicle.A corresponding machine learning algorithm can, for example, be configured to detect objects or obstacles, such as other motor vehicles and / or pedestrians, in image data captured by an optical sensor of a motor vehicle. The driver assistance system and / or the function of the autonomously driving vehicle can then be controlled based on the detected objects of the motor vehicle in such a way that safety-critical situations can be avoided when using or operating the motor vehicle.

[0007] Machine learning algorithms are also known which are designed or trained based on corresponding labeled training data to determine at least one pixel in image data captured by an optical sensor of a motor vehicle and a distance to an object represented in the at least one pixel from the image data. An advantage here is, for example, that no corresponding distance sensor, such as a lidar sensor, is required to determine a distance to the at least one pixel, and the machine learning algorithm can also be used in motor vehicles that only have a camera but no corresponding distance sensor.

[0008] A stixel is a superpixel representation of depth information in an image or image data in the form of a vertical stick or rod that approximates the nearest obstacles within a certain vertical section of the scene.

[0009] However, it turns out to be disadvantageous that often only a small amount of training data is available for training a machine learning algorithm which is designed to determine at least one stixel in image data captured by an optical sensor of a motor vehicle and in each case a distance to an object represented in the at least one stixel from the image data, and known methods for generating additional training data for training a machine learning algorithm which is designed to determine at least one stixel in image data captured by an optical sensor of a motor vehicle and in each case a distance to an object represented in the at least one stixel from the image data, which methods are based, for example, on morphological operations, are often prone to errors.

[0010] From the publication DE 10 2019 200 147.5 a method for providing ground truth data is known, wherein GPS data recorded during a journey are provided, the GPS data are displayed together with routes, and corresponding ground truth data are generated from the GPS data by selecting the GPS data together with the respectively assigned route.

[0011] The invention is therefore based on the object of specifying an improved method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0012] The object is achieved by a method for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to the features of patent claim 1.

[0013] The object is also achieved by a system for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to the features of patent claim 8.

[0014] Disclosure of the invention

[0015] According to one embodiment of the invention, this object is achieved by a method for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the method comprises generating image data, for example image data about an environment of a motor vehicle, wherein the image data is based on information acquired by at least one optical sensor, wherein the at least one optical sensor has at least one distance sensor, determining objects represented in the image data, generating at least one stixel from the image data based on the determined objects, and, for each of the at least one stixel,each comprising determining a distance to the object represented in the corresponding pixel based on the image data and information acquired by the at least one distance sensor, wherein determining a distance comprises transforming coordinates in a coordinate system representing one of the at least one distance sensor into coordinates in a coordinate system representing a ground plane.

[0016] The fact that image data, for example image data about the surroundings of a motor vehicle, is generated means that the image data is directly captured by a corresponding sensor, for example a camera attached to the motor vehicle, or that data captured by other optical sensors of the motor vehicle, in particular a lidar sensor, is converted into image data.

[0017] Image data also refers to data that can be displayed as an image or graphic using a special program. A sensor, also known as a detector, (measured variable or measuring) transducer, or (measured) sensor, is a technical component that can measure certain physical or chemical properties and / or the material composition of its environment qualitatively or quantitatively as a measured variable.

[0018] An optical sensor is defined as a sensor that can detect objects using light or similar radiation. Distance sensors are also sensors that measure the distance between the sensor and an object. An example of such a distance sensor is a lidar sensor, which can detect objects using laser radiation, or objects based on the radiation reflected or backscattered by the objects. Lidar sensors are used, among other things, for distance measurement. Based on the principle of time-of-flight measurement, distances can be determined from the backscattered radiation.

[0019] A coordinate system representing a distance sensor is also understood to be a coordinate system whose origin coincides with the position of the distance sensor and which defines the position and orientation of the distance sensor.

[0020] A coordinate system representing the ground plane or a coordinate system of the ground plane is also understood to be a world coordinate system whose origin lies on the ground plane or ground, and from which the coordinates of points lying on the ground plane can be taken.

[0021] Coordinate transformations or transformations of coordinates also refer to conversions or changes in the coordinate values ​​when changing from one coordinate system to another.

[0022] Thus, through comparatively simple image processing operations and coordinate transformations, a large number of pixels with corresponding distance information, or training data, can be generated for training the machine learning algorithm without the need for complex and resource-intensive adjustments or operations. Furthermore, the process can be implemented using relatively simple sensors already installed in conventional motor vehicles.

[0023] Furthermore, the method, which is based on a coordinate transformation of actually acquired data, is comparatively less prone to errors. Coordinate transformations are particularly used when a problem can be solved more easily in another coordinate system, where the corresponding distance value can be read off the world coordinate system representing the base plane or can be easily derived from the corresponding coordinates, for example, based on known norms.

[0024] Overall, an improved method for generating training data for training a machine learning algorithm is thus specified, which algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0025] In one embodiment, the at least one optical sensor further comprises a camera, wherein the image data is image data captured by the camera, and wherein the step of generating at least one stixel comprises generating a segmentation of the image data.

[0026] Segmentation is the division of image data into individual columns or regions with related content.

[0027] This has the advantage that sufficient training data can be reliably and accurately generated to generate the image data even if no or only a small amount of data is available from the at least one distance sensor. The step of generating a segmentation of the image data can involve applying a machine learning algorithm trained to segment image data. This allows the step of generating a segmentation to be automated and made less error-prone.

[0028] However, the step of generating a segmentation of the image data involving the application of a machine learning algorithm trained to segment image data is only one possible embodiment. Rather, the segmentation can also be generated, for example, by applying other image processing algorithms suitable for generating a segmentation.

[0029] Furthermore, the information acquired by the at least one distance sensor for each of the at least one stixel can each include information about at least one point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected. This has the advantage that the distance to the corresponding stixel can be determined directly without the need for further conversions, which also allows for further resource savings.

[0030] In particular, the information detected by the at least one distance sensor may include information about a point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected and which has at least a predetermined height relative to the base area.

[0031] The fact that a point has at least a predetermined height relative to the ground plane means that the corresponding point, along a normal or orthogonal line passing through the point, has a corresponding distance from an intersection point between the normal and the ground plane. This ensures that the coordinates of the corresponding point in the coordinate system representing at least one distance sensor can be reliably and easily transformed into coordinates in the coordinate system representing the ground plane.

[0032] A further embodiment of the invention also provides a method for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the method comprises generating training data for training the machine learning algorithm, wherein the training data is generated by a method described above for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data,and training the machine learning algorithm based on the generated training data.

[0033] Thus, a method is specified for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, which is based on training data generated by an improved method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.In particular, comparatively simple image processing operations and coordinate transformations can generate a large number of pixels with corresponding distance information, or training data, for training the machine learning algorithm, without the need for complex and resource-intensive adjustments or operations. Furthermore, the method can be implemented using relatively simple sensors already installed in conventional motor vehicles. Furthermore, the method for generating training data, which is based on a coordinate transformation of actually recorded data, is comparatively less prone to errors.In particular, coordinate transformations are used when a problem can be solved more easily in another coordinate system, whereby the corresponding distance value can be read in the world coordinate system representing the base plane or can be derived from the corresponding coordinates in a simple way, for example based on known norms.

[0034] A further embodiment also provides a method for controlling a controllable system by a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the method comprises providing a machine learning algorithm for controlling the controllable system, wherein the machine learning algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data,and wherein the machine learning algorithm has been trained by a method described above for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, and comprises controlling the controllable system based on the provided machine learning algorithm. A controllable system is understood here to mean, in particular, a robotics system, for example, a driver assistance system of a motor vehicle or a function of an autonomously driving motor vehicle.

[0035] Thus, a method for controlling a controllable system based on a machine learning algorithm is specified, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the machine learning algorithm was trained based on training data generated by an improved method for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and a distance to the at least one stixel from the image data.In particular, comparatively simple image processing operations and coordinate transformations can generate a large number of pixels with corresponding distance information, or training data, for training the machine learning algorithm, without the need for complex and resource-intensive adjustments or operations. Furthermore, the method can be implemented using relatively simple sensors already installed in conventional motor vehicles. Furthermore, the method for generating training data, which is based on a coordinate transformation of actually recorded data, is comparatively less prone to errors.In particular, coordinate transformations are used when a problem can be solved more easily in another coordinate system, whereby the corresponding distance value can be read in the world coordinate system representing the base plane or can be derived from the corresponding coordinates in a simple way, for example based on known norms.

[0036] A further embodiment of the invention also provides a system for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the system comprises a first generation unit, which is designed to generate image data, wherein the image data is based on information acquired by at least one optical sensor, wherein the at least one optical sensor has at least one distance sensor, and wherein the system further comprises a first determination unit, which is designed to determine objects represented in the image data, a second generation unit, which is designedto generate at least one stixel from the image data based on the detected objects, and a second determination unit which is designed to determine, for each of the at least one stixel, a distance to the object represented in the corresponding stixel based on the image data and information detected by the at least one distance sensor, wherein determining a distance comprises transforming coordinates in a coordinate system representing the at least one distance sensor into coordinates in a coordinate system representing a ground plane, and wherein the training data are each based on the image data, a stixel generated therein, and a distance to the object represented by the corresponding stixel.

[0037] Thus, an improved system for generating training data for training a machine learning algorithm is specified, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data. Comparatively simple image processing operations and coordinate transformations can generate a plurality of stixels with corresponding distance information, or training data, for training the machine learning algorithm, without the need for complex and resource-intensive adaptations or operations. Furthermore, the system can be implemented using relatively simple sensors that are already installed in conventional motor vehicles.In addition, the system's generation of training data, which is based on a coordinate transformation of actually acquired data, is comparatively less prone to errors. Coordinate transformations are particularly used when a problem can be solved more easily in another coordinate system, where the corresponding distance value can be read off the world coordinate system representing the ground plane or can be easily derived from the corresponding coordinates, for example, based on known norms.

[0038] In one embodiment, the at least one optical sensor further comprises a camera, wherein the image data is image data captured by the camera, and wherein the first generation unit is configured to generate a segmentation of the image data. This has the advantage that sufficient training data can be reliably and accurately generated even if no or only a small amount of data is available to generate the image data.

[0039] The first generation unit can be configured to apply a machine learning algorithm trained to segment image data to segment the image data. This allows the generation of a segmentation to be automated and made less error-prone.

[0040] However, the first generation unit being configured to apply a machine learning algorithm trained to segment image data is only one possible embodiment. Rather, the first generation unit can also be configured to apply other image processing algorithms suitable for generating segmentation.

[0041] Furthermore, the information acquired by the at least one distance sensor for each of the at least one stixel can each include information about at least one point on the object represented in the corresponding stixel at which rays emitted by the lidar sensor are reflected. This has the advantage that the distance to the corresponding stixel can be determined directly without the need for further conversions, which also allows for further resource savings.

[0042] In particular, the information acquired by the at least one distance sensor can include information about a point on the object represented in the corresponding pixel, at which rays emitted by the at least one distance sensor are reflected, and which has at least a predetermined height relative to the base plane. This ensures that the coordinates of the corresponding point in the coordinate system representing the at least one optical sensor can be reliably and easily transformed into coordinates in the coordinate system representing the base plane.

[0043] A further embodiment of the invention also provides a system for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the system comprises a provision unit, which is designed to provide training data for training the machine learning algorithm, wherein the training data is generated by a system described above for generating training data for training a machine learning algorithm, which is designedat least one stixel in image data acquired by an optical sensor and for each of the at least one stixel to determine a distance to an object represented in the at least one stixel from the acquired image data, and a training unit which is designed to train the machine learning algorithm based on the provided training data.

[0044] Thus, a system for training a machine learning algorithm is specified, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data. This system is based on training data generated by an improved system for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data. In particular, a plurality of stixels with corresponding distance information can be generated by comparatively simple image processing operations and coordinate transformations.Training data can be generated for training the machine learning algorithm without the need for complex and resource-intensive adaptations or operations. Furthermore, the system can be implemented using relatively simple sensors already installed in conventional motor vehicles. Furthermore, the generation of training data by the training data generation system, which is based on a coordinate transformation of actually recorded data, is comparatively less prone to errors. Coordinate transformations are used in particular when a problem can be solved more easily in a different coordinate system, where the corresponding distance value can be read off the world coordinate system representing the ground plane or simply derived from the corresponding coordinates, for example, based on known norms.

[0045] A further embodiment of the invention also provides a system for controlling a controllable system based on a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the system comprises a provision unit which is designed to provide a machine learning algorithm for controlling the controllable system, wherein the machine learning algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data,and wherein the machine learning algorithm has been trained by a method described above for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, and a control unit which is designed to control the controllable system based on the provided machine learning algorithm,

[0046] Thus, a system for controlling a controllable system based on a machine learning algorithm is specified, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the machine learning algorithm was trained based on training data generated by an improved system for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.In particular, comparatively simple image processing operations and coordinate transformations can generate a large number of pixels with corresponding distance information, or training data, for training the machine learning algorithm, without the need for complex and resource-intensive adaptations or operations. Furthermore, the system can be implemented using relatively simple sensors already installed in conventional motor vehicles. Furthermore, the generation of training data by the system for generating training data, which is based on a coordinate transformation of actually recorded data, is comparatively less prone to errors.In particular, coordinate transformations are used when a problem can be solved more easily in another coordinate system, whereby the corresponding distance value can be read in the world coordinate system representing the base plane or can be derived from the corresponding coordinates in a simple way, for example based on known norms.

[0047] In summary, the present invention provides a method for generating training data for training a machine learning algorithm and, in particular, an improved method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0048] The described designs and further training courses can be combined as desired.

[0049] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned.

[0050] Short description of the drawings

[0051] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0052] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements shown in the drawings are not necessarily drawn to scale. They show:

[0053] Fig. 1 shows a flowchart of a method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to embodiments of the invention;

[0054] Fig. 2 shows a schematic block diagram of a system for generating training data for training a machine learning algorithm which is configured to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to embodiments of the invention.

[0055] Fig. 3 shows a flowchart of a method for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to a first embodiment of the invention; and

[0056] Fig. 4 shows a flowchart of a method for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to a second embodiment of the invention. In the figures of the drawings, like reference numerals denote like or functionally equivalent elements, parts, or components, unless otherwise stated.

[0057] Fig. 1 shows a flowchart of a method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data 1, according to embodiments of the invention.

[0058] Stixels have proven to be a compact and efficient way of preparing image data captured by an optical sensor, such as a motor vehicle video camera, for further processing by downstream processes, such as driver assistance systems of a motor vehicle or functions of an autonomously driving vehicle.

[0059] A stixel is a superpixel representation of depth information in an image or image data in the form of a vertical stick or rod that approximates the nearest obstacles within a certain vertical section of the scene.

[0060] Such stixels can be determined, for example, based on image data and motion algorithms. Furthermore, machine learning algorithms are also known which are designed or trained based on corresponding labeled training data to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data. An advantage here is, for example, that no corresponding distance sensor, such as a lidar sensor, is required to determine a distance to the at least one stixel, and the machine learning algorithm can also be used in motor vehicles that only have a camera but no corresponding distance sensor.However, it turns out to be disadvantageous that often only a small amount of training data is available for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, and known methods for generating additional training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, are often prone to errors.

[0061] Fig. 1 shows a method 1, which comprises a step 2 of generating image data, for example image data about the surroundings of a motor vehicle, wherein the image data are based on information acquired by at least one optical sensor, wherein the at least one optical sensor has at least one distance sensor, a step 3 of determining objects represented in the image data, a step 4 of generating at least one stixel from the image data based on the determined objects, and a step 5 of, for each of the at least one stixel, respectively determining a distance to the object represented in the corresponding stixel based on the image data and information acquired by the at least one distance sensor, wherein the determining of a distance in each case comprises a transformation of coordinates in a,a coordinate system representing the at least one distance sensor in coordinates in a coordinate system representing a ground plane.,

[0062] Thus, through comparatively simple image processing operations and coordinate transformations, a large number of pixels with corresponding distance information, or training data, can be generated for training the machine learning algorithm without the need for complex and resource-intensive adjustments or operations. Furthermore, the process can be implemented using relatively simple sensors already installed in conventional motor vehicles.

[0063] Furthermore, method 1, which is based on a coordinate transformation of actually acquired data, is comparatively less prone to errors. Coordinate transformations are particularly used when a problem can be solved more easily in another coordinate system, where the corresponding distance value can be read off the world coordinate system representing the base plane or can be easily derived from the corresponding coordinates, for example, based on known norms.

[0064] Overall, an improved method 1 for generating training data for training a machine learning algorithm is thus specified, which algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0065] The machine learning algorithm can, for example, be an artificial neural network.

[0066] According to the embodiments of Fig. 1, the at least one optical sensor further comprises a camera, wherein the image data is image data captured by the camera, and wherein step 4 of generating at least one stixel comprises generating a segmentation of the image data.

[0067] In particular, the individual pixels can be generated by first segmenting the image data, then determining the convex hull for each detected or identified object, and filling the determined convex hull in such a way that it appears as a continuous area. This filling can include filling down to a floor or ground plane, and larger objects can be filled first. The image data are then divided into columns, and for each column, a highest and lowest point, or a start and end point of the convex hull of the object that appears prominently in the corresponding column, or closest to a viewer of the image data, is determined.However, if such a highest and lowest point cannot be determined, the procedure for the corresponding column can be aborted.

[0068] However, if no camera for capturing image data is present, the image data may also be generated based on data acquired by other sensors, for example, a distance sensor such as a lidar sensor, by applying an algorithm to the acquired data which is configured to convert the acquired sensor data into image data.

[0069] The segmentation can also be instance segmentation. This prevents a common convex hull from being created for objects that are represented overlapping or directly adjacent to each other in the image data.

[0070] According to the embodiments of Fig. 1, generating a segmentation of the image data further comprises applying a machine learning algorithm trained to segment image data.

[0071] The information detected by the at least one distance sensor further comprises, for each of the at least one stixel, information about at least one point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected.

[0072] In particular, for each of the at least one stixel, based on the information acquired by the at least one distance sensor, it can be checked whether the information contains information about at least one point on the object represented in the corresponding stixel, which lies between the highest point and the lowest point of the convex hull of the object represented by the corresponding stixel on the corresponding convex hull. If such information cannot be determined, the method for the corresponding stixel or the corresponding column can be aborted.

[0073] According to the embodiments of Fig. 1, the information detected by the at least one distance sensor comprises information about a point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected and which has at least a predetermined height relative to the base area.

[0074] In particular, based on the information about the at least one point on the object represented in the corresponding stixel, which lies between the highest point and the lowest point of the convex hull of the object represented in the corresponding stixel on the corresponding convex hull, the point can be determined which has the shortest distance to the at least one distance sensor and which at the same time has at least the predetermined height relative to the base area.

[0075] Subsequently, based on the detected point or the corresponding information, a distance to the object represented in the corresponding pixel can be determined by transforming corresponding coordinates in a coordinate system representing one of the at least one distance sensors into coordinates in a coordinate system representing a ground plane. If the object is an overhanging object, i.e., an object that protrudes backward or has a protruding part, the determined distance can also be adjusted by applying a corresponding correction algorithm, or overhang correction algorithm.

[0076] Thus, Fig. 1 describes a method which, in addition to at least one stixel, also generates information about a distance from the at least one distance sensor to an object represented in the at least one stixel.

[0077] The training data for training the machine learning algorithm is based on the image data, a stixel generated in it and a distance to the object represented in the corresponding stixel.

[0078] The training data can then be used to train a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data.

[0079] The machine learning algorithm can, for example, be trained based on corresponding labeled training data to control a controllable system, whereby the controllable system can be, for example, a driver assistance system of a motor vehicle or a function of an autonomously driving motor vehicle, such as adaptive cruise control.

[0080] Fig. 2 shows a schematic block diagram of a system for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data 10, according to embodiments of the invention.

[0081] As shown in Fig. 2, the system comprises a first generation unit 11, which is designed to generate image data, wherein the image data is based on information acquired by at least one optical sensor, wherein the at least one optical sensor comprises at least one distance sensor, and wherein the system 10 further comprises a first determination unit 12, which is designed to determine objects represented in the image data, a second generation unit 13, which is designed to generate at least one stixel from the image data based on the determined objects, and a second determination unit 14, which is designed to determine, for each of the at least one stixel, a distance to the object represented in the corresponding stixel based on the image data and information acquired by the at least one distance sensor, wherein determining a distance in each case comprises transforming coordinates in a,a coordinate system representing the at least one distance sensor in coordinates in a coordinate system representing a ground plane, and wherein the training data are each based on the image data, a stixel generated therein and a distance to the object represented in the corresponding stixel.

[0082] The first generation unit can comprise a receiver configured to receive corresponding sensor data and can be implemented based on code stored in a memory and executable by a processor. The first determination unit, the second generation unit, and the second determination unit can each be implemented, for example, based on code stored in a memory and executable by a processor.

[0083] According to the embodiments of Fig. 2, the at least one optical sensor further comprises a camera, wherein the image data is image data recorded by the camera, and wherein the first generation unit 11 is designed to generate a segmentation of the image data.

[0084] In this case, the first generation unit 11 is particularly designed to apply a machine learning algorithm, which is trained to segment image data, in order to segment the image data.

[0085] According to the embodiments of Fig. 2, the information detected by the at least one distance sensor in turn comprises, for each of the at least one stixel, information about at least one point on the object represented in the corresponding stixel at which rays emitted by the lidar sensor are reflected.

[0086] In particular, the information detected by the at least one distance sensor in turn comprises information about a point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected and which has at least a predetermined height relative to the base area.

[0087] The at least one distance sensor can be, for example, a lidar sensor.

[0088] In addition, the illustrated system 10 is designed to carry out a method described above for generating training data for training a machine learning algorithm which is designed to use at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel.

[0089] Fig.3 shows a flowchart of a method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data 20, according to a first embodiment of the invention.

[0090] In a step 21, sensor data is received, in particular image data from a camera and dense lidar data, or lidar data with a high detection density or a lidar point cloud with high density.

[0091] Subsequently, in a step 22, based on the lidar point cloud, lidar ground points (Ground), i.e. points representing a ground, and points representing an object (L_obstacle), are determined, for example by means of a plane fit algorithm.

[0092] In a step 23, the lidar ground points are then projected onto the camera image using camera and lidar extrinsics, camera intrinsics and a corresponding projection model in order to obtain a depth image for the represented objects.

[0093] In a step 24, this depth image is then divided into bars or vertical sticks of fixed width, whereby in a step 25, corresponding ground points within the corresponding bar are determined for each bar.

[0094] Subsequently, in a step 26, the point closest to the camera point or center of the camera parallel to the ground direction is determined for each rod.

[0095] In a step 27, the ground point closest to this point within the corresponding rod is then determined, whereby based on this ground point, the point of the corresponding object representation closest to the ground point and then the highest point of the corresponding object representation are determined.

[0096] Fig. 4 shows a flowchart of a method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data 30, according to a second embodiment of the invention.

[0097] In a step 31, sensor data is received, in particular a semantically segmented image from a camera and a sparse point cloud from a lidar sensor. In a step 32, a convex hull is then calculated to extend the representation of the objects above the ground, for example, vehicles, pedestrians, animals, or guardrails, to the ground. In a step 33, the semantically segmented image is then divided into bars or vertical sticks of fixed width. In a step 34, for each bar, the lidar is projected onto the corresponding bar and the nearest 3D point is determined that lies within a range between a highest point and a lowest point of a corresponding object representation.

[0098] Subsequently, in a step 35, the intersection point between the direction vector of the camera and the orthogonal to the ground plane is calculated in order to determine the one more precise lowest point of the object.

Claims

Claims 1 . A method for generating training data for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the method (1) comprises the following steps: Generating image data, wherein the image data is based on information acquired by at least one optical sensor, and wherein the at least one optical sensor comprises at least one distance sensor (2); Determining objects represented in the image data (3); Generating at least one stixel from the image data based on the detected objects (4); and For each of the at least one stixel, in each case determining a distance to the object represented in the corresponding stixel based on the image data and information detected by the at least one distance sensor, wherein the determining of a distance comprises in each case transforming coordinates in a coordinate system representing one of the at least one distance sensor into coordinates in a coordinate system representing a ground plane, and wherein the training data are in each case based on the image data, a stixel generated therein and a distance to the object represented in the corresponding stixel (5).

2. Method (1) according to claim 1, wherein the at least one optical sensor further comprises a camera, wherein the image data is image data recorded by the camera, and wherein the The step of generating at least one stixel (4) comprises generating a segmentation of the image data.

3. The method (1) of claim 2, wherein generating a segmentation of the image data comprises applying a machine learning algorithm trained to segment image data.

4. Method (1) according to one of claims 1 to 3, wherein the information detected by the at least one distance sensor for each of the at least one stixel each comprises information about at least one point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected.

5. The method (1) according to claim 4, wherein the information detected by the at least one distance sensor comprises information about a point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected and which has at least a predetermined height relative to the base area.

6. A method for training a machine learning algorithm which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, the method comprising the following steps: Providing training data for training the machine learning algorithm, wherein the training data is generated by a method for generating training data for training a machine learning algorithm, which is designed to include at least one stixel in image data captured by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the to determine the acquired image data generated according to one of claims 1 to 5; and Training the machine learning algorithm based on the provided training data.

7. A method for controlling a controllable system based on a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, the method comprising the following steps: Providing a machine learning algorithm for controlling the controllable system, wherein the machine learning algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, and wherein the machine learning algorithm has been trained by a method for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to claim 6; and Controlling the controllable system based on the provided machine learning algorithm.

8. System for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and for each of the at least one stixel a distance to an object represented in the at least one stixel from the acquired image data, wherein the system a first generation unit which is designed to generate image data, wherein the image data is based on information acquired by at least one optical sensor, wherein the at least one optical sensor has at least one distance sensor, and wherein the system further comprises a first determination unit which is designed to determine objects represented in the image data, a second generation unit which is designed to generate at least one stixel from the image data based on the determined objects, and a second determination unit which is designed to determine, for each of the at least one stixel, a distance to the object represented in the corresponding stixel based on the image data and information acquired by the at least one distance sensor, wherein determining a distance in each case comprises transforming coordinates in a,a coordinate system representing the at least one distance sensor in coordinates in a coordinate system representing a ground plane, and wherein the training data are each based on the image data, a stixel generated therein and a distance to the object represented in the corresponding stixel.

9. The system (10) according to claim 8, wherein the at least one optical sensor further comprises a camera, wherein the image data is image data captured by the camera, and wherein the first generating unit (11) is configured to generate a segmentation of the image data.

10. The system (10) of claim 9, wherein the first generating unit (11) is configured to apply a machine learning algorithm trained to segment image data to segment the image data.

11. System (10) according to one of claims 8 to 10, wherein the information detected by the at least one distance sensor for each of the at least one stixel respectively contains information about at least one point on the object represented in the corresponding stixel, at which rays emitted by the lidar sensor are reflected.

12. System (10) according to claim 11, wherein the information detected by the at least one distance sensor comprises information about a point on the object represented in the corresponding stixel at which rays emitted by the at least one distance sensor are reflected and which has at least a predetermined height relative to the base surface.

13. A system for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the system comprises a provision unit, which is designed to provide training data for training the machine learning algorithm, wherein the training data were generated by a system for generating training data for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to one of claims 8 to 12, and a training unit,which is designed to train the machine learning algorithm based on the provided training data., 14. System for controlling a controllable system based on a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, wherein the system comprises a provision unit, which is designed to implement a machine learning algorithm for controlling the controllable system to provide, wherein the machine learning algorithm is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, and wherein the machine learning algorithm has been trained by a method for training a machine learning algorithm, which is designed to determine at least one stixel in image data acquired by an optical sensor and, for each of the at least one stixel, a distance to an object represented in the at least one stixel from the acquired image data, according to claim 13, and a control unit which is designed to control the controllable system based on the provided machine learning algorithm.

Images