Device and method for generating a tag object of a vehicle's surroundings

Abstract

The present invention relates to a method and a labeling system for generating a label object, wherein the label object is used for symbolic description of an object in the environment of a mobile device (e.g., a robot or a vehicle). The label object (620) includes at least one attribute of the object (120) at a first time (t1) based on an observation (420) of the object (120). The method comprises the following steps: selecting from the observations (420): a first observation (421) detected at a first time (t1), a second observation (422) detected at a second time (t2), wherein the second time (t2) is a time before the first time (t1), and a third observation (423) detected at a third time (t3), wherein the third time (t3) is a time after the first time (t1); and obtaining at least one attribute of the object (120) using the selected observations (421, 422, 423).

Description

Technical Field

[0001] The invention relates to a method and a labeling system for generating label objects for symbolic description of objects in the environment or surroundings of a mobile device, such as a robot or a vehicle, in particular a mobile device capable of at least partially automated movement. The invention also relates to a program element, a computer-readable medium and an application. Background Art

[0002] For a mobile device (e.g. a robot or a vehicle), it is important to symbolically describe one or more objects of the environment or surroundings of the mobile device. In order to symbolically describe the objects, the objects are detected, for example, by a camera, and attributes are provided to the objects, for example, by a human operator, for example, based on image data of the surroundings of the vehicle. For example, the first detected object may be provided with the attribute "house with a certain color", or the second detected object may be provided with the attribute "vehicle with a certain speed". This process is also referred to as "marking" objects. The detected and "marked" objects may be referred to as "labeled objects". However, this manual method may be both time-consuming and expensive. Summary of the invention

[0003] One aspect of the present invention relates to a method for generating a label object from observation of an object, the label object comprising at least one attribute of the object at a first moment, the method comprising the following steps:

[0004] – Select from observations

[0005] The first observation detected at the first moment,

[0006] a second observation detected at a second time instant, wherein the second time instant is a time instant before the first time instant,

[0007] a third observation detected at a third time, wherein the third time is a time after the first time;

[0008] Using the selected observation, for example using a labeling module, at least one property of the object is determined.

[0009] A tag object associates an observation with a tag. A tag includes a symbolic description of a static and / or dynamic object - in particular a time-dependent description. A tag object can be seen as a representation of an object in the surroundings of a mobile device (e.g. a robot or a vehicle that is at least partially driven automatically). Objects of the surroundings (i.e., to a certain extent the "real world") can be, for example, houses, trees, another vehicle, a roadway, a guardrail and / or other objects. Attributes can be, for example, "house", "tree", "vehicle"; attributes can, for example, include a color and / or include the attribute: "static" or "dynamic" tag object. In addition, tags can also relate to regulatory elements (regulatorische Elemente) - for example entry restrictions or speed limits.

[0010] Static tags describe static objects with the help of their properties. In this case, static objects are objects with zero velocity - for example, houses, trees, guardrails, traffic signs, landmarks or so-called ground planes (ground features that can be driven and cannot be driven). Dynamic objects or moving objects (for example, other traffic participants, such as people or other vehicles) can have a non-zero velocity. Dynamic objects can have a time-dependent state; dynamic objects can be described, for example, by a trajectory. The tag of a dynamic object describes it with the help of its properties (for example, position, velocity, acceleration, orientation, etc.). A dynamic object can be another object in space - for example, another traffic participant, for example, a person P at position s (for example in x / y coordinates) with velocity v, acceleration a, direction w.

[0011] The robot may be, for example, a cleaning vehicle, a lawn mower, or agricultural equipment. The vehicle may be a land vehicle, in particular a car, a transporter, a truck, a land vehicle, or an amphibious vehicle.

[0012] The observations can be detected, for example, with the aid of one or more sensors. The one or more sensors can be sensors of one or more types, or different or identical comparison mechanisms of cameras and / or radar sensors can be used. The sensor can in particular be a sensor of a mobile device. The object is detected by the sensor and, for example, is stored as an observation in a memory of the control system. In this case, the memory can be arranged in the mobile device, for example inside a robot or a vehicle, the memory can be arranged outside the mobile device, for example in a server or another vehicle, or the memory can also be divided into a plurality of memories inside and outside the mobile device. In the memory, the surroundings detected by the mobile device, for example the own vehicle ("host vehicle"), can be stored as an observation sequence, wherein the surroundings are stored in a plurality of memories according to the sensor modality used. Different properties can be stored. In this case, each element of the observation sequence can have a time stamp. In the case of a camera, for example, the observations can be stored unprocessed as a raw image ("rawimage") or after preprocessing (for example with the aid of an equalizer).

[0013] Here, the observation includes at least one first observation, a second observation and a third observation. Here, the first observation is detected at a first moment. The first moment can be the real present. In particular, in the case of offline processing (e.g. with the aid of multiple observations), the first moment can be a moment that is before the real present and has been stored in the form of a file. In this case, the first moment can be defined as a reference moment - to a certain extent as the "present for measurement", i.e. during the detection of the object.

[0014] The second moment is a moment before the first moment, so that the second moment is in the past relative to the first observation. In this case, the second moment may be days, hours, minutes, seconds, or only a fraction of a second earlier than the first moment. For example, in the case of a camera and / or radar sensor (i.e. in the case of a device capable of taking multiple images per second), the second moment may be a moment at which the previous "image" has been taken. The previous "image" may also relate to a sequence of images, i.e. the second moment may include multiple moments.

[0015] The third moment is the moment after the first moment, that is, the third moment is in the future relative to the first observation. If the first moment is, for example, in the real present, the third observation, for example, can be formed by a prediction system. In the case of (for example, by means of multiple observations) offline processing, the third moment can be a moment later than the first moment. The third moment can be a few days, hours, minutes, seconds, or only a fraction of a second later than the first moment. The time distance from the third moment to the first moment can be the same as the time distance from the second moment to the first moment. The third moment can include multiple moments and can, for example, include a sequence of moments. The moments can have equidistant, dynamically configured and / or dynamically configurable time intervals.

[0016] In the case of using the selected observation, the labeling module obtains at least one attribute of the object. Sometimes the observation or multiple observations are referred to as unlabeled samples. Sometimes the observation or multiple observations with at least one attribute are referred to as labeled samples. Therefore, the labeled sample represents a plurality of recorded or stored observations. The labeled sample may include a symbolic description of the vehicle environment. This obtaining of the attribute can be achieved, for example, by adding a single or multiple object attributes (labels) to multiple observations. Thus, symbolic abstract attributes of the object can be generated. Here, the labeling module determines the object and its attributes, for example, by processing the observation with the aid of one or more pattern recognition algorithms and multi-target tracking algorithms. The obtaining uses the first observation, the second observation and the third observation here, i.e., the obtaining takes into account the present, past and future of the observation to a certain extent. This is sometimes referred to as the overall processing (holistische Verarbeitung) of the observation. In this overall processing, the label belonging to the determined moment is automatically generated by using the observations before, at and after the determined moment. By the overall processing of the observation, a particularly high accuracy and reliability of the label can be achieved. In particular, it has been shown that in many cases, labels generated in this way can achieve higher accuracy and reliability than using other algorithmic methods that only consider the time before and at the detection moment but not after this moment, as is the case in the present invention.

[0017] This label is also called a reference label or ground-truth label. In this case, the label refers to one or more objects in the surrounding environment (i.e. in the real world). The objects can be static objects or dynamic objects. For example, a reference label or ground-truth label can be used to train a device that uses a machine learning method.

[0018] The method can be used for both offline and online annotation. For online annotation, the following variant can be used: This variant takes into account the details of online processing and / or processing in a mobile device (e.g., a robot or vehicle independent of the network) - such as limited computing time and / or storage space.

[0019] In one embodiment, each of the observations consists of a single observation, in particular a plurality of single observations, and these single observations are respectively detected by means of different contrast mechanisms. For example, a single observation can be detected by means of a plurality of sensors or a single sensor. For example, different contrast mechanisms can be detected by means of different types of sensors. Observations using multiple sensor types are sometimes referred to as "multimodal". For example, different contrast mechanisms can be detected by means of the same sensor type with different pre-processing and / or post-processing. Examples can be different attached elements (e.g. different lenses in the case of a camera), or different filters and / or algorithms for post-processing. Thus, for example, measurements specific to the sensor modality can be performed - such as measurements of pixel values, position measurements, velocity components, etc., etc.

[0020] Individual observations can also be detected with the aid of the same comparison mechanism and with the aid of sensors of different quality values ​​(qualitative Wertigkeit). For example, a standard sensor can be used in combination with a so-called reference sensor. Here, the reference sensor can have a higher quality (for example, in terms of a higher signal-to-noise ratio, a higher imaging accuracy and / or a higher spatial or temporal resolution). The reference sensor can be a calibrated sensor. In addition to the multimodal sensor in one embodiment, one or more of the following sensors can also be involved here: the sensor is used for automated driving functions and is arranged in the vehicle or is arranged to be connected to the vehicle. These additional reference sensors can improve the accuracy of the overall generated label by fusing the additional sensor data with the data of the standard sensor group. Compared with the accuracy and / or range of the standard sensor group, the reference sensor has at least as good, and in some cases even better, accuracy and range of action. The reference sensor can be used to check the standard sensor.

[0021] In one embodiment, the individual observations are fused into the observations before they are used to determine the properties of the object. Here, data from different sensors and / or different sensor types are fused. For example, the data provided by a camera and a radar sensor can be semantically connected in a manner associated with the same object, so that the property estimation of the object is based on information from different sensors. The label defining the object can be used for each individual in a multi-modal sensor (e.g., radar, lidar, or camera data), or the label can also be used for a combination of multiple sensor types. This can be used, for example, for evaluation and verification applications, or it can also be used for machine learning methods.

[0022] In one embodiment, the classification includes a comparison of the first observation with the second observation and / or with a third observation.

[0023] In one embodiment, the step of obtaining includes classification or regression analysis, wherein the classification or regression analysis includes: classifying the constructed label object into types, and / or classifying the constructed label object into states or performing regression analysis on the states of the constructed label object. Type classification is sometimes also referred to as "object type classification". Type classification may include, for example, attributes such as "house", "vehicle", "yellow", "reflective". State classification is sometimes also referred to as "state type classification". State classification may include, for example, attributes such as "static" or "dynamic". Other types of classifications of observations may also be used.

[0024] In one embodiment, the state classification includes classifying the tag object as a static tag object or a dynamic tag object. A static tag object may be, for example, a house, a guardrail, or a parked vehicle. A dynamic tag object may be, for example, a moving bicycle or a moving vehicle. The state classification may be performed by means of a variety of methods. One of these methods may, for example, include a difference analysis (for example, between the first, second, and third observations) or a model-based process. Here, if the position change is not analyzed, the annotation system classifies the observation as a static tag object by means of a difference analysis of the relevant measurement data or by means of a model-based process. For example, if the difference analysis analyzes the position change (i.e., if the position change measured by the sensor is not caused solely by the movement of the mobile device itself), the annotation system classifies the observation as a dynamic tag object. Static tag objects may obtain classification updates. Therefore, for example, a parked vehicle that has been classified as a static tag object may move; in this case, the same vehicle is reclassified as a dynamic tag object. However, if a vehicle that has been classified as a dynamic tag object stops, the vehicle remains a dynamic tag object, but it has the attribute "Speed ​​= 0".

[0025] If the difference analysis finds the disappearance or unreasonable appearance of a stored object, the annotation system classifies the label object as a disturbance object. A disturbance object can be, for example, a temporary artifact (also called a "ghost") that has no corresponding relationship in the real world. Classification as static, dynamic or clutter label objects (Clutter-Label-objekt) constitutes part of the properties of a label object.

[0026] In one embodiment, the labeling system is also configured to analyze the sequence of labels (object attributes) and / or observations by means of another difference analysis, and / or the labeling system is configured to generate an association between observations and objects represented by means of labels. In addition, the association between measurements and objects can also be added to the labeled samples in the form of other labels, which are automatically generated by unlabeled samples. Compared with the (first) difference analysis, the other difference analysis and / or association generation can include longer sequences of measurements and / or labels. The other difference analysis / association can also benefit from overall processing, in that the estimation of the attribute at a certain moment and / or the association at the certain moment take into account the sensor data before, at and after the moment. The other difference analysis or association can also be implemented as an iterative process. Here, when the values ​​calculated in the iteration (for example, for a certain attribute) and / or the association of the sensor measurement with the object represented by means of the label converge, the iterative process is terminated. This can be used, for example, to first generate multiple possible hypotheses when there is ambiguity (Mehrdeutigkeit) in the observation, and in subsequent processing steps, the multiple possible hypotheses are weighed against each other and made credible or untrustworthy. This can improve the accuracy of transmitting the measurement data to the labeled samples. In one embodiment, the iterative process for estimating the object properties at a certain moment can use sensor data before, at and after the moment (i.e., the overall processing can be performed). In one embodiment (such as offline processing), the iterative process does not need to meet real-time conditions, so the iterative process may be relatively very time-consuming and computationally intensive. For example, the iterative process can be implemented on a server or other particularly high-performance processor system. The results of the iterative process can then be provided to a processor system in the mobile device that has less computing performance, at least in some cases.

[0027] In one embodiment, the classification includes the use of a model. Here, the model can be used for type classification and / or state classification. Thus, for example, a model of a house or a vehicle can be used. For example, motion models that interact with each other can also be used - for example, so-called interacting multiple models (IMM). The model can involve a model trained based on training data. In this case, the training data represents a sample of recorded and labeled sensor data. For example, the model can represent a conditional statistical distribution of the sensor data or a conditional statistical distribution of features extracted from the sensor data, which conditional statistical distribution is related to the type or state to be classified or to be determined by regression analysis. The conditional distribution is estimated from the training data.

[0028] In one embodiment, the classification includes using an artificial neural network. For example, the artificial neural network is trained with a series of classified label objects (e.g., with different types of moving and non-moving vehicles, pedestrians, houses, guardrails, etc.). The accuracy of the classified label objects can then be further improved with other methods.

[0029] In one embodiment, the classification includes the use of a Bayesian filter, such as a Kalman filter. In this case, a Bayesian filter or a Kalman filter can be used, for example, to estimate object properties. In particular, errors that appear as unstable dynamics in the sensor measurement sequence can be reduced. In order to further improve the accuracy of the description of the temporal behavior, an interacting motion model, such as a so-called interacting multiple model (IMM), can be used.

[0030] In one embodiment, the classification or regression analysis includes the use of a Bayesian smoother, in particular a fixed interval Bayesian smoother. In one embodiment, an optimal fixed interval smoothing filter (Rauch-Tung-Striebel-Filter) can be used to estimate the object properties. Therefore, due to the holistic processing, the system can benefit from a higher estimation accuracy, because in this case, the measurements before, at and after a certain moment are taken into account in the estimation of the effective property at this moment.

[0031] In one embodiment, the classification includes the use of a graph-based optimization. In this case, the nodes of the graph represent object properties (labels) and the edges of the graph represent observations and physical constraints or motion models of dynamic objects. The graph defines an error function that is related to the measurement data, the object properties, the sensor model and the motion model. By means of iterative optimization, the total error of the graph, which is the sum of the errors of each edge, can be minimized by finding the best solution in the state space of the object properties in the nodes. This is a global optimization, since each of the measurements contained in the graph can have an influence on all parameters (properties) to be optimized of the graph.

[0032] In one embodiment, the classification and / or regression analysis comprises the following steps:

[0033] – Generate a temporary static label object, a temporary dynamic label object or a temporary nuisance label object;

[0034] - Associating an observation with a temporary tag object and determining based thereon whether the measurement on which the observation is based is a static measurement, a dynamic measurement or a noise measurement;

[0035] - completing the types and properties of the temporary static tag objects and / or dynamic tag objects according to the respectively associated observations;

[0036] – evaluating a temporary static label object or a temporary dynamic label object according to a quality metric and / or a quality metric;

[0037] In response to the quality metric and / or the quality metric being below a predetermined threshold, branching back to the associated step;

[0038] In response to the quality metric and / or the quality metric reaching or exceeding a predetermined threshold value, generating a final label object using a temporary static label object or a temporary dynamic label object.

[0039] These steps can also be performed iteratively with a certain number of iterations. The number of iterations can be determined as a predefined number. The number of iterations or the end of the iterative method can be defined by a termination criterion. The termination criterion can be derived, for example, from a quality criterion, for example from an estimate of the distance to the object, for example, when the first estimate of the distance deviates from the second estimate of the distance by less than 30 cm.

[0040] In one embodiment, using the first observation includes comparing the first observation with a list of example objects. Here, when processing sensor measurements, the annotation module can access example objects (e.g., models) that are stored in a model memory. The model can involve a model specific to the sensor modality, i.e., for example, there may be a model for processing or recognizing camera images and a model independent of the above model for processing or recognizing radar measurements. Additionally, there may be a model for sensor data fusion. In addition, there may be a model (motion model) that is used to describe the temporal characteristics of dynamic objects, and thereby achieve or improve object tracking. The model may involve a model generated based on training data, or may involve a manually defined model with parameters that are defined based on physical relationships or optimized based on empirical values. The automatically trained model may involve a model trained with the aid of an artificial neural network, for example, a model trained with the aid of a so-called deep neural network. By using a list of example objects, for example, other attributes (e.g., attributes of type classification) can be added to the attributes of the label object.

[0041] Another aspect of the present invention relates to a labeling system for generating a label object from observations of an object, the label object comprising at least one attribute of the object at a first moment, the labeling system having:

[0042] A memory configured to store observations, wherein the observations include:

[0043] The first observation detected at the first moment,

[0044] a second observation detected at a second time instant, wherein the second time instant is a time instant before the first time instant,

[0045] a third observation detected at a third time instant, wherein the third time instant is a time instant after the first time instant,

[0046] A labeling module is provided for determining at least one property of the object using the selected observation.

[0047] In one specific embodiment, each of the observations consists of individual observations, wherein the individual observations are respectively detected by means of different comparison mechanisms.

[0048] In one specific embodiment, each of the observations consists of individual observations, wherein the individual observations are each detected by means of the same comparison mechanism and by means of sensors with different quality values.

[0049] In one specific embodiment, the individual observations are each fused to form the observation before the observations are used to determine the properties of the object.

[0050] In one embodiment, the obtaining includes a classification or regression analysis step, wherein the classification or regression analysis includes classifying the constructed label object into types and / or classifying the constructed label object into states or performing regression analysis on the states of the constructed label object.

[0051] In one implementation, the state classification includes classifying the tag object as a static tag object or a dynamic tag object.

[0052] In one embodiment, the classification includes a comparison of the first observation with the second observation and / or with a third observation.

[0053] In one embodiment, the classification or regression analysis includes using a model and / or using an artificial neural network and / or using a Bayesian filter and / or using a Bayesian smoother and / or using a graph-based optimization.

[0054] In one embodiment, the memory is arranged in the mobile device (for example, inside the robot or vehicle) and / or arranged to be connected to the robot or vehicle, and the labeling module is arranged on a server outside the mobile device. Here, the list for the model and object properties can also be arranged in a server outside the mobile device. Therefore, in this embodiment, during driving, the sensor data measured in the mobile device is stored on a permanent storage medium, and after the driving is completed, the sensor data is transmitted to a server computer or a server cluster outside the mobile device. The generation of the label is carried out on the server computer or the server cluster. Here, the labeling system can use the following server hardware: the server hardware has higher performance than the hardware usually available in the mobile device. This allows the use of algorithms that cannot be used due to the requirements of computing power and storage capacity in the mobile device, and due to these algorithms, higher accuracy and reliability of the generated labels can be achieved. In addition, a machine learning method can also be used, which is based on a form or a combination of different forms, and the machine learning method generates additional information or directly generates a reference label by means of overall processing of the sensor data, and the additional information is considered to improve the accuracy of the automatically generated reference label.

[0055] In one embodiment, the memory is arranged in the cloud and the labeling module is arranged on a server outside the mobile device. The list of model parameters and object properties can also be arranged on a server outside the mobile device. Here, during driving, the sensor data measured in the mobile device can be transmitted to a server computer or a server cluster outside the mobile device, for example, via a wireless Internet connection. The generation of the label is carried out on the server computer, on the server cluster or in the cloud.

[0056] In one embodiment, the memory and the annotation module are arranged in the mobile device. The memory and the list can also be arranged in the mobile device. In one embodiment, the list in the mobile device includes a true subset of the model on the server. In this case, in order to accommodate the smaller computing performance and / or storage capacity of the processor system (as often used in mobile devices) compared to the server, a selection is made from the measured data so that reference tags are automatically generated only for a part of the measured data.

[0057] Another aspect of the invention relates to a robot or a vehicle, in particular an at least partially automated vehicle, having a labeling system (eg as described above and / or below).

[0058] A further aspect of the invention comprises a program element which is configured to carry out one of the methods when it is executed on a processor unit.

[0059] Another aspect of the invention comprises a computer-readable medium on which the program element is stored.

[0060] Another aspect of the invention relates to the use of a labeling system as described above for generating labeled samples, in particular for the development of at least partially automated vehicles.

[0061] In addition, measures for improving the present invention will be further illustrated below in combination with the description of preferred embodiments of the present invention based on the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings show:

[0063] Figure 1 Schematically illustrates a system according to an embodiment of the present invention;

[0064] Figure 2 schematically illustrates a timing diagram for detecting a single observation according to one embodiment of the present invention;

[0065] Figure 3 Schematically illustrates a part of a marking system according to an embodiment of the present invention;

[0066] Figure 4 Schematically illustrates a part of a marking system according to another embodiment of the present invention;

[0067] Figure 5 Schematically illustrates the configuration of a system according to an embodiment of the present invention;

[0068] Figure 6 Schematically illustrates another configuration of a system according to an embodiment of the present invention;

[0069] Figure 7 Schematically illustrates another configuration of a system according to an embodiment of the present invention;

[0070] Figure 8 A method according to one specific embodiment of the present invention is shown. DETAILED DESCRIPTION

[0071] Figure 1A system with a labeling system 200 according to one embodiment of the present invention is schematically shown. The labeling system 200 is connected to a sensor group 300 having a plurality of individual sensors 301, 302. The individual sensors 301, 302 may have or use different contrast mechanisms. Here, in the embodiment shown, the first individual sensor 301 is implemented as a camera (schematically represented by a lens), and the second individual sensor 302 is implemented as a radar sensor; that is, the second individual sensor 302 is of a different sensor type than the individual sensor 301 and uses a different contrast mechanism. The sensor group 300 is configured to detect raw measurement values ​​of an object 120 in the surroundings or environment 100, and transmit the raw measurement values ​​to a memory 400 via a wire or interface 310. The content of the raw measurement values ​​is sensor-specific; the raw measurement values ​​may, for example, relate to pixel values, position data (e.g., radar position) or determined properties of an object (e.g., velocity components), etc. Typically, not only measurement values ​​of the object are generated, but also background measurements (e.g., background pixels in an image) or false positive measurements (e.g., ghost positions of radar sensors). The object 120 may be one object or (as shown here) multiple objects. The sensor group 300 captures each of the objects 120 multiple times (e.g., in an equidistant time series) at a first time t1, a second time t2, and a third time t3. The measurements of the object 120 as well as the background measurements and the false positive measurements are then stored in the memory 400 as observations 420 or a sequence of observations 420. Here, the observations 420 have: a first detection observation 421 detected at the first time t1, a second detection observation 421 detected at the second time t2, and a third detection observation 423 detected at the third time t3.

[0072] The annotation system 200 also has an annotation module 500. The annotation module converts the sensor observation 420 into a sequence of labels 620, wherein the annotation module 500 processes the observation 420 with the aid of an overall algorithm to generate the label 620. When processing the observation 420, the annotation module 500 accesses the model 570 stored in the model memory 550. The model may be a model specific to the sensor modality - for example, there may be a model for processing or recognizing camera images and a model independent of this for processing or recognizing radar measurements. In addition, there may be a model for sensor data fusion. In addition, there may be a model (motion model) that is used to describe the temporal characteristics of dynamic objects and thereby achieve or improve object tracking. The model may be a model generated based on training data, or may be a manually defined model that has parameters defined based on physical relationships or parameters optimized based on empirical values. The automatically trained model may be a model trained with the aid of an artificial neural network (e.g., a deep neural network).

[0073] Figure 2 A time diagram for detecting a single observation according to one embodiment of the present invention is schematically shown along the time axis indicated by t. Here, at times t1, t2, t3, a single sensor 301 and 302 (see Figure 1 ) detects a single observation. Here, the single sensors 301, 302 may have or use different comparison mechanisms. Thus, the first single sensor 301 may detect a single observation 421a at a first time t1. The first single sensor 301 may use a first comparison mechanism to detect the single observation 421a; the single observation is represented by a triangle. The first moment may be, for example, the real present. The first moment may also (especially in the case of offline processing) be a moment before the real present, which has been stored in the form of a file, for example. In this case, the first moment may be defined as a reference moment - to a certain extent as the "present for measurement", i.e., during the detection of the object. At this first moment t1, the second single sensor 302 also detects a single observation 421b. The second single sensor 302 may use a second comparison mechanism to detect the single observation 421b; the single observation is represented by a circle. The single observations 421a and 421b may then be combined, in particular fused, into an observation 421. The combined observation 421 is represented by a rectangle. In addition, the first single sensor 301 and the second single sensor 302 further detect a single observation 422a or 422b at a second time t2. The second time t2 is in the past relative to the first time t1. The single observations 422a and 422b can be combined, for example, fused into observation 422. In addition, the first single sensor 301 and the second single sensor 302 detect a single observation 423a or 423b at a third time t3. The third time t3 is in the future relative to the first time t. In the case of offline processing, the third time t3 can be a later detection time than the first time. In the case of online processing, the detection at the third time can include a prediction. The time distance from the third time to the first time can be the same as the time distance from the second time to the first time. The single observations 423a and 423b can be combined, for example, fused into observation 423.

[0074] Figure 3A portion of a labeling system 200 according to an embodiment of the present invention is schematically shown. Here, a classification or association module 510 is shown, which classifies sensor observations 420 including observations 421, 422 and 423 in the memory 400 as static measurements 522 and / or dynamic measurements 524 and / or noise measurements (false positive measurements) 526. This can be achieved, for example, by means of a difference analysis of the sensor observations 420 with the properties of the following objects: the object was previously generated by "spawning", tracked by "tracking" in the case of dynamic objects, and the properties of the object are represented by the representation 620. The result of the difference analysis is an association with the existing object, which leads to an assignment as static, dynamic or "noisy". In some sensor modalities, this assignment can also be achieved at least partially directly based on the measured properties (for example, speed in the case of radar), or this assignment can be achieved by a model-based classifier learned in the training data (for example, by means of a neural network). If the difference analysis concludes that the sensor observation 420 is associated with a static object, or if the model-based classification concludes that the sensor observation 420 comes from a static object, the observation 420 is classified as a static measurement 522. No position changes mean: in the case of a non-moving mobile device, the sensor observations 420 in the sequence of at least two elements are only subject to changes due to sensor noise or changing ambient conditions; in the case of a moving mobile device, the movement can be calculated from the at least two elements. Some annotation systems 200 use a Kalman filter in the difference analysis; this improves the analysis in particular in the case of a moving vehicle. Observation 522 is a sensor observation 420 classified as a static measurement. In contrast, observation 524 is an observation 420 classified as a dynamic measurement. Finally, observation 526 is an observation 420 classified as a noise or a noise label object. Reference numeral 622 denotes a property derived from a measurement or a matched copy of a previously derived property (e.g., position, size) of a static object, the properties of which are stored in the object memory 600, and reference numeral 624 denotes a derived or matched property (e.g., velocity) of a dynamic object, the properties of which are stored in the object memory 600. The final object property 620 is set taking into account all the properties 622 or 644 set or matched in time. Thus, 622 or 624 is the overall optimized property. The property 622 or 624 is set or matched taking into account the observation 522 or 524, respectively. In the case of some sensor modalities (e.g., radar), each observation 420 (e.g., radar position) is classified either as a static measurement 522, a noise measurement 526, or a dynamic measurement 524. In this case, the observation 420 is the same as one of the measurements 522, 526, or 524, and the other two measurements do not exist for the observation.In the case of other sensor modalities (e.g., cameras), a plurality of processed measurements 522 and 524 may be derived from observations 420 (e.g., images from a camera). In this case, 522 and 524 are also referred to as features. In the example of camera images, 522 may relate to segmented static objects in the image, and 524 may relate to segmented dynamic objects in the image.

[0075] If observations 420 are classified as static, dynamic, or noisy, then models 570 stored in model memory 550 may be used. If processed measurements or features 522, 526, and / or 524 are generated from observations 420 alternately (depending on the sensor modality), then models stored in model memory 550 (see also ) are also typically used. Figure 1 ) model 570.

[0076] Figure 4 A part of a tagging system 200 according to another embodiment of the present invention is schematically shown. Figure 4 To a large extent Figure 3 The same reference numerals represent the same parts throughout. Figure 4 and Figure 3 The difference is that an iterative process is performed to determine the static attributes 622 and the dynamic attributes 624. In this case, the iterative process takes into account a longer sequence of observations 420 than in the classification / association module 510. In addition, in the case of a sequence of dynamic attributes 624, a Bayesian smoother is also used, such as an optimal fixed interval smoothing filter (RTS filter).

[0077] Figure 5 The configuration of a system according to an embodiment of the present invention is schematically shown. During driving, observation or sensor data 420 measured in the vehicle 110 are stored in a permanent storage medium 400. After the driving is completed, the sensor data 420 are copied to a memory 400a on a server computer or a server cluster 150 outside the vehicle 110 via a transmission path 401. The server computer or server cluster 150 can also be implemented as a cloud. Next, the analysis of the data and the generation of the label 620 as described above are performed on the server 150.

[0078] Figure 6Another configuration of a system according to another embodiment of the present invention is schematically shown. Here, during driving, the sensor data measured in the vehicle is copied to a server cluster or a memory 400b on the cloud 160 outside the vehicle 110 via a wireless Internet connection 402 (or via another transmission path). Next, the data is copied from there to a memory 400a on the server 150 via a transmission path 403. Then, the analysis of the data and the generation of the label 620 as described above are performed on the server 150. Alternatively, the analysis of the data and the generation of the label can be performed directly on the server cluster in the cloud 160, thereby eliminating the copying to the memory 400a via the transmission path 403.

[0079] Figure 7 Another configuration of a system according to an embodiment of the present invention is schematically shown. In this case, sensor data 420 measured in the vehicle 110 are processed in the vehicle 110 itself by means of the annotation system 200. For example, the sensor data 420 can be detected by means of three sensors 300 placed at different locations in the vehicle 110 and can be stored in the memory 400 as a sequence of observations 420. The annotation system 200 has Figure 3 or Figure 4 However, for reasons of weight, volume and / or power consumption, in particular the memory for the model parameters is designed to be small and the processor system performance is designed to be weak.

[0080] Figure 8 1 and 2. A method according to an embodiment of the present invention is shown. In step 701, by placing different positions (such as Figure 7 ) to detect the object 120 (see Figure 1 ), and the object can be stored in the memory 400 as a sequence of observations 420. In step 702, the stored object 420 is converted into a set of labels 620 with the help of the labeling system 200 and stored in the label memory 600. The sequence of sensor observations is called an unlabeled sample. The combination of the sequence of sensor observations and the generated labels is called a labeled sample. Depending on the required accuracy and reliability of the labels, according to the accuracy that can be achieved with the sensors of the sensor group and the algorithm of the labeling system, the labels 620 can be improved in another step 703 in the following way: a human operator manually corrects possible errors and inaccuracies of the labels with the help of suitable tools for displaying and visualizing the sensor raw data and the generated labels in order to produce an improved version of the label set. To this end, the display tool can support the change of the labels according to the input of the human operator.

Claims

1. A method for generating a label object (620) from an observation (420) of an object (120), the label object comprising at least one attribute of the object (120) at a first time (t1), the method comprising the following steps: Select from the observations (420): a first observation (421) detected at said first instant (t1), A second observation (422) detected at a second time (t2), wherein The second moment (t2) is a moment before the first moment (t1), a third observation (423) detected at a third time (t3), wherein the third time (t3) is a time after the first time (t1); At least one property of the object (120) is determined using selected observations (421, 422, 423), wherein the selected observations (421, 422, 423) are classified as static measurements, dynamic measurements, or noise measurements by means of a difference analysis of the selected observations (421, 422, 423), wherein if the difference analysis does not analyze a position change of the object (120), the selected observations (421, 422, 423) are classified as static measurements, if the difference analysis analyzes a position change of the object (120), the selected observations (421, 422, 423) are classified as dynamic measurements, and if the difference analysis results in a false positive measurement of the object (120), the selected observations (421, 422, 423) are classified as noise measurements.

2. The method according to claim 1, in, Each of the observations (421, 422, 423) is composed of a single observation (421a, 421b; 422a, 422b; 423a, 423b), wherein the single observations (421a, 421b; 422a, 422b; 423a, 423b) are detected by means of different contrast mechanisms, respectively, and / or Each of the observations (421, 422, 423) consists of a single observation (421a, 421b; 422a, 422b; 423a, 423b), wherein the single observations are respectively detected by means of the same comparison mechanism and by means of sensors with different quality values.

3. The method according to claim 2, in, Before the observations (421, 422, 423) are further used to determine the properties of the object (120), the individual observations (421a, 421b; 422a, 422b; 423a, 423b) are respectively fused into the observations (421, 422, 423).

4. The method according to any one of the preceding claims, in, The obtaining includes the steps of classification or regression analysis, The classification or regression analysis includes: classifying the constructed label object (620) by type, and / or classifying the constructed label object (620) by state or performing regression analysis on the state of the constructed label object (620).

5. The method according to claim 4, in, The classification comprises a comparison of the first observation (421) with the second observation (422) and / or with the third observation (423), and / or The state classification includes classifying the tag object (620) as a static tag object or a dynamic tag object, and / or Wherein, the classification includes usage models, and / or Wherein, the classification includes using an artificial neural network, and / or Wherein, the classification includes using a Bayesian filter, and / or Wherein, the classification or the regression analysis includes using a Bayesian smoother, and / or Wherein said classification or said regression analysis comprises using graph-based optimization.

6. The method according to claim 4, in, The classification and / or regression analysis comprises the following steps: Generate a temporary static label object, a temporary dynamic label object or a temporary noise label object; Associating (510) the observation (421, 422, 423) with a temporary tag object and determining based thereon whether the measurement on which the observation (421, 422, 423) is based is a static measurement, a dynamic measurement or a noise measurement; According to the respectively associated observations (421, 422, 423), the type and attributes of the temporary static label object and / or the temporary dynamic label object are improved; evaluating the temporary static label object or the temporary dynamic label object according to a quality metric and / or a quality metric; In response to the quality metric and / or the quality metric being below a predetermined threshold, branching back to the step of associating; In response to the quality metric and / or the quality metric reaching or exceeding the predetermined threshold, a final label object is generated by using the temporary static label object or the temporary dynamic label object (620).

7. The method according to claim 1 or 2, in, Using the first observation (421) includes comparing the first observation (421) to a list (550) of example objects (570).

8. A labeling system (200) for generating a label object (620) from an observation (420) of an object (120), the labeling system being configured to perform the method according to any one of claims 1 to 7, wherein: The tag object includes at least one attribute of the object (120) at a first time (t1), and the tagging system (200) has: a memory (400) configured to store the observations (420), The observation (420) includes: a first observation (421) detected at said first instant (t1), a second observation (422) detected at a second time (t2), wherein the second time (t2) is a time before the first time (t1), a third observation (423) detected at a third time (t3), wherein the third time (t3) is a time after the first time (t1), A labeling module (500) is provided for determining at least one property of the object (120) using the selected observations (421, 422, 423).

9. The annotation system (200) according to claim 8, in, Each of the observations (421, 422, 423) is composed of a single observation (421a, 421b; 422a, 422b; 423a, 423b), wherein the single observations (421a, 421b; 422a, 422b; 423a, 423b) are detected by means of different contrast mechanisms, respectively, and / or Each of the observations (421, 422, 423) consists of a single observation (421a, 421b; 422a, 422b; 423a, 423b), wherein the single observations are respectively detected by means of the same comparison mechanism and by means of sensors with different quality values.

10. The annotation system (200) according to claim 9, in, Before the observations (421, 422, 423) are further used to determine the properties of the object (120), the individual observations (421a, 421b; 422a, 422b; 423a, 423b) are respectively fused into the observations (421, 422, 423).

11. The marking system (200) according to any one of claims 8 to 10, The obtaining includes the steps of classification or regression analysis, in, The classification or the regression analysis includes: classifying the constructed label object (620) by type, and / or classifying the constructed label object (620) by state or performing regression analysis on the state of the constructed label object (620).

12. The annotation system (200) according to claim 11, in, The state classification includes classifying the tag object (620) as a static tag object or a dynamic tag object, and / or wherein the classification or the regression analysis comprises a comparison of the first observation (421) with the second observation (422) and / or with the third observation (423), and / or Wherein, the classification or the regression analysis includes using a model, and / or Wherein, the classification or the regression analysis includes using an artificial neural network, and / or Wherein, the classification or the regression analysis includes using a Bayesian filter, and / or Wherein, the classification or the regression analysis includes using a Bayesian smoother, and / or Wherein said classification or said regression analysis comprises using graph-based optimization.

13. The annotation system (200) according to any one of claims 8 to 10, further comprising a memory (400), in which the observations (420) are stored, in, The memory (400) is arranged in the vehicle (110), and the annotation module (500) is arranged on a server (150) outside the vehicle (110), and / or wherein the memory (400) is arranged in the cloud (160), and the annotation module (500) is arranged on a server (150) outside the vehicle (110), and / or The memory (400) and the labeling module (500) are arranged in the vehicle (110).

14. A computer program product comprising a program element, which is configured to cause a server (150) or another processor system to execute the method according to any one of claims 1 to 7 when the computer program product is executed on the server or another processor system. 15 . A computer-readable medium having a computer program stored thereon, the computer program comprising instructions for causing a computer to execute the method according to claim 1 when the computer program is executed by the computer.

16. An application method for generating a labeled sample, the application method using the labeling system (200) according to any one of claims 8 to 13 or using the method according to any one of claims 1 to 7 to generate a labeled sample.

17. The application method according to claim 16, wherein: The application method is used for the development of a vehicle that drives at least partially automatically.

Images