Method for generating designation data and method for adjusting a model

Abstract

The invention relates to a method for generating designation data for adjusting a model which represents an environment based on points with designations, said method comprising: providing a message (M), which has a message time (tm), wherein for one or each of several objects in the environment, the message comprises information about a position, and information about a designation (B) of the object; determining a ground truth for a selected state of the model, based on the one object or at least one of the several objects, in each case comprising: providing the designation of the object for at least one point, corresponding to the position, in the selected state of the model; and providing the ground truth as designation data.

Description

[0001] Description

[0002] title

[0003] Method for generating label data and method for fitting a model

[0004] The present invention relates to a method for generating label data for adapting a model that depicts an environment based on points with labels, a method for adapting such a model, as well as a system for data processing and a computer program for carrying out the same.

[0005] Background of the invention

[0006] Models of an environment can be used in various fields. For example, in the transportation sector or other areas where vehicles or other mobile devices move within the environment, such models can be useful for coordinating movement or planning movement, and the like. Such models can, for example, be based on points with labels that represent the environment.

[0007] Disclosure of the invention

[0008] According to the invention, a method for generating designation data, a method for adapting such a model, as well as a data processing system and a computer program for implementing the method are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0009] The invention generally deals with models that depict an environment based on labeled points. The points represent objects or parts of objects in the environment, and the label indicates which object it is or to which object or type of object a particular point belongs. In this sense, one can also speak of a classification of the points. An example of such points are so-called point clouds or point sets, which can be determined, for example, using environmental sensors such as radar sensors or the sensor data acquired with them.

[0010] Such a model can, for example, include or be based on a deep neural network; however, other types of machine learning models are also conceivable.

[0011] Such a model can represent the environment, in particular from the perspective of a specific device. The device can be, for example, a base station, i.e., a stationary device or apparatus, but can also be a mobile device that moves or is intended to move in the environment, in particular, a vehicle that moves at least partially automatically.

[0012] As mentioned, the model can be determined based on sensor data, such as from one or more radar sensors. For this purpose, the (raw) sensor data can first be processed into a set of points (point cloud), each of which then maps or represents an object or a part (e.g., a part of a surface) of an object. For example, such a point corresponds to a location on the object where a radar signal was reflected. In this respect, such a point cloud then represents the environment, particularly from the perspective of a device that includes the radar sensor.

[0013] Several such point clouds can then be combined or matched to obtain the model. In other words, a first point cloud can represent an initial state of the model, and with each new point cloud obtained, the model can be expanded and / or improved. Thus, with each newly added point cloud, a new state of the model can be obtained.

[0014] The new sensor data or the points determined with it can always come from the same sensor, but this does not have to be the case. Point clouds based on different sensors can also be combined. However, in this case, care must be taken to consider the relative positions and / or orientations of the sensors.

[0015] While such a model can, in itself, represent the environment well, the individual points, or at least some of them, should also contain a label or classification, or such a label or classification should be assigned to the points. Only then can the model provide information about what a specific object is, for example, a vehicle, a house wall, a traffic sign, a tree, or the like.

[0016] It has now been found that there is an exchange of messages between devices in various areas. These messages each contain information about a position for one or each of several objects in the environment, and information about a designation (e.g. a class) of the object. The information about the position can, for example, include the position directly, but also a so-called bounding box for the position. Such a bounding box specifies a spatial area in which the actual position of the object lies with a certain probability. In addition, such messages can also contain information about a speed and an orientation of the objects. In addition, such messages can also contain information about a position, possibly also with a bounding box, a speed and an orientation of the device sending the message.

[0017] The information about the objects can be collected using various sensors of the device sending the message, such as lidar sensors, radar sensors, cameras and other types of sensors.

[0018] A concrete example of such messages are so-called CPM messages (CPM stands for "Collective Perception Message"). These types of messages are exchanged between participants in road traffic, for example, and serve to mutually communicate various information, for example, to warn of potential collisions. Such messages are transmitted via wireless communication links such as mobile communications, but also via other signal transmission methods.

[0019] Such messages are now used to generate designations or, in general, designation data for an aforementioned model. For this purpose, a message with a message time is provided; this message can, for example, have been received in a device such as a base station. The message time specifies a time at which the message was sent, for example, and can therefore be a time according to the message's timestamp. For one or each of several objects in the environment, the message comprises information about a position and information about a designation of the object. As mentioned, the information about the position can, in particular, comprise a bounding box.

[0020] Furthermore, a ground truth is then determined for a selected state of the model, based on the one or at least one of the multiple objects. In this case, the label of the object is determined for at least one point corresponding to the position in the selected state of the model. In particular, a comparison can be made between the positions of objects from the message and the positions of points in the model. In particular, if the positions are present as bounding boxes, it is possible, for example, to check which points of the model or of the selected state of the model lie in which bounding box. The relevant label belonging to the bounding box can then be assigned to the relevant points. This ground truth is then provided as the label data. The label data is generated by providing the ground truth as label data.

[0021] At this point it should be mentioned that, depending on the situation, the position, speed and orientation of the sender of the message may need to be taken into account in order to obtain the position of the objects from the perspective of a device for which the model applies or is to be used. This takes advantage of the fact that the CPM messages mentioned contain information about the positions and labels of objects in the environment. This information is used to obtain labels for a model particularly easily, quickly and accurately, especially since the CPM messages are already available. In particular, a new standard requires CPM messages to include the position with a bounding box with a confidence of 95%, i.e. there is a 95% probability that the actual position of the object lies within the bounding box. This allows particularly precise labels to be created.In principle, the procedure also works with other messages, as long as they contain information about a position and a name of the object.

[0022] Based on such label data, the model can then be adapted. For this purpose, label data generated as mentioned above is provided. Labels are then added to points in the model based on the label data. This can be done, for example, using backpropagation.

[0023] The adapted model can then be deployed. In particular, this can happen whenever a new message is received. In this context, the model can also be referred to as online learning, as the model is then continually adapted and improved in real time.

[0024] In one embodiment, the message for the one or each of the multiple objects in the environment further comprises information about a speed of the object. Then, for the one or at least one of the multiple objects, the position is adapted to the selected state of the model, based on the speed of the object. This can therefore in particular also include adapting the bounding box. In the event that the message time does not correspond, or does not correspond sufficiently precisely, to a time for which the selected state of the model applies, more precise designation data for the model can be obtained in this way. If a message for the one or each of the multiple objects in the environment further comprises information about an acceleration of the object, this can be used for an even more precise adaptation of the position or the bounding box.

[0025] In one embodiment, from a plurality of states of the model at a given time, the state whose time is closest to the message time is determined as the selected state of the model. In this way, a state can be selected that corresponds as precisely as possible—in terms of time—to the information from the message. If necessary or desired, the aforementioned adjustment of the position can then additionally be performed via the speed.

[0026] In one embodiment, one or each of several, in particular all, of the plurality of states are each determined as follows. Sensor data (e.g. as a sensor data set) with a sensor time is provided, wherein the sensor data set has been recorded from the environment by means of an environmental sensor, in particular a radar sensor. Based on the sensor data, a set of points (so-called point cloud) is then generated that maps the environment. Based on the set of points and a current state of the model, a new state of the model is then generated, wherein the sensor time is used as the state time of the new state of the model. In this way, several states of the model can be kept at different times, from which a suitable one is then selected for determining the designation data.A maximum number of states can also be specified, so that when this number is reached, the oldest state can be discarded with each new state.

[0027] In one embodiment, a signal designed as an information signal and / or control signal is output based on the generated designation data and / or the provided adapted model in order to classify environmental sensor data of an environment-sensing, in particular imaging, sensor, e.g. a vehicle- or infrastructure-side camera, in response to the output signal and / or to control a mobile or stationary device, in particular a base station and / or a vehicle, preferably using the classified environmental sensor data.

[0028] A data processing system according to the invention comprises means for executing the method according to the invention or its method steps. The system can be a computer or server, e.g., in a so-called cloud or cloud environment. However, it is also conceivable that such a data processing system is a computer or a control unit in such a mobile or stationary device.

[0029] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0030] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0031] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

[0032] Brief description of the drawings Figure 1 shows schematically an environment for explaining the invention in one embodiment.

[0033] Figure 2a shows schematically a sequence of a method according to the invention in one embodiment.

[0034] Figure 2b shows a diagram to explain the invention.

[0035] Embodiment(s) of the invention

[0036] Figure 1 schematically shows an environment 100 to explain the invention in one embodiment. The environment here represents, by way of example, a road traffic environment. The environment comprises, by way of example, a road 102, two trees 104, 106 next to the road 102, and a house 108. In addition, two vehicles 110a, 110b are located on the road 102. Furthermore, a base station 120 with a radar sensor 122 and a central computing system, which can represent a cloud, for example, are shown. The environment of the base station can be detected by means of the radar sensor 112, and in particular objects in the environment. Possible objects here include, for example, the trees 104, 106, the house 108, and the vehicles 110a, 110b.

[0037] The vehicle 110a, for example, has a radar sensor 112 with which the surroundings of the vehicle 110a can be detected. In addition, the vehicle 110a can also comprise further and / or other sensors for detecting the surroundings and, in particular, objects therein. Possible objects here include, for example, the trees 104, 106, the house 108, and the vehicle 110b. Furthermore, the vehicle 110a has a computing and / or communication system 114, by means of which the vehicle can wirelessly exchange messages 140, e.g., so-called CPM messages.

[0038] Vehicle 110b can be configured like vehicle 110a. Messages 140 can be exchanged, in particular, between vehicles 110a, 110b, but also between a vehicle and base station 120. Base station 120 can, for example, repeatedly detect the surroundings using radar sensor 122. Based on the sensor data detected in each case, a point cloud of the surroundings can then be created; with each new set of sensor data, a model of the surroundings can be updated. This applies equally to, for example, one of the vehicles and its radar sensor.

[0039] Figure 2a schematically illustrates a method according to the invention in one embodiment as a type of flowchart or sequence diagram. This will be explained using an example environment such as that shown in Figure 1. Figure 2b shows a diagram illustrating a temporal sequence with a time t.

[0040] In a step 200, sensor data 202 with the sensor time is provided. The sensor data 202 is acquired, for example, by radar sensor 122 from the environment 100. In a step 204, based on the sensor data 202, a set 206 of points—a so-called point set—is generated that maps the environment. In a step 208, a new state of a model 210 is determined, which model maps the environment 100 based on points with labels. The sensor time is used as the state time of the new state of the model.

[0041] Figure 2b shows, as an example, five sensor times t1 to t5, which can then simultaneously serve as state times, as well as corresponding states X1 to X5 of the model. If no model initially exists, a first state X1 of the model can be generated using the sensor data or points at sensor time t1. Using the sensor data or points at sensor time t2, the state X1 of the model can then be updated as the current state, generating a new state X2 of the model. This can be continued with new sensor data at new sensor times to obtain a (new) state of the model with corresponding state times.

[0042] The previous states are retained, e.g., by caching. A maximum number of states can be specified, for example. Once this limit is reached, the oldest states can be discarded.

[0043] In a process essentially separate from the generation of the model and its states, a message M with a message time tm is provided in a step 220, e.g., after the message has been received in the base station or an executing computing system. The message M can, for example, correspond in type to message 140 according to Figure 1.

[0044] The message M includes information about a position, a speed of the object, and a designation of the object for one or each of several objects in the environment 100. If the message was sent by the vehicle 110a according to Figure 1, possible objects include, for example, the trees 104, 106, the house 108, and the vehicle 110b.

[0045] For any of these objects, the position is represented by a bounding box B, the velocity by v and the label (class) by K.

[0046] In a step 222, from a plurality of states of the model at a respective state time, the state whose state time is closest to the message time tm is selected to obtain a selected state of the model. In Figure 2b, the selected state would be, for example, state X3.

[0047] In a step 224, the position of one or at least one of the plurality of objects is adapted to the selected state X3 of the model based on the object's velocity v. If the object is the house 108 according to Figure 1, it will have a velocity of zero (relative to the base station 120, taking into account the velocity of the vehicle 110a), so no adjustment would be necessary here. If the object is the vehicle 110b according to Figure 1, it will have a velocity other than zero (although this depends on whether the vehicle 110b was moving at the time of detection), so an adjustment would be necessary here.

[0048] Likewise, the object can be the vehicle 110a itself as the sender of the message.

[0049] It should be noted that such an adjustment may not be necessary if the speed is low. If the adjustment is not necessary due to the speed (due to high speed and / or

[0050] In a step 226, a ground truth 228 is then determined for the selected state X3 of the model, based on the one or at least one of the multiple objects. For this purpose, in step 230, the designation B of the object is provided for at least one point in the selected state X3 of the model corresponding to the position. The position is, in particular, the position adjusted as described above, i.e., if applicable, the adjusted bounding box.

[0051] The ground truth can, for example, initially be the empty set, and for each object the intersection between the bounding box of the object and points of the selected state of the model can be supplemented by the relevant label for these points.

[0052] In a step 232, the ground truth is then provided as label data. In a step 234, the labels, based on the label data, are then added to points in the model. This can be done, for example, by means of backpropagation. This can be done, for example, as part of or during training of the model, whereby, in particular, the same training method and the same loss function as for a previous training (e.g., when a model is initially created) can be used. However, a different training method and / or a different loss function can also be used.

[0053] It doesn't matter which state of the model the labels are added to, since the different states are primarily used to determine the ground truth. However, once the labels are present, they are assigned to points that are the same for different states of the model. In a step 236, the model adapted in this way is then provided. These steps can be repeated for each newly received message, so that the model is adapted online, i.e., in real time or runtime. The same applies to determining a new state of the model each time new sensor data is received.

Claims

Claims 1 . A method for generating label data for adapting a model that represents an environment (100) based on points with labels, comprising: Providing (220) a message (M) with a message time (tm), wherein the message (M) comprises, for one or each of a plurality of objects (104, 106, 108, 110a, 110b) in the environment (100), information about a position (P) and information about a designation (B) of the object; Determining (228) a ground truth (230) for a selected state (X2) of the model based on the one or at least one of the plurality of objects, each comprising: providing the label (B) of the object for at least one point corresponding to the position (P) in the selected state (X2) of the model; and Generating (232) the label data by providing (232) the ground truth (230) as label data.

2. The method according to claim 1, wherein the message for the one or each of the plurality of objects in the environment (100) further comprises information about a speed (v) of the object (104, 106, 108, 110a, 110b), the method further comprising: Adjusting (224), for the one or at least one of the plurality of objects, the position (P) at the selected state (X3) of the model based on the velocity (v) of the object (104, 106, 108, 110a, 110b).

3. The method of claim 1 or 2, further comprising: Determining (222), from a plurality of states (X1, X2, X3, X4, X5) of the model at a respective state time, the state whose state time is closest to the message time (tm) as the selected state of the model.

4. The method according to claim 3, wherein one or each of several of the plurality of states (X1, X2, X3, X4, X5) has been determined as follows: Providing (200) sensor data (202) with sensor time (t1, t1 , t3, t4, tt) which have been detected from the environment (100) by means of an environmental sensor, in particular a radar sensor (122); Determining (204), based on the sensor data (202), a set (206) of points that represent the environment (100); Determining (208) a new state (X2) of the model based on the set (206) of points and, if present, a current state (X1) of the model, wherein the sensor time is used as the state time of the new state of the model.

5. A method for adapting a model representing an environment based on points with labels, comprising: Providing (232) label data, wherein the label data has been generated according to a method according to any one of the preceding claims; Adding (234) labels to points in the model based on the label data; and Deploy the customized model.

6. The method of claim 5, wherein adding labels is based on error feedback.

7. Method according to one of the preceding claims, wherein the model comprises or is based on a deep neural network.

8. The method according to any one of the preceding claims, wherein the model depicts the environment (100) of a device, wherein the device is or comprises a base station.

9. The method according to any one of claims 1 to 7, wherein the model depicts the environment (100) of a device, wherein the device is or comprises a mobile device that moves or is intended to move in the environment, in particular a vehicle that moves at least partially automatically.

10. Method according to one of the preceding claims, wherein a signal designed as an information signal and / or control signal is output based on the generated designation data and / or the provided adapted model in order to classify environmental sensor data of an environment-sensing, in particular imaging, sensor in response to the output signal and / or to control a mobile or stationary device, in particular a base station and / or a vehicle.

11. A data processing system comprising means for carrying out the method according to any one of the preceding claims.

12. A computer program comprising instructions which, when executed by a computer, cause the program to carry out the method steps of a method according to any one of claims 1 to 10 when executed on the computer.

13. A computer-readable storage medium on which the computer program according to claim 12 is stored.

Images