Method for training a neural network for determining features of objects for object tracking

Abstract

According to various embodiments, a method is provided for training a neural network for determining features of objects for object tracking, the method comprising: generating a training dataset with a plurality of training data elements, wherein each training data element has a first set of sensor data relating to a first state of an environment with a set of a plurality of objects and a second set of sensor data relating to a second state of the environment, wherein in the second state the positions of the objects have at least partially changed compared to the first state; determining features of the objects by feeding the first set of sensor data to the neural network and determining features of the objects by feeding the second set of sensor data to the neural network; and determining a loss depending on the generated features and training the neural network to reduce the loss.

Description

[0001] Description

[0002] title

[0003] Method for training a neural network to determine features of objects for object tracking

[0004] State of the art

[0005] The present disclosure relates to methods for training a neural network to determine features of objects for object tracking.

[0006] Comprehensive recognition of a vehicle's surroundings forms the basis for driver assistance systems and automated driving functions. The detection, classification, and tracking of objects, such as other road users, are particularly important. Today, a variety of sensors are used to monitor the vehicle's surroundings. These include cameras, radar, LiDAR, and ultrasound systems.

[0007] Object tracking can be achieved using features, so-called re-ID features. However, these are typically determined in such a way that they are "globally" and permanently unique (and not just within a scene in which object tracking is to be performed), which in turn limits the flexibility of their selection and thus impairs their use for object tracking.

[0008] Therefore, more effective approaches for feature extraction for object tracking are desirable. Disclosure of the invention

[0009] According to various embodiments, a method for training a neural network for determining features of objects for object tracking is provided, comprising:

[0010] • Configuring a neural network having a network architecture such that, when fed with a set of sensor data containing one sensor data item for each of a plurality of objects, the neural network determines a feature vector for each object dependent on all of the sensor data items;

[0011] • Generating a training data set with a plurality of training data elements, wherein each training data element comprises a first set of sensor data relating to a first state of an environment with a set of multiple objects and a second set of sensor data relating to a second state of the environment, wherein in the second state the positions of the objects are at least partially changed compared to the first state;

[0012] • For each training data item, o determining features of the objects by feeding the first set of sensor data to the neural network and o determining features of the objects by feeding the second set of sensor data to the neural network;

[0013] • Determining a loss that, for each training data item, penalizes, for each object, a deviation between the feature determined by the neural network for the object from the first set of sensor data and the feature determined by the neural network for the object from the second set of sensor data, and penalizes a lack of deviation between the feature determined by the neural network for the object from the first set of sensor data and, for each other of the objects, the feature determined by the neural network for the other object from the second set of sensor data; and

[0014] • Training the neural network to reduce the loss.

[0015] The neural network trained in this way can then be used to extract (re-ID) features (i.e., as a feature extractor), which are then employed in object tracking. The approach described above (e.g., in the embodiments described below) enables improved association of measured values ​​with object tracks, which has a beneficial effect on the overall performance of object tracking. This means that objects can be tracked more accurately, fewer track breaks occur, and fewer objects are lost (missed).

[0016] According to the approach described above, a scene context (in particular other objects in the same scene) is used for training (and, if necessary, for extracting Re-ID features). This allows scene-specific features to be used to associate measured values ​​with object tracks. This allows Re-ID features to be extracted that are better suited to the task of object tracking, which in turn has a positive impact on the overall performance of object tracking. Since the extraction of Re-ID features is adapted to the specific requirements of object tracking, the methods can be implemented more computationally efficiently (e.g., using smaller neural networks). A network architecture from a class of network architectures is used that is suitable for extracting scene-specific Re-ID features.

[0017] Various examples of implementation are given below.

[0018] Embodiment 1 is a method for training a neural network to determine features of objects for object tracking as described above.

[0019] Embodiment 2 is the method according to embodiment 1, comprising configuring the neural network with a network architecture such that, when supplied with a set of sensor data containing a sensor data item for each of a plurality of objects, the neural network performs the feature vector via processing in multiple stages, wherein at least one stage generates feature components for each of the objects, max-pooling of the feature components over the objects is performed, and the feature components and the result of the max-pooling of the feature components are supplied to the following stage.

[0020] This ensures that the neural network determines a feature vector for each object based on all sensor data elements. This means that when determining a feature vector for an object, it takes the other objects into account, thus determining the feature vectors based on both a local context (sensor data elements of the individual object) and a global context of the objects (result of max-pooling across the objects). Alternatively, a transformer network can be used.

[0021] Embodiment 3 is the method according to embodiment 1 or 2, wherein each training data element contains information about which object in the first state corresponds to which object in the second state, based on the determined features of the objects it is determined which objects in the first state correspond to which objects in the second state and the loss is determined by comparing the information contained in the training data elements about which object in the first state corresponds to which object in the second state with the result of the determination based on the determined features which objects in the first state correspond to which objects in the second state.

[0022] The ground truth can therefore be defined as the association of objects (e.g., in the form of an association matrix). This trains the neural network to select features that are particularly well suited for object tracking. The result of determining, based on the identified features, which objects in the first state correspond to which objects in the second state is, for example, a soft association matrix.

[0023] Embodiment 4 is a method for tracking objects, comprising

[0024] • Collecting sensor data of a first set of objects at a first time;

[0025] • Collecting sensor data from a second set of objects at a second time;

[0026] • Feeding the sensor data of the first set of objects to the neural network trained according to one of the embodiments 1 to 3 to generate first features;

[0027] • Feeding the sensor data of the second set of objects to the trained neural network to generate second features; and

[0028] • Pairwise associating objects of the first set of objects of the second set based on the first features and the second features. Embodiment 5 is the method according to embodiment 4, comprising acquiring sensor data from objects in a scene at the first time, grouping the objects into a plurality of first groups based on their spatial proximity, selecting one of the first groups as the first set of objects, acquiring sensor data from objects in the scene at the second time, grouping the objects into a plurality of second groups based on their spatial proximity, and selecting one of the second groups as the second set of objects, such that the second set of objects is the second group that is closest to the first set of objects within the scene.

[0029] Objects in scenes are thus treated as (spatial) clusters. This facilitates the differentiation of features based on features, since not all objects in a scene need to be distinguished, but only those within a group.

[0030] Embodiment 6 is a data processing device (e.g. a control device) which is configured to carry out a method according to one of the embodiments 1 to 5.

[0031] Embodiment 7 is a computer program having instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0032] Embodiment 8 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0033] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0034] Figure 1 shows a vehicle.

[0035] Figure 2 illustrates object tracking using Re-ID features. Figure 3 illustrates the association of tracked objects with currently detected objects.

[0036] Figure 4 illustrates feature extraction according to one embodiment.

[0037] Figure 5 illustrates a feature extraction according to a further embodiment.

[0038] Figure 6 shows a possible architecture of a neural network for Re-ID feature extraction.

[0039] Figure 7 shows an example in which pedestrian trajectories are depicted that can be used to train a Re-ID feature extractor.

[0040] Figure 8 shows the formation of sets of positive Re-ID feature pairs and negative Re-ID feature pairs, on the basis of which different losses (cost functions) can be calculated for training a Re-ID feature extractor.

[0041] Figure 9 illustrates the procedure for training a Re-ID feature extractor, where the loss is calculated based on a computed association between measured values ​​of consecutive measurement times.

[0042] Figure 10 shows a flowchart illustrating a method for training a neural network to determine features of objects for object tracking.

[0043] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects. Various examples are described in more detail below.

[0044] Figure 1 shows a (e.g. autonomous) vehicle 101.

[0045] In the example of Figure 1, the vehicle 101, for example a car or truck, is provided with a vehicle control device 102.

[0046] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103.

[0047] For example, the stored control software (computer program) includes instructions that, when executed by the processor, cause the processor 103 to implement one or more neural networks 107.

[0048] The data stored in memory 104 may include, for example, image data captured by one or more cameras 105. The one or more cameras 105 may, for example, capture one or more grayscale or color photos of the surroundings of the vehicle 101.

[0049] The vehicle control device 102 can determine, based on the image data, whether and which objects, e.g. fixed objects such as traffic signs or road markings or moving objects such as pedestrians, animals and other vehicles, are present in the surroundings of the vehicle 101.

[0050] Image data is used here only as an example of sensor data and data from other sensors (ultrasound, LiDAR, etc.) can also be used.

[0051] The vehicle 101 can then be controlled by the vehicle control device 102 according to the results of the object detection. For example, the vehicle control device 102 can control an actuator 106 (e.g., a brake) to control the speed of the vehicle, e.g., to decelerate the vehicle. For this purpose, it is typically desirable not only to detect objects but also to track them, e.g., to predict their future position by observing how they have moved in the past and to gain a consistent understanding of the current (traffic) situation.

[0052] The tracking of objects, for example, is implemented using recursive filters (e.g., Kalman filters), which perform three processing steps: prediction, association, and update. During prediction, existing objects are first predicted to the current measurement time using motion models. Subsequently, current measurements (e.g., object detection in camera images) are associated with the predicted objects. Finally, the predicted objects are corrected using the associated measurements in an update step. The association of measurements to object tracks is particularly important, since erroneous associations lead to inaccurate object tracks or even the loss of object tracks. The basis for the association are distance metrics, which map the proximity of measurements and tracked objects in a feature space. Used metrics include, for example,Euclidean distance or Mahalanobis distance. Typically, geometric features such as object position, object size, or object speed are used to calculate the distance. Based on the calculated distances, measurements are then assigned to tracked objects using an association algorithm. Examples of such algorithms are the Hungarian algorithm or various versions of the nearest neighbor algorithm. Due to the significant influence of geometric features, problems mainly arise when measurements and objects cannot be clearly associated with one another in these dimensions (e.g., closely spaced groups of pedestrians, inaccurate predictions due to prolonged occlusions, etc.). These problems during the association process can be mitigated or completely prevented by adding suitable features.Features are considered suitable if they support a clear association of measurements with objects. These can be, for example, view-based features that describe the appearance of the objects (example: If only one pedestrian in a group of pedestrians is wearing a red jacket, then the red jacket is a good feature for tracking this pedestrian). The extraction of so-called re-identification features (re-ID features for short) is used to support object tracking, for example, in surveillance systems for train stations, airports, or pedestrian zones, to enable the recognition of people in different cameras, even over longer periods of time.

[0053] Figure 2 illustrates object tracking using Re-ID features.

[0054] For this purpose, measured values ​​203 (which are determined from sensor data, e.g., camera data 201, by preprocessing 202) are enriched with Re-ID features obtained from feature extraction 206 before association 204 with tracked objects 205 to form extended measured values ​​207, so that the subsequent association 204 (and in particular the distance calculation) can be performed both on the basis of the geometric features and on the basis of the Re-ID features. Currently detected objects (corresponding to measured values ​​203) are associated with the prediction 208 of tracked objects 205 (and subsequently an update 209 of the tracked objects 205 is performed). This provides tracking information 210 (e.g., in the form of trajectories) about the detected objects as output.

[0055] Figure 3 illustrates the association of tracked objects 301 with currently detected (“measured”) objects 302 (from measured values ​​305, e.g., detections as object bounding boxes) via a distance calculation 303 using an association algorithm 304. The distance calculation can be based on the typically used geometric features (position, size, speed, etc.) and / or re-ID features.

[0056] The re-ID features are typically extracted using deep neural networks (e.g., one of the neural networks 107). The training of such networks is based on large annotated datasets containing numerous objects (e.g., people) with a multitude of views per object. The goal of training is to find suitable weights for the neural network so that a feature space suitable for re-identifying the objects is created (into which the neural network maps inputs (e.g., views of objects)). This means that different views of the same object (e.g., the same person) should be mapped to features (feature vectors) that have a short distance in the feature space, whereas the distances to feature vectors to which views of other objects are mapped should be maximized.

[0057] A feature map (feature extraction) trained in this way is suitable for recognizing objects from a large database and can therefore also be helpful for association during tracking.

[0058] In detail, Re-ID features for use during object tracking should have the following properties:

[0059] • Scene-specific features: The features should be suitable for distinguishing objects from other objects within the same scene. For this to be possible, it is not necessary for an object to be distinguishable from all objects (including those in other scenes). Typical methods for extracting Re-ID features, however, only achieve the latter. With reference to the previous example: If a person is the only one in a scene wearing a red jacket, the red jacket is a good feature for distinguishing the person from other people in the scene. However, a red jacket is certainly not a good feature for distinguishing a person from all other people in the world. This means that different features than those suitable for classic Re-ID applications may be suitable for tracking applications. Furthermore, the features for one and the same person can differ depending on the scene.Depending on the context of the scene, it may be the red jacket in one case, and perhaps a little later, the umbrella being carried (e.g. if a second person wearing a red jacket enters the scene).

[0060] • Constancy over a short period of time: In object tracking, objects only need to be recognized over a short time horizon. Typically only from one measurement time to the next (a few milliseconds). In the case of occlusions, it may be necessary to recognize objects within a few seconds. Thus, it is only necessary that the re-ID feature for an object is almost constant within this short period of time, but not over minutes, hours, or days. For example, a person only needs to be recognized within a scene and differentiated from other people. If the same person reappears in another scene, it is not necessary to know that it is the same person. With regard to the above desired properties of a feature extraction (ieAccording to various embodiments, an approach is provided that enables improved object tracking by improving the association of measured values ​​(e.g., object detections) with object tracks (i.e., tracked objects), thus enabling improved object tracking. An improvement in object tracking is reflected in more accurate object trajectories, fewer track breaks, and fewer object losses.

[0061] To improve association during object tracking, Re-ID features are used (e.g., as described with reference to Figures 1 to 3). However, the training and extraction of the Re-ID features are tailored to the specific requirements for feature extraction (as described above). In particular, according to various embodiments, a method for extracting Re-ID features during tracking, specific network architectures for Re-ID feature extractors, and methods for training suitable Re-ID feature extractors are provided.

[0062] Common methods for extracting Re-ID features train corresponding deep neural networks on large annotated datasets with the goal of distinguishing different views of an object (e.g., a person) from views of other objects (e.g., all other people). A fully trained neural network is then applied to each detected object individually to generate Re-ID features for that object. For example, if five people are detected in a scene, the network is executed five times independently (once for each detection).

[0063] In contrast, according to various embodiments, an approach is provided as illustrated in Figure 4.

[0064] Figure 4 illustrates feature extraction according to one embodiment.

[0065] Here, a re-ID feature extractor 401 (neural network) is used, which uses a plurality of measured values ​​(e.g., detections) 402 as input and generates corresponding re-ID feature vectors 403 as output for the plurality of measured values ​​402. This approach enables the feature extractor 401 to extract scene-specific features—i.e., features that are particularly well suited for distinguishing or recognizing the measured values ​​of the scene (in the example, for distinguishing the five people).

[0066] This implies that the scene context plays a major role in feature extraction. Referring to the red jacket example above: If only one person in the scene is wearing a red jacket, the red jacket will be a good feature for recognition. If there are multiple people wearing red jackets in the scene, the feature extractor will use other features to distinguish or distinguish between the two.

[0067] Use recognition of people (e.g. hair color, umbrella worn, etc.).

[0068] Figure 5 illustrates feature extraction according to another embodiment.

[0069] According to the embodiment of Figure 5, measured values ​​of a scene 501 are first grouped into clusters 502, 503 (whispered) and then feature extraction 504, 505 takes place for each cluster 502, 503. This has the advantage that the Re-ID feature extraction can be concentrated on the measured values ​​for which there is a high risk of false association (e.g., closely spaced groups of pedestrians). With reference to the above example of the red jacket: Even if there are two people wearing red jackets in a scene, a red jacket can be a suitable feature for distinguishing one person from the other (and thus recognition), e.g., if the two people wearing red jackets are far apart from each other (e.g., on different sides of the road) and thus belong to different clusters 502, 503.

[0070] According to a further embodiment, the Re-ID feature extraction takes into account a weighting of the measured values. In detail, the weights should reflect how strongly individual measured values ​​should be distinguishable from other measured values. The Re-ID feature extraction can then concentrate on extracting suitable distinguishing features. Weights can, for example, be specified in pairs (between all measured values), so that each weight indicates how important it is to distinguish between the two measured values. The weights reflect, for example, the separability based on the geometric features (such as position, extent, velocity), i.e., poorly separable measured values ​​receive a high weight. The approach described with reference to Figure 5 can be seen (or implemented) as a special case of such weighting: pairwise weights are 1 for all pairs of measured values ​​of the same cluster, 0 otherwise.

[0071] According to various embodiments, an architecture is used for a neural network for Re-ID feature extraction (to be applied in the manner mentioned above) that satisfies the following properties:

[0072] • The input can contain any number N of measured values ​​in the form of N input data vectors (which can also be considered input feature vectors). Measured values ​​can, for example, be pedestrian detections, whose input vectors contain, for example, associated raw sensor data (image pixels, LiDAR point clouds, radar reflections, etc.) or quantities derived from them.

[0073] • The N measured values ​​can be entered in any order, ie the calculation of the Re-ID features is independent of the order.

[0074] • The output of the network is N Re-ID feature vectors, where N corresponds to the number of input measurements.

[0075] Figure 6 shows a possible architecture of a neural network for Re-ID feature extraction.

[0076] The input of the network is the measured values ​​601 (e.g., pedestrian detections). The measured values ​​are unordered (because the order should be irrelevant for the extracted features). Each measured value is characterized by certain properties in the form of an input data vector (e.g., raw sensor data for the detections such as image pixels, LiDAR points, radar reflections, or data derived therefrom). The output 605 of the network corresponds to the calculated Re-ID feature vectors (i.e., one feature vector for each measured value of the input 601).

[0077] The first layer 602 processes the data of each measured value independently of one another using a fully meshed layer (ID-conv) for each measured value. This layer is characterized by a weight matrix M and a bias vector b. To get from the input representation (measured value data) to the first hidden layer, the same weight matrix M and the same bias vector b are used for each measured value (so-called weight sharing). The output of such ID-conv layers is a new representation of the measured value data (measured value-specific local feature vectors). An ID-conv layer thus operates locally, in which only the features within a measured value are combined (local context). In the example, an ID-conv layer extracts individual features for each pedestrian (e.g., regarding their shape, color, etc.).

[0078] This is followed by a pooling layer 603, which reduces the measured-value features to a single (so-called global) feature. In the example shown, the maximum of the features of the first hidden layer is calculated (global max pooling). Such a pooling layer aggregates information across multiple measured values, allowing dependencies between the measured values ​​(global context) to be mapped. This means that the pooling layer 603 allows the features previously extracted individually for each pedestrian to be combined and related to one another.

[0079] To utilize this global relationship between the features in the subsequent calculation, the global feature vector can be appended to the measured-value-specific local feature vectors. Subsequent ID-conv layers 604 (similar to the first layer 602, possibly together with a respective max-pooling layer 603) can therefore also access the global context during their measured-value feature extraction. In the example: Features that distinguish individual pedestrians can be highlighted.

[0080] As shown in Figure 6, it is possible to string together several ID-conv + Pooling + Append processing blocks to ultimately extract complex Re-ID feature vectors 605.

[0081] The architecture shown in Figure 6 is a concrete implementation that can be modified in many ways. A generalized basic architecture can be described as follows: • The input is an unordered list of measured values. The output of the network architecture is a list of Re-ID feature vectors (one output vector per input measured value).

[0082] • The architecture uses any number of layers to perform a measurement-based extraction of (local) features. In one embodiment, weight sharing is implemented in the measurement-based feature extraction.

[0083] • The architecture includes at least one global pooling layer, which combines the local features extracted on a measured-value basis into a global feature vector. Furthermore, additional pooling operators can be used after the various measured-value feature extraction layers. In one embodiment, the pooling layers use the max-pooling operator. However, other pooling operators (e.g., average pooling) are also conceivable.

[0084] • The architecture uses at least one concatenation module (append), which combines global (pooled) feature vectors with the local (measurement-wise) feature vectors.

[0085] Various extensions are possible for the basic architecture described, such as:

[0086] • Global feature vector: The pooled feature vector can be further processed before concatenation to the local feature vectors, e.g. using fully meshed layers.

[0087] • Order: (Local or global) features from earlier layers can also be used directly in later layers, e.g. with the help of so-called skip connections.

[0088] The training of the Re-ID feature extractor is based on data sets in which object trajectories are annotated. This means that objects are annotated at individual measurement times and an association exists between the measurement times (e.g., using object IDs).

[0089] Figure 7 shows an example of pedestrian trajectories 701 to 705 that can be used to train a Re-ID feature extractor. Pedestrians marked with a "+" refer to the same sensor measurement time. In addition, pedestrian annotations at earlier and later measurement times are shown.

[0090] Figure 8 shows the formation of sets of positive Re-ID feature pairs 801 and negative Re-ID feature pairs 802, on the basis of which various losses (cost functions) for training a Re-ID feature extractor 803 can be calculated.

[0091] As shown in Figure 8, for training according to various embodiments, it is provided that the Re-ID feature extraction is applied to at least two consecutive measurement times and the set of positive pairs 801 and the set of negative pairs 802 are formed from the resulting Re-ID feature vectors.

[0092] Positive pairs correspond to two feature vectors that describe the same object (e.g., a pedestrian) at different measurement times. Negative pairs correspond to two feature vectors that describe different objects (e.g., two different pedestrians) at the same or different measurement times. Whether feature vectors describe different or the same object is known for the training data of the training dataset (through appropriate annotation of the input data vectors contained in the training dataset).

[0093] When this approach is applied to a large number of short sequences (i.e., consecutive measurement times) of the annotated training dataset, large amounts of positive and negative pairs can be generated.

[0094] During training, the weights of the neural network are adjusted such that the distance between the feature vectors of positive pairs is reduced (ideally minimized), while the distance between the feature vectors of negative pairs is increased (ideally maximized). This can be achieved, for example, using backpropagation of a suitable loss (cost function) and gradient descent. For example, a contrastive loss, triplet loss, multiclass N-pair loss, or constellation loss is used as the cost function.

[0095] The training of the neural network is carried out as follows:

[0096] 1) Generation of multiple sequences consisting of two or more measurement points

[0097] 2) Calculation of the Re-ID feature vectors by applying the neural network

[0098] 3) Generation of positive and negative pairs of the calculated Re-ID feature vectors

[0099] 4) Calculating the loss

[0100] 5) Adaptation of the network weights by backpropagation of the loss and gradient descent.

[0101] The above training steps 1) to 5) can be repeated as often as desired.

[0102] According to a further embodiment, it is provided that the weights of the neural network are adapted on the basis of a loss which reflects the quality of the association between measured values ​​of two consecutive time steps.

[0103] Figure 9 illustrates the procedure for training a Re-ID feature extractor, where the loss is calculated based on a computed association between measured values ​​of consecutive measurement times.

[0104] For this purpose, the Re-ID feature vectors 901, 902 of the measured values ​​are first extracted for two consecutive time steps, then a distance matrix 903 is calculated and, on this basis, a soft association matrix 905 is calculated by a differentiable association module 904 (e.g., Deep Hungarian Network). “Soft” here means that its entries (association values) approximate the binary entries (i.e., 0 or 1) of a real association matrix. Finally, by comparing the soft association matrix with the real association matrix (which is known from the labels, i.e., annotations, of the training data set), a loss 907 can be calculated, on the basis of which the network weights can be adjusted by backpropagation and gradient descent 908. It should be noted that this loss also penalizes assigning different features to the same object.different objects are assigned different characteristics, since the soft association matrix then has small association values ​​for the same objects or high association values ​​for different objects.

[0105] In summary, according to various embodiments, a method is provided as shown in Figure 10.

[0106] Figure 10 shows a flowchart 1000 illustrating a method for training a neural network to determine features of objects for object tracking.

[0107] In 1001, a neural network is configured with a network architecture (ie, a neural network is provided with such an architecture) such that, when supplied with a set of sensor data containing one sensor data item for each of a plurality of objects, it determines a feature vector for each object depending on all of the sensor data items.

[0108] In 1002, a training data set is generated with a plurality of training data elements, wherein each training data element comprises a first set of sensor data relating to a first state of an environment having a set of multiple objects and a second set of sensor data relating to a second state of the environment, wherein in the second state the positions of the objects are at least partially changed compared to the first state.

[0109] In 1003, for each training data item, features of the objects are determined by feeding the first set of sensor data to the neural network, and features of the objects are determined by feeding the second set of sensor data to the neural network.

[0110] In 1004, a loss is determined that, for each training data item, for each object, penalizes a deviation between the feature determined by the neural network for the object from the first set of sensor data and the feature determined by the neural network for the object from the second set of sensor data, and penalizes a lack of deviation between the feature determined by the neural network for the object from the first set of sensor data and, for each other of the objects, the feature determined by the neural network for the other object from the second set of sensor data.

[0111] In 1005, the neural network is trained to reduce the loss.

[0112] Here, 1003, 1004 and 1005 can take place repeatedly in alternation, for example, the loss is determined for batches of training data elements (ie the training data set can, for example, have batches each with a plurality of training data elements (of the described form), for each of which a loss is generated and the neural network is trained for each batch to reduce the respective loss).

[0113] The method of Figure 10 may be performed by one or more computers having one or more data processing units. The term "data processing unit" may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be processed according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other manner of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be executed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0114] According to various embodiments, the method is thus particularly computer-implemented. After training, the neural network can be applied to sensor data obtained from at least one sensor to obtain features that are then used for object tracking. The result of the object tracking can then be used to control a robotic device.

[0115] For example, after training, the neural network is used to generate a control signal for a robotic device by feeding it with sensor data regarding the robotic device and / or its environment. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0116] Various embodiments may receive and use sensor data from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc.

[0117] Although specific embodiments have been shown and described herein, it will be recognized by those skilled in the art that the specific embodiments shown and described may be substituted for a variety of alternative and / or equivalent implementations without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.

Claims

Claims 1. A method for training a neural network to determine features of objects for object tracking, comprising: configuring a neural network with a network architecture such that, when supplied with a set of sensor data containing a sensor data element for each of a plurality of objects, it determines a feature vector for each object depending on all of the sensor data elements; generating a training data set with a plurality of training data elements, wherein each training data element comprises a first set of sensor data relating to a first state of an environment comprising a set of a plurality of objects and a second set of sensor data relating to a second state of the environment, wherein in the second state the positions of the objects are at least partially changed compared to the first state; For each training data item, Determining features of the objects by feeding the first set of sensor data to the neural network and Determining features of the objects by feeding the second set of sensor data to the neural network; Determining a loss that, for each training data item, penalizes, for each object, a deviation between the feature determined by the neural network for the object from the first set of sensor data and the feature determined by the neural network for the object from the second set of sensor data, and penalizes a lack of deviation between the feature determined by the neural network for the object from the first set of sensor data and, for each other of the objects, the feature determined by the neural network for the other object from the second set of sensor data; and training the neural network to reduce the loss. The method of claim 1, comprising configuring the neural network with a network architecture such that, when supplied with a set of sensor data containing a sensor data item for each of a plurality of objects, the neural network traverses the feature vector via multi-stage processing, wherein at least one stage generates feature components for each of the objects, max-pools the feature components across the objects, and supplies the feature components and the result of the max-pooling of the feature components to the following stage.Method according to claim 1 or 2, wherein each training data element contains information about which object in the first state corresponds to which object in the second state, based on the determined features of the objects it is determined which objects in the first state correspond to which objects in the second state, and the loss is determined by comparing the information contained in the training data elements about which object in the first state corresponds to which object in the second state with the result of the determination based on the determined features which objects in the first state correspond to which objects in the second state. Method for tracking objects, comprising:. Collecting sensor data of a first set of objects at a first time; Collecting sensor data from a second set of objects at a second time; Feeding the sensor data of the first set of objects to the neural network trained according to one of claims 1 to 3 to generate first features; Feeding the sensor data of the second set of objects to the trained neural network to generate second features; and Pairwise associating objects of the first set with objects of the second set based on the first features and the second features. The method of claim 4, comprising acquiring sensor data from objects in a scene at the first time, grouping the objects into a plurality of first groups based on their spatial proximity, selecting one of the first groups as the first set of objects, acquiring sensor data from objects in the scene at the second time, grouping the objects into a plurality of second groups based on their spatial proximity, and selecting one of the second groups as the second set of objects, such that the second set of objects is the second group closest to the first set of objects within the scene. A data processing device configured to perform a method according to any one of claims 1 to 5. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5.A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5.

Images