METHOD FOR PROVIDING AN OBJECT TRACKING FUNCTION

Description

[0001] The invention relates to a method for providing an object tracking function. Furthermore, the invention relates to a data processing device, a computer program, and a computer-readable storage medium.

[0002] In the field of augmented reality, a camera pose relative to an object is determined and displayed based on captured camera images using object tracking functions. Based on this determined camera pose, additional information can be overlaid on the object in the camera images. However, the implementation of these object tracking functions is currently unsatisfactory.

[0003] From Jason Rambach et al., Learning 6DoF Object Poses from Synthetic Single Channel Images, 2018 IEEE INTERNATIONAL SYMPOSIUM ON MIXED AND AUGMENTED REALITY ADJUNCT (ISMAR-ADJUNCT), IEEE, 16.10.2018, doi:10.1109 / ISMAR-ADJUNCT.2018.00058, pages 164 - 169, a method for estimating an object pose is known in which the method is trained using synthetically generated images.

[0004] From Alex Kendall and Roberto Cipolla, Modelling uncertainty in deep learning for camera relocalization, arXiv:1509.06825 [cs.LG], 18.02.2016, doi:10.1109 / ICRA.2016.7487679, ISBN 978-1-4673-8026-3, pages 4762 - 4769, a method for estimating uncertainty in deep learning for camera localization is known.

[0005] From Yi Li et al., DeeplM: Deep Iterative Matching for 6D Pose Estimation, Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO), arXiv:1804.00175v3 [cs.CV], May 14, 2019, https: / / doi.org / 10.1007 / s11263-019-01250-9 ,A method for iteratively adjusting a six-dimensional pose estimation is known, in which a rendered image is compared with a captured image.

[0006] From Karim Koreitem et al., Synthetically Trained 3D Visual Tracker of Underwater Vehicles, OCEANS 2018 MTS / IEEE CHARLESTON, IEEE, 22.10.2018, doi:10.1109 / OCEANS.2018.8604597, pages 1 - 7, a method for visually tracking an underwater vehicle is known that was trained with synthetically generated images.

[0007] Chris Laan, "Real-time 3D car pose estimation trained on synthetic data," March 28, 2019, https: / / web.archive.org / web / 20200216214053 / https: / / labs.laan.com / blog / real-time-3d-car-pose-estimation-trained-on-synthetic-data.html, describes a method for training using synthetically generated training data. Laanlabs, "Real-time 3D car pose estimation on phone," March 28, 2019, https: / / www.youtube.com / watch?v=I-Q5hgl-d64 ,A related demonstration video is known.

[0008] From Su Hao et al., Reader for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views, 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, December 7, 2015, pages 2686-2694, DOI: 10.1109 / ICCV.2015.308, a method for estimating a viewpoint using a convolutional mesh trained with rendered 3D model views is known.

[0009] From Artem Rozantsev, On rendering synthetic images for training an object detector, Computer Vision and Image Understanding, Academic Press, USA, Vol. 137, January 20, 2015, pages 24-37, ISSN: 1077-3142, DOI: 10.1016 / J.CVIU.2014.12.006, a method for training an object detector using synthetic images is known.

[0010] The invention is based on the objective of improving a method for providing an object tracking function.

[0011] The problem is solved according to the invention by a method with the features of claim 1. Furthermore, the problem is solved according to the invention by a device with the features of claim 7, a computer program with the features of claim 8, and a computer-readable storage medium with the features of claim 9. Advantageous embodiments of the invention are set forth in the dependent claims.

[0012] In particular, a method for providing an object tracking function is provided, wherein the object tracking function provides a sensor pose relative to an object to be tracked based on acquired sensor data from a sensor, wherein the object tracking function is provided by means of an object-specific trained deep neural network, and wherein the object-specific trained deep neural network is trained using simulated sensor data.

[0013] Furthermore, in particular a device for data processing is created, comprising means for carrying out the process steps of the procedure.

[0014] Furthermore, a computer program is created, in particular comprising instructions which, when the computer program is executed by a computer, cause it to perform the procedural steps of the procedure.

[0015] In addition, a computer-readable storage medium is created, comprising instructions which, when executed by a computer, cause it to perform the procedural steps of the process.

[0016] The method and device enable the training of the deep neural network used to provide the object tracking function without a training dataset based on acquired sensor data. For this purpose, simulated sensor data, also known as synthetic sensor data, is used. This simulated sensor data is generated based on a three-dimensional object model, such as CAD data. Starting from this object model, the simulated sensor data is generated, striving for the greatest possible realism. The training dataset comprises a multitude of simulated sensor data, each generated for a variety of different sensor poses.After the training phase, the trained neural network can, in an inference phase, estimate a sensor pose relative to the object being tracked based on acquired sensor data and provide this information, particularly in the form of sensor pose data. The sensor pose encompasses both a position (e.g., in Cartesian coordinates x, y, and z) and an orientation (e.g., yaw, pitch, and roll angles) of the sensor relative to the object being tracked, resulting in a total of six degrees of freedom. In principle, the use of a quaternion would also be possible, in which case there would be seven degrees of freedom.

[0017] The advantage of the invention is that it eliminates the need for extensive acquisition and compilation of sensor data to train the deep neural network. This saves effort, time, and costs.

[0018] A further advantage of the invention is that, after training the deep neural network, an object tracking function can be provided without requiring detailed object data, such as construction details, to be known or provided. The trained deep neural network is ready for use without further input parameters and therefore also enables its use in security-relevant applications where an object model, for example in the form of detailed CAD data, cannot or should not be provided for reasons of confidentiality.

[0019] In principle, it is possible to use both simulated and recorded sensor data. This recorded data is then added to the simulated sensor data.

[0020] Based on the provided sensor pose, augmented reality content overlaid with the sensor data can then be displayed. Furthermore, control data for controlling actuators could also be derived.

[0021] A deep neural network is in particular an artificial neural network, especially a convolutional neural network.

[0022] The deep neural network is trained and deployed on an object-specific basis; that is, for each object or class of objects to be tracked, a deep neural network specifically trained for that purpose is provided and used. This has the advantage that the deep neural network is tailored to a single object, thus ensuring high accuracy in the object tracking function.

[0023] The sensor is, in particular, a camera, specifically a camera operating in the visible wavelength range. The sensor data consists, in particular, of camera images, specifically RGB camera images of the object being tracked. The camera can also be a depth camera (RGB-D), which additionally provides depth information.

[0024] Sensor data can, in principle, be one-dimensional or multi-dimensional. In particular, sensor data consists of two-dimensional camera images captured and provided by a camera.

[0025] The method is implemented in particular as a computer-implemented invention.

[0026] In particular, the process is executed using a computing device that can access memory. The computing device can be a combination of hardware and software, for example, program code that runs on a microcontroller or microprocessor.

[0027] The provided object tracking function, using a trained deep neural network, can be run on a relatively small computer. In particular, the application can be used without access to the CAD data of the object being tracked, meaning that such data does not need to be provided and can therefore remain confidential.

[0028] In one embodiment, sensor poses relative to the object being tracked, for which the simulated sensor data is generated, are randomly selected. This reduces the number of sensor poses while still ensuring that, on average, all sensor poses are covered by the training data. This saves computing power and therefore costs and effort. To randomly select the sensor poses, they can be chosen such that the mean sensor pose calculated by averaging all sensor poses is zero, thus ensuring a uniform distribution despite the random selection.

[0029] In one embodiment, at least one property of the object being tracked is modified to generate the simulated sensor data. Such a property could be, for example, a surface property, such as the color or reflectivity of the object's surface. This allows for the creation of object variants based on the object model, thereby increasing the variance of the object within the training data for the deep neural network. Furthermore, variations in geometry can also be taken into account, such as tolerances, errors, and / or defects that occur during the object's manufacturing process.

[0030] In a further embodiment, at least one sensor property is taken into account when generating the simulated sensor data. These properties can be, in particular, characteristics in the form of induced sensor artifacts. If the sensor is a camera, these properties can be, for example, noise, blur, white balance, and / or camera sensitivity. To generate the simulated sensor data, these parameters are, for example, changed stepwise in simulated camera images.

[0031] In one embodiment, at least one environmental property is taken into account when generating the simulated sensor data. Such an environmental property could, for example, be a background scene in front of which the object to be tracked is located. To make the deep neural network independent of the specific background scene, a multitude of different background scenes are simulated in the simulated sensor data. In the case of simulated camera images, for example, a background image can be chosen randomly without altering the object to be tracked, which is depicted in an object pose. However, it may also be possible to additionally consider the depicted object or...To modify a simulated camera image of the object being tracked within the object's area, depending on a selected background scene, for example by mapping or simulating a surface reflection from part of the background scene onto the object. Other environmental properties can include, for example, occlusion of the object or lighting conditions (e.g., light sources, cast shadows, lens flares, etc.).

[0032] In particular, domain randomization can be performed on the simulated sensor data to increase the amount of simulated sensor data. This represents another way to make the simulated sensor data more realistic.

[0033] In one embodiment, the simulated sensor data is generated at least partially by means of photorealistic rendering. This allows for a particularly realistic simulation of the sensor data, utilizing techniques already known, such as those used in filmmaking or computer games.

[0034] It is intended that a loss function (or error function or target value function) is used when training the deep neural network, taking into account the viewing distance of the sensor to the object being tracked. The loss function incorporates an error determined based on the difference between a sensor pose used in the simulation and a sensor pose estimated by the deep neural network. This difference can be normalized based on the viewing distance between the sensor and the object, or may already be normalized. This makes the error dimensionless, meaning it is independent of the actual object size. For example, a large error at a specific camera position might result in a negligible error when the captured camera image is superimposed on a virtual image in an augmented reality environment.

[0035] To derive the error, it can also be implemented that three-dimensional points of the object in both the simulated and the estimated sensor pose are projected onto a two-dimensional surface. The error is then determined via correspondences between pairwise corresponding points on the two-dimensional surface. This is advantageous in an augmented reality application scenario, as the deep neural network can be optimized specifically for a two-dimensional overlay.

[0036] To determine a lot value for the lot function (i.e., an error value), at least three comparison points are required. These comparison points can be chosen randomly or derived from the object's CAD data, for example, via the object's vertices. The following steps, for instance, can be performed to determine the lot value. 1. Two sensor positions (e.g., camera positions) are defined: one based on the sensor pose used in the simulation, and the other based on the sensor pose estimated by the deep neural network. 2. Two objects corresponding to the sensors (or cameras) are placed at the origin, each with relevant points. 3. The sensors (or cameras) are moved to the origin, with each object being shifted by the corresponding offset. 4. The sensors (or cameras) are rotated to an orientation with angles (0, 0, 0), and the corresponding object is rotated around the origin accordingly. The sensors (or cameras) are then aligned. 5. At least three points are determined for each of the two objects, which correspond to each other in pairs.An average deviation of at least three points from each other determines an error and defines the loss value of the loss function (however, if a correspondence on a two-dimensional surface is evaluated as described above, at least two points are sufficient).

[0037] In one embodiment, neurons are randomly deactivated during an inference phase of the deep neural network. Specifically, each subsequent inference iteration involves a further random selection of deactivated neurons, so that the deep neural network exhibits a partially modified structure or connection each time it is inferred. For this purpose, a layer is reserved in the deep neural network in which individual neurons are randomly deactivated during the inference phase. The Gal dropout method can be used for this purpose. The multiple iterations with a partially modified structure enable statistical analysis of the results. For example, a variance can be determined as a measure of the uncertainty of the estimated sensor pose.

[0038] After performing an inference phase, it may be planned that the inference phase is repeated in order to continuously estimate and provide a current sensor pose based on currently acquired sensor data using the trained deep neural network.

[0039] The invention is explained in more detail below with reference to preferred embodiments and the figures. These show: Fig. 1 a schematic representation of an embodiment of the data processing device for providing an object tracking function; Fig. 2 a schematic data flow diagram to illustrate an embodiment of the method; Fig. 3 a schematic representation of an application example of the trained deep neural network; Fig. 4 a schematic representation of the setup of an embodiment of the deep neural network.

[0040] In Fig. 1 Figure 1 shows a schematic representation of an embodiment of the device 30 for carrying out the method. The device 30 comprises means 31 for carrying out the method. The means 31 are a computing unit 32 and a memory 33. To carry out the method steps, the computing unit 32 can access the memory 33 and perform arithmetic operations in it.

[0041] Memory 33 stores a deep neural network 1, in particular a convolutional neural network, and a training dataset 2. The computing unit 32 generates the training dataset 2, which comprises a multitude of simulated sensor data 8, in particular simulated camera images. The simulated sensor data 8 are generated by the computing unit 32 based on a three-dimensional object model 4 of an object to be tracked. The object model 4 includes, for example, CAD data of the object to be tracked.

[0042] Based on the training data set 2 or the simulated sensor data 8, the computing device 32 trains the deep neural network 1 during a training phase.

[0043] During a subsequent inference phase, sensor data 5 acquired from a sensor 20, e.g., a camera, is fed to the computing unit 32. The acquired sensor data 5 is applied to the deep neural network 1 by the computing unit 32, i.e., fed to an input layer of the deep neural network 1. At an output layer of the deep neural network 1, an estimated sensor pose 12 is provided as an inferred result of the deep neural network 1, which is output by the computing unit 32, for example, as a sensor pose signal 7.

[0044] It may be provided that the structure, parameters and weighting data of the trained deep neural network 1 are exported by the computing device 32 as a data package, so that the trained deep neural network 1 can be used, for example, in another device without having to be retrained beforehand.

[0045] In Fig. 2 A schematic data flow diagram illustrating one embodiment of the method is shown. In process section 100, simulated sensor data is generated and provided. Process section 100 comprises process steps 101 and 102.

[0046] In process step 101, labels are generated for a training dataset. This is done by compiling a selection of sensor poses 6. This can be done randomly, aiming for a uniform distribution of the sensor poses 6 around the object. A sensor pose 6 comprises a three-dimensional position and an orientation of the sensor relative to the object, defined by three angular values, resulting in a total of six degrees of freedom for the sensor pose. The object is located at a coordinate origin. The sensor is specifically a camera, so the sensor pose is a camera pose. The result of process step 101 is a list of sensor poses or labels.

[0047] The selection of random camera poses can be performed as follows, for example. An icosahedron is created around the object by placing icosahedral points. Points are then evenly distributed across the resulting triangles of the icosahedron, meaning they are spaced at equal intervals. From this set of points, points are selected that are equidistant from the origin (located within the object). Based on this set, a new set of points is generated, with each point's distance from the object set to one to three times the object's radius or one to three times its maximum dimensions. For each point in the resulting point set, at least one random sensor orientation or camera orientation is chosen. A set of sensor poses or camera poses is then available for simulation.

[0048] In a subsequent process step 102, simulated sensor data 8 are generated or simulated based on the selected sensor poses 6. If the sensor is a camera, this is done in particular by simulating or calculating a camera image for each camera pose. For this purpose, the camera image is simulated based on the camera pose and an object model of the object, for example by calculating a virtual three-dimensional representation of the object and then projecting it onto the camera plane based on the camera pose selected from the list.

[0049] It may be provided that the simulated camera images (simulated sensor data 8) are generated, at least partially, by means of photorealistic rendering. Random variations of the object may be used to account for variance that typically occurs during the manufacture of the real object.

[0050] The result of process step 102 is a stack of simulated sensor data 8 or camera images, each corresponding to a sensor pose 6.

[0051] To carry out a training phase, the corresponding markers or sensor poses 6 (camera poses) and the simulated sensor data 8 (simulated camera images) are fed into the deep neural network.

[0052] In training section 200, the deep neural network is trained. For this purpose, the simulated sensor data 8 or the camera images are first augmented in a process step 201 by a multitude of changes, that is, modified in each case to increase the data basis during training.

[0053] It may be provided that at least one property of the object being tracked is changed in the simulated sensor data 8 or in the camera images. In a camera image, for example, a color or a reflection property of a surface of the object being tracked can be changed.

[0054] Furthermore, it may be stipulated that at least one sensor property is taken into account. For example, in the case of a camera used as a sensor, sensitivity or white balance can be considered. Additionally, sensor noise or image noise can be taken into account. The aim here is to replicate the properties of the sensor or camera as realistically as possible, that is, also considering any variations in these properties.

[0055] Furthermore, it can be stipulated that at least one environmental property is taken into account. For example, sensor data caused by the environment, such as background scenery behind an object in a camera image, can be considered and modified. Thus, the environment surrounding the object or its associated sensor data can be altered. For instance, colors or the motif of a background scene in a camera image can be changed. In this way, the deep neural network can be trained to handle commonly occurring variations.

[0056] The result of process step 201 is a list of extended simulated sensor data 9 and an associated list of extended sensor poses 6, whereby no new sensor poses 6 have been added at this point, but rather a large number of mutually different sensor data 9 have been generated for each sensor pose 6 from the originally simulated sensor data 6.

[0057] If the sensor data 6 consists of camera images, the following processing operations can be performed in particular to provide the large number of differing sensor data 9: A change in object color, object reflectance degree and / or object reflectance content or variants, a color and / or brightness shift, a change in the background, e.g., a randomly selected image as a background, a random selection of an image area, a color and / or brightness shift, a rotation and / or mirroring of image content, noise and / or blurring, a change in lighting conditions (e.g., light sources, specular highlights, etc.), obscuring of randomly selected image areas, normalization, changes based on sensor properties: change in color, brightness, noise, distortion, focal length, pixel count, motion blur, saturation, contrast, and artifacts (e.g., vignetting, flare, chromatic aberration, etc.).

[0058] In a process step 202, the deep neural network 1 is then trained. During the training phase, weights of the deep neural network 1 are adjusted in a manner known per se, whereby in a process step 203 lot values ​​11 of a lot function are calculated on the basis of a comparison between a sensor pose 12 or camera pose estimated by the deep neural network 1 and the sensor pose 6 or camera pose used in the simulation.

[0059] In process step 203, it is provided that a loss function is used in which a viewing distance of the sensor to the object to be tracked is taken into account.

[0060] After the calculated lot value 11 converges, the training phase ends. Deep Neural Network 1 is then trained and can be applied to acquired sensor or camera data to estimate a camera pose. Deep Neural Network 1 is specifically trained on one object.

[0061] In Fig. 3 Figure 1 shows a schematic representation of an application of the trained deep neural network 1. In the embodiment shown, a sensor 20 is a camera 21, which provides sensor data 5 in the form of camera images 22.

[0062] The sensor data 5 and camera images 22 are processed according to the instructions in the Fig. 2 The trained deep neural network 1 is fed the data as input according to the methods shown. Based on the sensor data 5 and the camera images 22, the trained deep neural network 1 estimates a sensor pose 12 and a camera pose 23, respectively.

[0063] Based on the estimated sensor position 12 or camera position 23, a superposition 24 of captured sensor data 5 or a captured camera image 22 can then be performed with additional information 25 ("Augmented Reality"), which in the shown representation includes a three-dimensional object model of the vehicle superimposed on the vehicle.

[0064] In Fig. 4 Figure 1 shows a schematic representation of the structure of an embodiment of the deep neural network 1. Camera images 22, for example, are fed into the input layer 40. The deep neural network 1 comprises an input layer 40. A pre-trained feature extractor 41 can follow the input layer 40, which extracts features from the supplied sensor data, such as edges, colors, shapes, or generally object features in the captured camera images.

[0065] This is followed by a Gal dropout layer 42, in which neurons are randomly deactivated during an inference phase. Repeated application of this allows for a statistical analysis of the inferred results.

[0066] Dropouts can also occur during the training phase to counteract the effects of overadaptation of the deep neural network 1.

[0067] Following the Gal dropout layer 42 is a multilayer perceptron block 43, which, in addition to the feature extractor 41, is trained during the training phase by changing weights and / or thresholds in individual neurons. Results inferred by the deep neural network 1, i.e., the estimated sensor pose 12, are provided at an output layer 44. Reference symbol list

[0068] 1 Deep Neural Network 2 Training Dataset 4 Three-Dimensional Object Model 5 Acquired Sensor Data 6 Sensor Pose (Label) 7 Sensor Pose Signal 8 Simulated Sensor Data 9 Modified Sensor Data 11 Loss Value 12 Estimated Sensor Pose 20 Sensor 21 Camera 22 Camera Image 23 Camera Pose 24 Overlay 25 Additional Information (Object Model) 30 Device 31 Means 32 Computing Device 33 Memory 40 Input Layer 41 Feature Extractor 42 Gal Dropout Layer 43 Perceptron Block 44 Output Layer 100-102 Process Steps 200-203 Process Steps

Claims

1. Method for providing an object tracking function, wherein the object tracking function provides, on the basis of acquired sensor data (5) from a sensor (20), a sensor pose (12) relative to an object to be tracked, wherein the object tracking function is provided by means of an object-specifically trained deep neural network (1), and wherein the object-specifically trained deep neural network (1) is trained using simulated sensor data (8), wherein a loss function (11) is used in the training of the deep neural network (1), in which loss function an observation distance from the sensor (20) to the object to be tracked is taken into account, wherein an error is taken into account for the loss function (11), which error is determined on the basis of a difference between a sensor pose (12) used in the simulation and a sensor pose (12) estimated by means of the deep neural network, characterized in that the difference is normalized on the basis of the observation distance between the sensor (20) and the object.

2. Method according to claim 1, characterized in that sensor poses (6) relative to the object to be tracked, for which sensor poses the simulated sensor data (8) are generated, are selected randomly.

3. Method according to claim 1 or 2, characterized in that at least one property of the object to be tracked is changed to generate the simulated sensor data (8).

4. Method according to any of the preceding claims, characterized in that at least one property of the sensor (20) is taken into account in the generation of the simulated sensor data (8).

5. Method according to any of the preceding claims, characterized in that at least one environment property is taken into account in the generation of the simulated sensor data (8).

6. Method according to any of the preceding claims, characterized in that during an inference phase of the deep neural network, neurons (1) are randomly deactivated.

7. Device (30) for processing data, comprising means (31) for carrying out the method steps of the method according to any one of claims 1 to 6.

8. Computer program, comprising commands which, when the computer program is executed by a computer, cause said computer to carry out the method steps of the method according to any one of claims 1 to 6.

9. Computer-readable storage medium, comprising commands which, when executed by a computer, cause said computer to carry out the method steps of the method according to any one of claims 1 to 6.