Computer-implemented method for generating annotations for objects from image data

Abstract

The invention relates to a computer-implemented method for generating annotations for objects from image data, wherein the method comprises the following steps: - Receiving image data (S10a) comprising multiple images from a visual sensor; - Segmenting the images and identifying objects (S12a) in the segments; - Reconstructing the trajectories (S18) of the objects by comparing the position of the objects in successively recorded images; and - Generating an annotation for the object using the trajectory.

Description

[0001] The invention relates to a method for generating annotations for objects from image data. It therefore lies in the field of computer vision and automated image processing. State of the art

[0002] Determining the trajectory of objects from individual images captured by visual cameras is a complex task that presents several technical challenges. The core problem lies in the fact that a camera typically only provides two-dimensional image data, while a trajectory describes the movement of an object in three-dimensional space. Reconstructing three-dimensional motion from these two-dimensional projections of the world onto a camera image requires overcoming several technological hurdles.

[0003] The initial problem is that the camera image is a perspective projection, in which depth information is largely lost. While objects closer to the camera appear larger and farther away objects smaller, the image itself provides no explicit information about how far an object is from the camera. The lack of this depth information makes it extremely difficult to determine an object's actual position in space from a single image. One way to solve this problem is to use multiple cameras positioned in a stereo setup. By analyzing the discrepancy between the two camera images (stereo vision), the depth information can be reconstructed. However, this requires precise calibration of the cameras to minimize geometric distortions.Furthermore, the system must be able to reliably identify corresponding points in both images.

[0004] Even when depth information is available through stereo vision or other techniques, such as depth cameras, the next challenge is tracking the movement of objects across a series of images to determine their trajectory. This is where algorithms come into play, calculating motion vectors by detecting objects across multiple image sequences and comparing their positions in each image. However, this is not trivial, as the visual appearance of an object can vary significantly depending on the camera angle, lighting, occlusion by other objects, and the object's own movement. Object detection and tracking in image sequences are often performed using image processing and machine learning methods, such as optical flow algorithms, Kalman filters, or neural networks.

[0005] Optical flow, one of the most widely used methods for motion tracking, estimates the movement of an object in an image as a continuous shift in pixel intensity between successive frames. However, this method requires relatively small and slow motion, as optical flow becomes inaccurate with abrupt movements or significant changes in the image scene. Kalman filters or particle filters, on the other hand, offer a probabilistic approach by predicting an object's position over time and then correcting it using new image data. Both methods, however, struggle with complex scenarios, such as fast, unpredictable movements or ambiguities caused by occlusion.

[0006] Another key difficulty arises from the fact that cameras are passive sensors that only provide information when sufficient light is present. This means that in poorly lit environments or at night, image quality decreases significantly, which can severely impair the accuracy of object tracking. Additionally, the lack of texture on certain surfaces or homogeneous image backgrounds makes reliable object tracking difficult, as it becomes challenging to find enough visual features to track across successive images.

[0007] The invention is therefore based on the objective of proposing a method by which positions and trajectories for objects in image data can be determined automatically and without the intervention of experts with special domain knowledge.

[0008] The problem is solved by the subject matter of the independent claims. Disclosure of the invention

[0009] According to a first aspect of the invention, this problem is solved by a computer-implemented method for determining trajectories for objects from image data, the method comprising the following steps: - Receiving image data comprising multiple images from a visual sensor; - Segmenting the images and identifying objects within the segments; - Reconstructing the trajectories of the objects by comparing the position of the objects in successively taken images; and - Generating an annotation for the object using the trajectory.

[0010] Any suitable sensor capable of receiving and generating an image of visible and / or near-infrared light can be used as a visual sensor. In particular, CCD or CMOS sensors can be used, which, depending on the design, may be fitted with a lens. The sensors thus provide image data suitable for further electronic processing. This image data includes information about the sensor's surroundings, with each pixel, starting from the focus of the visual sensor, containing information about objects in one direction from the sensor's environment.

[0011] The image data can preferably originate as a stream consisting of several temporally sequential images, i.e., a video stream. This stream can, for example, be acquired by a sensor system and processed by a computing unit that communicates with the sensor system and executes the proposed procedure. It should be ensured that the processing of the image data does not take longer than the acquisition time. The readout speed, the data transmission rate, and the data processing speed should be coordinated to optimize the system's efficiency.

[0012] The image data always comprises a chronologically ordered sequence of images depicting a scene. This scene could be, for example, a traffic scene or a monitored production where errors or malfunctions are to be detected. Image data generated by a surveillance system can also be processed and analyzed using the proposed method.

[0013] To process the image data, each image is segmented. Image segmentation involves dividing the image into segments relevant to the overarching question and background areas that are not of interest. Examples of relevant segments include objects in a traffic situation, workpieces and / or machines and machine parts in a production line, or people in a monitored area.

[0014] Segmentation can be performed using established image analysis algorithms or modern machine learning algorithms. Among machine learning algorithms, convolutional neural networks (CNNs) are particularly noteworthy, as their structure makes them especially suitable for machine learning applications in image processing.

[0015] Each object detected in the image data is instantiated and thus separated from the rest of the image at the data processing level. Instantiation allows for better differentiation of the objects during the process, as the context of each object is defined.

[0016] If the proposed method is used, for example, to track vehicles in the vicinity of the sensor system, the detected vehicles can be instantiated. This allows the vehicles to be tracked individually, which improves the accuracy of determining the trajectory and thus the prediction of their positions.

[0017] Over time, the instantiated objects will change their position relative to the position of the sensor system, either because the objects themselves are moving, because the sensor system is moving, or a combination of both. Comparing the positions of an object in two or more images reveals a four-dimensional relationship, adding a temporal dimension to the three spatial dimensions. From this four-dimensional relationship, a trajectory can be determined for each object in the image.

[0018] An important aspect of this is the four-dimensional reconstruction of the sensor's environment, from which the object's annotations can be derived. This is particularly relevant when the annotations are three-dimensional frames placed around the objects in four-dimensional space. These frames can be used in combination with empirical data to represent the objects' positions in four-dimensional space.

[0019] Methods such as Nerfs, Gaussian Splatting or Gaussian Ray Tracing can be used in particular for the four-dimensional reconstruction of the sensor's environment.

[0020] The trajectories of objects can be used to predict their positions, which in turn can play a role in other functions. For example, driver assistance systems can use this prediction to generate appropriate control commands for an autonomous or semi-autonomous vehicle, or a monitoring system can trigger an alarm and containment measures upon detecting a fault. The measures taken depend on the intended use and circumstances of the method, but do not significantly influence its execution. Intervention by a skilled person is not necessary to carry out the proposed method, so the invention fulfills its objective.

[0021] In one embodiment, the determined trajectories of the objects in one image are used to reconstruct the trajectories of the objects in subsequent images, provided that the objects are still visible in the subsequent images.

[0022] By using sequentially acquired images, the single-image approach of conventional image analysis is replaced by a four-dimensional approach for reconstructing the environment. Segmentation of objects in the image data is used to obtain information about the detected objects.

[0023] In one embodiment, the method is represented by a process pipeline with several modules, wherein the process pipeline includes a module for reconstructing four-dimensional trajectories for objects and a module for extracting positions from the four-dimensional trajectories. The module for extracting positions is downstream of the module for reconstructing the trajectories, and the output of the module for extracting positions is used as input to the module for reconstructing the four-dimensional trajectories for temporally subsequent images.

[0024] A process pipeline is a sequence of software modules that are linked together and can be partially parallelized. The first module(s) typically read or receive data via a corresponding communication interface. This may involve preprocessing, such as decoding, filtering, metadata extraction, or, particularly with image data, cropping to a specific area.

[0025] A data packet passes through the process pipeline and is successively processed by the modules. The outputs of the preceding modules serve as input for the subsequent modules. Depending on the configuration, data packets can be processed in parallel by multiple modules or can skip modules. This depends on the type and function of the modules.

[0026] One of the modules determines the position of each detected object in the image. Various techniques can be used for this. For example, a three-dimensional grid can be overlaid on the image, onto which the detected objects are projected. Based on the size of the objects, the distance to the sensor can be estimated, provided the object itself is known. However, positional errors can arise, for example, from uncertainty in estimating the object's size.

[0027] The position of each object is stored, for example, in a temporary or working memory. The object is then assigned an identifier, such as an identification number, to allow it to be identified for further processing. This step is also known as instantiation.

[0028] The position of each object is then needed to determine the trajectories of the objects in subsequent images and is thus fed back into the preceding module for determining the objects' trajectories. From several positions of an object, a trajectory can be determined, which can then be used to predict the object's position in the current or subsequent images. This prediction, in turn, can be used to better estimate the object's position in the current image and reduce positional inaccuracies. For example, plausibility checks can be performed for this purpose. If a vehicle is traveling at a certain speed in a certain direction, it is unlikely that it would suddenly change direction, for example, by turning sharply.

[0029] Which movements are plausible and which are not depends on many factors. These include, for example, the type of objects being recorded, the frequency with which the analyzed images are captured, the monitored process and its steps, etc.

[0030] The use of a process pipeline with modules for trajectory determination and position determination improves the overall accuracy with which the positions of the objects in the image data can be determined.

[0031] In one embodiment, the method further comprises the following steps: - Receiving image data comprising multiple images using LIDAR, with each image from the visual sensor being associated with a LIDAR image; - Detecting objects and their positions in the LiDAR images.

[0032] The positions of the objects from the LIDAR images are used to reconstruct the trajectories of the objects.

[0033] LiDAR (Light Detection and Ranging) is a remote sensing technology used for three-dimensional distance measurement and surface structure mapping. The basic principle of LiDAR is based on measuring the time it takes for an emitted light pulse to be reflected by a target object and return to the transmitter. Typically, a laser is used as the light source. Because laser beams are coherent light that can be generated with very short wavelengths, high precision in distance measurement is possible.

[0034] A LiDAR system can send multiple laser pulses per second towards a target object or landscape. Once these pulses hit a surface, they are reflected, and the LiDAR device measures the time it takes for the pulse to return. This measured time is then used to calculate the distance, since the speed of light is known.

[0035] A modern LiDAR system consists of several components: an emitter, a detector, and a signal processing unit. A LiDAR system can scan large areas and generate a point cloud. A point cloud is a collection of data points in three-dimensional space that represents the position of each reflected laser pulse. The quality of the point cloud depends on the pulse rate and wavelength of the laser, as well as the precision of the timing measurement. Typically, infrared or near-infrared light is used for terrestrial LiDAR, with wavelengths between 905 nm and 1550 nm being common.

[0036] LiDAR is used, for example, in the field of autonomous vehicles. Within the scope of the present invention, it can be used to create a highly accurate three-dimensional model of the environment and to detect obstacles. The high spatial resolution of LiDAR is crucial for reliably detecting objects such as pedestrians or traffic signs.

[0037] However, LiDAR also has some technical limitations. One of the biggest challenges is its limited range in adverse weather conditions. Because LiDAR relies on light waves, fog, rain, or dust significantly reduce its precision and range, as these particles can scatter or absorb the light before it reaches the target surface. It's important to note that light scattering is wavelength-dependent, meaning that laser light and visual light captured by a camera can react differently to varying environmental conditions.

[0038] Furthermore, processing the enormous amounts of data generated by LiDAR systems requires significant computing resources and storage. The resulting point clouds must be filtered, segmented, and interpreted, which necessitates complex algorithms and often machine learning. This makes LiDAR systems for vehicles relatively expensive.

[0039] This problem is solved by the described embodiment in that the processing of the LIDAR images takes place in parallel with the camera images and the segmentation can be taken from them.

[0040] Another problem can be the limited range of the LiDAR system. Typical terrestrial LiDAR systems have a range of about 100 to 200 meters, while airborne systems can reach several kilometers. In certain applications, such as autonomous vehicle navigation, this range is too short to provide sufficient time for safe decision-making at higher speeds.

[0041] Positioning based solely on LiDAR is therefore not suitable for every application. In a configuration that utilizes LiDAR and LiDAR data, LiDAR can be used, particularly for short-range applications (up to 100 m), to improve the positioning of objects in the sensor's vicinity. This is achieved by processing the LiDAR images by projecting the objects detected in the visual sensor's images onto the LiDAR image. This allows for improved object positioning, at least within the LiDAR system's operating range.

[0042] In one embodiment, the method further comprises the following step prior to reconstructing the trajectories: - Estimating depth information from the images of the visual sensor. The depth information from each image is used to reconstruct the trajectories.

[0043] Another method for improving position determination is depth estimation. This can be achieved, for example, by using points in the image whose distance is known. With static camera systems, fixed points can be used. These fixed points can be installations specifically designed for this purpose. In traffic, traffic signs, especially road markings, which are spaced at defined intervals, can be used. If no fixed points are available, empirical values ​​can be used, such as the height of people, which in most cases ranges from 1 meter for children to 2 meters for adults.

[0044] In one embodiment, the visual sensor comprises two cameras, wherein the depth information is determined from the combination of the images from the two cameras.

[0045] Another way to determine a distance is to use multiple cameras whose images are combined to calculate the distance to an object z. Using geometry and parallax, the distance to each object can be calculated, provided the distance between the cameras or the sensors they contain is known. This is typically the case with permanently installed sensor systems.

[0046] In some embodiments, several methods for determining the distance to the objects can be combined. The methods with the lowest errors can be used for the distance ranges. This allows for a high dynamic range in distance determination and reduces the error in position determination.

[0047] In one embodiment, the objects are vehicles, people, road signs or other objects in road traffic.

[0048] Objects in road traffic, such as people, traffic signs, and vehicles, are sufficiently recorded, resulting in a large database for these objects. Machine learning algorithms, in particular, can benefit from this large database by being specifically trained to determine distances using image data. However, even algorithms that don't employ machine learning can benefit from large datasets, as the experience gained from the data can also influence traditional algorithm design.

[0049] In another aspect, the invention relates to a computer program with program code for carrying out a method as described above when the computer program is executed on a computer.

[0050] In another aspect, the invention relates to a computer-readable data carrier containing the program code of a computer program for carrying out a method as described above when the computer program is executed on a computer.

[0051] In another aspect, the invention relates to a system for determining trajectories for objects from image data, wherein the system is configured to perform a method as described above.

[0052] The system can be, in particular, an on-board computer of a vehicle that receives image and / or LIDAR data from sensors inside the vehicle.

[0053] In summary, the present invention provides a computer-implemented method for determining trajectories for objects from image data, a computer program, a computer-readable data carrier, and a system for determining trajectories for objects from image data.

[0054] The described configurations and training programs can be combined in any way desired.

[0055] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or subsequently with regard to the exemplary embodiments that are not explicitly mentioned. Brief description of the drawings

[0056] The accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain the principles and concepts of the invention.

[0057] Other embodiments and many of the aforementioned advantages become apparent with reference to the drawings. The elements depicted in the drawings are not necessarily shown to scale.

[0058] They show: Fig. 1 schematically illustrates the process according to a first embodiment; Fig. 2 schematically illustrates the course of the procedure according to a second embodiment; and Fig. 3 schematically illustrates the process according to a third embodiment.

[0059] In the figures of the drawings, identical reference symbols denote identical or functionally equivalent elements, parts or components, unless otherwise stated.

[0060] Fig. Figure 1 schematically shows the inventive method according to a first embodiment, in which LIDAR images are used in addition to camera images.

[0061] In the first two steps, S10a and S10b, images from a camera and images from a LiDAR sensor system are provided. The camera and the LiDAR sensor system are aligned so that the image data can be superimposed and evaluated together.

[0062] In step S12a, the camera image data is segmented and the objects found within it are instantiated. Information from the visual spectrum is suitable for this purpose, as the objects can be visually distinguished. The segmentation and instantiation can be performed by a machine learning algorithm. The algorithm can be trained to recognize objects in the images that are relevant to the context of using the proposed method.

[0063] For example, the algorithm can be trained to recognize motor vehicles in images. To train the algorithm, images of various vehicle models can be fed into it, which can then be distinguished from the background based on shape, color (especially combinations of contiguous colored areas), and other factors. The same can also be done for pedestrians and traffic signs.

[0064] The quality of the algorithm's result is determined using a loss function. The algorithm is then adjusted so that the detection rate improves progressively with each iteration. This process is repeated until the loss function reaches an optimum.

[0065] Segmentation and instantiation are performed for each available image. In parallel, in step S12b, the distance to objects in all directions is determined in each LiDAR image. Identifying objects in LiDAR data is relatively difficult. However, a LiDAR image contains depth information that a camera image lacks.

[0066] The camera images are stored in a sequence with the instantiated objects in step S14a, and the LiDAR images are stored in a sequence with the depth information in step S14b. Both sequences can be combined in step S16 to generate a four-dimensional reconstruction of the sensor system's environment. The fourth dimension arises from the temporal sequence of the images. From this sequence, three-dimensional trajectories of the objects instantiated in the images can finally be obtained in step S18. For this purpose, the sequences are combined with the respective current images. For example, to determine the trajectory in the i-th image, images i-1 to in, with n>1, are stored as a sequence so that the movement of the objects can be determined from the n-1 preceding images and calculated for the i-th image.

[0067] Fig. Figure 2 shows an embodiment in which no LIDAR system is used. In this embodiment, the image data from a camera is processed in steps S10a to S14a. This processing of the image data is analogous to the embodiment shown in Fig. 1 is described.

[0068] However, this embodiment lacks the depth information that would be obtained from the LiDAR images. Therefore, in step S17, the position of the objects is fed back into the four-dimensional reconstruction of the environment. This initiates an iterative optimization loop, which allows the depth information to be estimated more accurately with each iteration.

[0069] The accuracy of the estimation can be increased by using a machine learning algorithm, specifically a convolutional neural network, for segmentation, instantiation, and depth-of-field estimation. This neural network can be trained to recognize different vehicle types and assign them a size. The recognized and identified vehicles can then be used as a fixed size to estimate the distance to neighboring objects. This can be achieved, for example, using a 3D grid projected onto the image or an iterative process for each individual object.

[0070] Once the positions for all objects in the image have been determined, the three-dimensional trajectory is provided in step S18.

[0071] Fig. Figure 3 shows a further embodiment of the invention. In this embodiment, images from a camera are provided in step S10a. The images are then processed in two parallel strands of a process pipeline. On the one hand, the images are processed analogously to the methods from the Fig. 1 and Fig. 2. In steps S12a and S14a, the images are processed, which includes segmenting the images, instantiating the objects detected in the images, and providing the image sequence and the instances it contains. In step S12c, depth information is estimated from the images, with this estimation being independent of the segmentation.

[0072] In this embodiment, the depth information for each image can be generated by a machine learning algorithm, in particular a deep learning model, and provided in the form of a depth map. Such a depth map is generated for each image, so that a sequence of depth maps is provided in step S14c.

[0073] The sequence of images with the instantiated objects and the sequence of depth maps are merged in step S16 to determine a position for each object in each image. This allows a trajectory for the objects in the images to be generated from the four-dimensional view in step S18.

[0074] The embodiment from Fig. 3 is distinguished from Fig. 2 in particular, in that in this embodiment no iterative loop is used, but the determination of the depth information is carried out in parallel with the processing of the images.

Claims

A computer-implemented method for generating annotations for objects from image data, wherein the method comprises the following steps: - Receiving image data (S10a) comprising multiple images from a visual sensor; - Segmenting the images and identifying objects (S12a) in the segments; - Reconstructing the trajectories (S18) of the objects by comparing the position of the objects in successively acquired images; and - Generating an annotation for the object using the trajectory. Computer-implemented method according to claim 1, wherein the determined trajectories of the objects in an image are used to reconstruct the trajectories of the objects in subsequent images, provided that the objects are still visible in the subsequent images. Computer-implemented method according to claim 2, wherein the method is represented by a process pipeline with several modules, the process pipeline comprising a module for reconstructing four-dimensional trajectories (S16) for objects and a module for extracting positions (S17) from the four-dimensional trajectories, wherein the module for extracting the positions is downstream of the module for reconstructing the trajectories, wherein the output of the module for extracting the positions is used as input to the module for reconstructing the four-dimensional trajectories for temporally subsequent images. A computer-implemented method according to one of the preceding claims, wherein the method further comprises the steps: - Receiving image data comprising multiple images using LIDAR (S10b), wherein each image of the visual sensor is associated with a LIDAR image; - Detecting objects and their positions in the LIDAR images (S12b), wherein the positions of the objects from the LIDAR images are used for the reconstruction of the trajectories of the objects. Computer-implemented method according to one of claims 1 to 3, wherein the method further comprises, prior to reconstructing the trajectories, the step of: estimating depth information from the images of the visual sensor (S12c), wherein the depth information of each image is used for the reconstruction of the trajectories. Computer-implemented method according to claim 5, wherein the visual sensor comprises two cameras, wherein the depth information is determined from the combination of the images from the two cameras. Computer-implemented method according to one of the preceding claims, wherein the objects are vehicles, persons, road signs or other objects in road traffic. Computer program with program code to execute a method according to any of the preceding claims when the computer program is executed on a computer. Computer-readable data carrier containing program code of a computer program for executing a method according to any one of claims 1 to 7 when the computer program is executed on a computer. System for generating annotations for objects from image data, wherein the system is configured to execute a method according to any one of claims 1 to 7.

Images