Method and apparatus for correlating optical flow over multiple images of an image detection device for a vehicle

Abstract

The invention relates to a method for correlating optical flow on a plurality of images of an image detection device for a vehicle (100). At least one tile is selected from a track map, wherein the track map comprises a plurality of flow tracks and is divided into a plurality of tiles. Subsequently, at least one flow track comprised by the tile is extended using flow vectors of a flow map which can be assigned to the flow track. Here, the flow map comprises a plurality of flow vectors based on a current image of the plurality of images.

Description

Technical Field

[0001] This invention relates to a method or apparatus for correlating optical flow across multiple images in an image detection device for a vehicle. A computer program is also protected by this invention.

[0002] Optical flow can be used to map motion onto image sequences. Background Technology

[0003] DE 10 2017 214 614 A1 discloses a method for testing the reasonableness of the flow vector hypothesis. Summary of the Invention

[0004] Against this backdrop, by means of the proposed solution, a method for correlating optical flow on multiple images of an image detection apparatus according to the invention, an apparatus for using the method according to the invention, and a corresponding computer program according to the invention are proposed. Through the measures enumerated in the preferred embodiments, advantageous extensions and improvements to the apparatus for correlating optical flow on multiple images of an image detection apparatus according to the invention can be achieved.

[0005] The described scheme enables the linking of flow vectors to determine optical flow over long time periods.

[0006] A method for correlating optical flow across multiple images from an image detection device includes the following steps:

[0007] Select at least one tile from the trajectory graph, wherein the trajectory graph comprises multiple flow trajectories and is divided into multiple tiles; and

[0008] When using flow vectors that can be assigned to flow trajectories in a flow graph, at least one flow trajectory included by the tile is extended. Here, the flow graph includes multiple flow vectors. The flow vectors are based on the current image among multiple images.

[0009] The image detection device can involve a camera, such as those used for environmental detection in vehicles. Image detection can capture images and provide them via an interface. Optical flow can describe the misalignment of image points between two images. This optical flow can be mapped using a flow graph. Each point in the flow graph can be assigned a flow vector. Flow trajectories can be formed by linking the flow vectors of the first flow graph with the flow vectors of the subsequent second flow graph when the endpoint of the flow vector of the first flow graph can be assigned to the starting point of the flow vector of the second flow graph. Flow trajectories formed in this way can be summarized in a trajectory graph. Flow trajectories can be defined by a starting point, any number of intermediate points, and an endpoint. If the flow trajectory is extended by another flow vector, the endpoint of the flow trajectory to date constitutes another intermediate point, and the endpoint of the other flow vector constitutes a new endpoint of the flow trajectory. The trajectory graph can be divided into regions, where each region can be assigned a tile. The trajectory graph can be continuously updated using the current flow vector. For example, the selected steps and extension steps of the method, as well as optional other steps, can be repeated until all tiles of the trajectory graph have been updated. According to one implementation, the steps of the method can be reimplemented as long as a new flow vector or a completely new flow graph is available.

[0010] When the endpoint of a flow trajectory can be assigned to the starting point of a flow vector, the flow trajectory can be extended in the extension step. For example, it can be checked whether the following flow vector exists in the current flow graph: the starting point of this flow vector can be assigned to the endpoint of the flow trajectory to be updated. If it exists, the found flow vector can be added to the existing flow trajectory. Thus, the endpoint of the found flow vector becomes the new endpoint of the extended flow trajectory. In this way, existing flow trajectories can be extended in a simple manner and, in principle, arbitrarily frequently.

[0011] If no flow vector is found in the current flow graph whose origin can be assigned to the end point of the flow trajectory to be updated, then the existing flow trajectory can be marked as inactive or invalid, for example, by using the corresponding validity attribute. Mark the flow trajectory. This method allows you to sort out flow trajectories that no longer continue.

[0012] The method can include the step of starting a new flow path: starting a new flow path when using flow vectors in a flow graph. This step can be performed when a block does not contain flow paths that can be assigned to flow vectors in a flow graph. For example, this can be the case when a block does not contain flow paths or, although it contains at least one flow path, no assignable flow vector can be found. Advantageously, this can offset the decrease in the number of flow paths.

[0013] In the activation step, a new flow trajectory with a starting point in the tile (i.e., the tile currently to be updated) can be activated. This can be the case when the Freigabesignal indicates that the tile is permitted to activate a new flow trajectory. Advantageously, in this way, flow trajectories can be activated first in areas that, for example, provide added value to downstream analysis and processing systems. According to one embodiment, a new flow trajectory can also be activated in a tile when the object signal indicates that the relevant object is mapped to an image segment of the current image belonging to the tile. Thus, the tracking of relevant objects is simplified when using a flow graph. For example, if the current tile is not permitted to activate a new flow trajectory, the activation step can be skipped or another tile can be selected in which a new flow trajectory can be activated.

[0014] Following the extension step, a sorting step can be performed, in which either a flow trajectory is sorted from the trajectory diagram or another flow trajectory is sorted. For example, sorting can be performed when the endpoint of a flow trajectory after extension is occupied by the endpoint of another flow trajectory. In this way, the intersection or merging of flow trajectories can be avoided.

[0015] Here, in the sorting step, to perform sorting, a flow trajectory or another flow trajectory is selected based on a comparison between the quality metric (Gütemaβ) of one flow trajectory and the quality metric of another flow trajectory. That is, a flow trajectory can be sorted if its quality metric is considered less relevant. For example, the length of the flow trajectory in pixels, the number of segments in the flow trajectory, the magnitude of the quantization error of the flow trajectory, or the origin of the flow trajectory can be used as a quality metric. Advantageously, this allows for a uniform distribution of the endpoints of the flow trajectories included in the trajectory map.

[0016] The sorting step can be performed or skipped based on the status signal. Advantageously, in this way, the flow trajectory can be sorted out or preserved precisely based on the status. For example, when using status signals, certain areas in the image can be excluded from sorting, or sorting can be omitted to reduce the calculation time for determining the shooting status.

[0017] When the extended flow trajectory (i.e., the flow trajectory extended in the extension step) meets the reopening criteria, an additional flow trajectory can be opened in the reopening step. For example, the number of flow trajectory segments, the length of the flow trajectory (i.e., the distance between the start and end points of the flow trajectory), or the acceleration of the flow trajectory can be used as the reopening criteria. This allows for the cancellation of sparsity in individual regions, which occurs when many flow trajectories are far from their starting points.

[0018] In the extension step, the endpoint of the flow trajectory before extension is retained as the midpoint of the flow trajectory. This allows for retrospective tracking of motion mapped through the flow trajectory, such as the motion of an object or a segment of an object.

[0019] In the extended steps, a validity attribute can be set for the flow trajectory, which indicates the validity of the flow trajectory. Thus, if the flow trajectory can continue, it can be marked as active or valid, for example. Otherwise, if the flow trajectory can no longer continue, it can be marked as inactive.

[0020] During the extension step, the quantization error of the flow trajectory can be updated. Quantization error can occur if the flow trajectory ends at a non-integer pixel, requiring rounding to the next integer pixel when extending the trajectory. Therefore, the quantization error reflects the deviation between the quantized endpoint of the flow trajectory used for the extension step and the actual endpoint. The magnitude of the quantization error is then updated during the extension step. Assigning the flow trajectory allows the quantization error to be taken into account, for example, when the flow trajectory is subsequently extended or when using a trajectory plot.

[0021] Therefore, in the extension step, the endpoint of the flow trajectory after extension can be corrected, for example, by the quantization error associated with the flow trajectory. For example, when extending the flow vector, the quantization error can be subtracted from the last endpoint of the trajectory. In this way, it can be ensured that the quantization error does not accumulate only in the positive direction or only in the negative direction over a long period of time.

[0022] The method can include a flow graph reading step: reading in a flow graph having a flow vector based on the current image among multiple images. In response to the flow graph reading, at least the selection and extension steps can be performed for at least one, multiple, or all tiles of the trajectory graph. For example, the reading step and subsequent steps can be repeated as long as a new flow graph subsequently becomes available in time. In this way, the trajectory graph can be continuously updated.

[0023] Advantageously, the steps of the method are implemented in real time. Advantageously, the trajectory map can be made available in a real-time updated manner.

[0024] The method can include the step of detecting the current image from at least a plurality of images when using an image detection device. To this end, the method can include the step of forming a flow graph using the current image. Known methods can be considered for forming the flow graph. Advantageously, the images provided by the image detection device can be processed immediately, and in real time, to update the trajectory graph.

[0025] This method can be implemented, for example, in software or hardware, or in a hybrid form of software and hardware, such as in a controller.

[0026] Furthermore, the present invention proposes an apparatus configured to perform, manipulate, or implement variations of the methods described herein within a corresponding device. Through these apparatus-based embodiments, the task on which the invention is based can also be solved quickly and effectively.

[0027] To this end, the device can have at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or actuator, and / or at least one communication interface for reading in or outputting data embedded in a communication protocol, wherein the at least one interface is used to read sensor signals from a sensor or to output data signals or control signals to an actuator. For example, the computing unit can be a signal processor, a microcontroller, or the like, wherein the storage unit can be flash memory, EEPROM, or magnetic storage. The communication interface can be configured for wirelessly and / or wiredly reading in or outputting data, wherein the communication interface capable of reading in or outputting wired data can read the data from a corresponding data transmission line electrically or optically, or can output the data to a corresponding data transmission line.

[0028] In this invention, the device can be understood as an electrical appliance that processes sensor signals and outputs control signals and / or data signals accordingly. The device can have an interface, which can be constructed in hardware and / or software. In the case of a hardware construction, the interface can be, for example, part of a so-called system ASIC, which contains various functions of the device. However, the interface can also be its own integrated circuit or at least partially composed of discrete structural elements. In the case of a software construction, the interface can be a software module, which, for example, exists on a microcontroller adjacent to other software modules.

[0029] Also advantageous is a computer program product or a computer program having program code that can be stored on a machine-readable carrier or storage medium (e.g., semiconductor memory, hard disk memory, or optical memory), and, particularly when the computer product or program is implemented on a computer or device, the program code is used to perform, implement, and / or manipulate the steps of a method according to one of the embodiments described above. Attached Figure Description

[0030] Embodiments of the solutions presented herein are shown in the accompanying drawings and described in more detail in the following description. The accompanying drawings show:

[0031] Figure 1 A schematic diagram of a vehicle having a device for associating optical flow according to an embodiment of the present invention is shown;

[0032] Figure 2 A graphical representation of sparse optical flow according to one embodiment is shown;

[0033] Figure 3 A flowchart illustrating a method for correlating optical flow according to one embodiment is shown;

[0034] Figure 4 A flowchart illustrating one embodiment of the steps for updating the flow trajectory;

[0035] Figure 5 A flowchart illustrating a method for updating a flow trajectory graph according to one embodiment is shown;

[0036] Figure 6 An updated trajectory plot according to one embodiment is shown;

[0037] Figure 7 An updated trajectory plot according to one embodiment is shown;

[0038] Figure 8 An indexed trajectory diagram according to one embodiment is shown;

[0039] Figure 9 An occupancy diagram is shown according to one embodiment;

[0040] Figure 10 A trajectory diagram according to one embodiment is shown;

[0041] Figure 11 A trajectory diagram according to one embodiment is shown;

[0042] Figure 12 An image showing a new flow trajectory according to one embodiment is shown;

[0043] Figure 13 A schematic diagram illustrating the data layout of a trajectory graph according to one embodiment;

[0044] Figure 14 An image showing a flow trajectory to be extended according to one embodiment is shown;

[0045] Figure 15 A flow diagram according to one embodiment is shown;

[0046] Figures 16 to 19 An image showing a flow trajectory according to one embodiment; and

[0047] Figure 20 A flowchart illustrating a method according to one embodiment is shown. Detailed Implementation

[0048] In the following description of advantageous embodiments of the invention, the same or similar reference numerals are used for elements shown in different figures and which have similar functions, wherein repeated descriptions of these elements are omitted.

[0049] Figure 1 A schematic diagram of a vehicle 100 having a device 102 for associating optical flow is shown according to an embodiment of the present invention. The vehicle 100 is, for example, a passenger car. Here, the application of the scheme described herein in the context of a vehicle 100 is merely exemplary. The described scheme can also be used, for example, in the context of surveillance cameras, robotics, or medicine.

[0050] Vehicle 100 has an image detection device 104, which detects, for example, the surrounding environment of vehicle 100, and exemplarily, the area in front of vehicle 100. For example, image detection device 104 includes one or more cameras. For example, image detection device 104 is a front-end video camera for vehicle 100.

[0051] Device 102 uses image 106 detected by image detection device 104 to determine trajectory map 108 and provides the trajectory map in a continuously updated form. Therefore, image 106 is also referred to as the input image. Exemplarily, the driving assistance system 110 of vehicle 100 uses trajectory map 108 to control driving assistance functions. According to one embodiment, trajectory map 108 maps the optical flow of image 106 detected by image detection device 104 over a longer time period.

[0052] Optical flow describes the misalignment of image points between two images in image 106. Hereinafter, the image points of image 106 are also referred to as pixels. Image 106 may be temporally misaligned, originating from a video sequence or from two different cameras of the image detection device 104. The temporal or spatial distance between images 106 has a significant impact on the optical flow result. The smaller the distance between images 106, the better the optical flow can be calculated.

[0053] Optical flow can exist in different forms. Dense optical flow calculates a flow vector for each pixel of image 106 or interpolates flow vectors. Quasi-dense optical flow can provide a flow vector for each pixel; however, the flow vector can also be invalid. When optical flow is shown as a flow graph, each pixel can, but does not necessarily, be described by a flow vector. Known methods can be considered as algorithms for creating the corresponding flow graphs. Sparse optical flow is optical flow that takes the form of a list with a small number (typically hundreds to thousands) of flow vectors. Here, for example, as described below... Figure 2 The optical flow shown depends on the time interval between images 106. For example, typical time intervals between consecutive images are 132ms, 264ms, or 528ms.

[0054] Dense or quasi-dense optical flow produces a result in the form of a flow graph, the resolution of which is typically the same as the resolution of the input image (i.e., image 106), or the resolution of the flow graph is a power of 2 of the input image's resolution; for example, one flow vector represents four pixels. This representation provides the user with the freedom to determine motion at any location within one of the images in image 106 without having to traverse all flow vectors. In contrast, the result of sparse optical flow exists as a flow list, which provides a start and end point for each entry (Eintrag), and the entries do not necessarily have a defined order. Within the framework of the method presented here, only the flow graph is observed.

[0055] Optical flow is used in a variety of applications. In the automotive field, optical flow is used in video cameras (e.g., image detection device 104) to describe the motion of a mapped scene. Various pieces of information can then be determined from the optical flow. These include, for example, the motion of the vehicle 100 itself (visual ranging), the motion of objects in the scene (vehicles, pedestrians, road signs, etc.), the depth of the scene (structure-from-motion), blind spots (e.g., lack of motion in the scene of the camera, such as image detection device 104), or the location of the vehicle 100 in the world.

[0056] All these applications also extend beyond the automotive sector, such as in robotics, spaceflight, and medicine. The implementation schemes described below primarily refer to automotive applications, but can be transferred in an unrestricted manner.

[0057] Figure 2A possible graphical representation of flow graph 200 is shown. Flow graph 200 represents sparse optical flow. Flow graph 200 includes multiple flow vectors 202. Here, the color or shape of each flow vector in flow vector 202 encodes the direction of the flow vector, and the saturation encodes the length of the flow vector. Here, the number of flow vectors 202 (also referred to simply as vectors) decreases as the time interval between images increases.

[0058] The shorter the temporal or spatial distance between the input image pairs for optical flow, the more accurate the determination of pixel motion (shown by flow vector 202), and the higher the availability, i.e., the number of flow vectors 202 for optical flow. Therefore, it is advantageous for the algorithm used to determine optical flow to have as small a distance as possible between the input images for optical flow.

[0059] Conversely, the greater the distance between input images, the fewer flow vectors 202 can be obtained, and the less accurate the result. Due to the distortion of the scene's geometry, image points may be assigned poorly, increasing inaccuracy.

[0060] From the perspective of algorithms utilizing optical flow results, longer time intervals, and consequently larger flow vector lengths, are generally advantageous. This is because motion described using flow vectors is subject to error. The longer the flow vector itself, the smaller the relative error of the flow vector. For example, Figure 2 The optical flow is shown on images with a time interval of 264 ms. With smaller time intervals between images, there are more but shorter flow vectors, and with larger time intervals between images, there are fewer but longer flow vectors.

[0061] The described scheme relates to a method for linking flow graphs, such as the flow graph 200 spanning multiple images shown. Here, for each starting point, the motion between the original image and each subsequent image is described. Not only the motion between the starting and ending images is calculated, but also the individual motions themselves. The result of the linked flow vectors is called a flow track. This assumes dense or quasi-dense optical flow between adjacent image pairs.

[0062] The advantage of the described scheme is that it can determine optical flow over any number of auxiliary points (intermediate images) over any length of time, and changes in the geometry of the image scene do not limit the accuracy and availability of the optical flow.

[0063] In addition to dedicated hardware (such as FPGA or ASIC), the described scheme can also be implemented in software, especially in real-time computation. Optionally, it is possible to achieve a uniform distribution of the endpoints of the flow trajectory. In particular, the described scheme can be used on an integrated image processing system for self-motion estimation, object recognition, and object tracking.

[0064] Unlike classic tracking algorithms, the described scheme aims not, or is not solely, to track a single ground marker or object. According to one embodiment, the goal is to generate an optical flow that is as uniformly distributed as possible over very long spatial or temporal distances, satisfying, for example, the aforementioned characteristics in terms of availability and accuracy. This is particularly important in the following two applications:

[0065] On the one hand, self-motion estimation requires the endpoints of the flow trajectory to be distributed as evenly as possible to avoid singularities. Or a degenerate flow vector configuration. For example, such a degenerate configuration exists when all flow vectors 202 lie on a single line, such as a horizontal line, or when all flow vectors 202 have the same origin. In the framework of the present invention, such a configuration is avoided by means of the following method.

[0066] On the other hand, optical flow can be used for depth reconstruction (structure-from-motion) of the mapped scene. This requires either time-varying camera movement or shooting with more than one camera. However, as mentioned above, if the temporal or spatial intervals between two shots need to be kept small, the sharpness of the separation between moving or stationary objects and the background also decreases. For example, if vehicles on a highway are approaching the end of a traffic jam, the stationary vehicles at the end of the jam need to be cleanly separated from the background or road. If this is not achieved accurately enough, vehicles cannot be identified as obstacles and braking cannot be initiated.

[0067] By linking the flow vector 202 over a longer time, the sharpness of object-background separation increases regardless of the scene content, and the accuracy of depth reconstruction also increases. To identify general objects, the scene needs to be scanned uniformly, which differs from the selection of specific features in the image used in classic tracking algorithms.

[0068] Figure 3 A flowchart illustrating a method for correlating optical flow according to an embodiment of the present invention is shown. For example, this enables methods such as... Figure 2 The optical flow shown is as follows: Figure 2 The flow vector shown supplements an additional optical flow, which is generated, for example, using at least one subsequent image or generated from a pair of subsequent images.

[0069] When the first flow graph is available, the method described in 300 begins. For example, such a flow graph is a list with flow vectors, as graphically represented in... Figure 2As shown in the diagram. For example, for each flow vector in the flow vectors, the flow graph includes the location of the starting point and the location of the ending point. For example, a first flow graph can be generated as long as a pair of images (from which the optical flow can be determined) are available.

[0070] According to one embodiment, the main algorithm of the method consists of three parts, which are presented in... Figure 3 The schematic diagram illustrates steps 302, 304, and 306 of the process. The main logic can be obtained through the so-called update track-routine shown in step 306, as described below. Figure 4 The main logic is described in more detail according to one embodiment. Before iteratively updating the flow trajectory in step 306, a seed point (seed) is determined in step 302, and then the seed point is further processed into an initialized flow trajectory in step 204.

[0071] As long as a new flow graph is available, repeat steps 302, 304, and 306, as shown in feedback (Rückkopplung) 308. A new flow graph can be generated whenever a new image or a new image pair is available.

[0072] According to one embodiment, the individual flow trajectories are summarized in a trajectory graph.

[0073] Figure 4 Shown in Figure 3 The flowchart illustrates one embodiment of step 306, which updates the flow trajectory of the trajectory graph. Step 306 includes multiple steps 410, 412, 416, and 418, some of which are optional and can be performed for each flow trajectory. In this way, each flow trajectory of the trajectory graph can be updated.

[0074] In step 410, it is checked whether the last trajectory point of the trajectory to be updated is valid. If it is valid, the updated trajectory is extended in step 412. Subsequently, collision detection is performed in optional step 414, and multi-index processing is performed in optional step 416.

[0075] If the result of step 410 indicates that the last trajectory point to be updated is invalid, then a reinitialization is performed in step 418.

[0076] The described scheme enables real-time updates to the flow trajectory. It is not necessary to have all intermediate images present at the outset. Instead, according to one embodiment, the trajectory is updated iteratively as new data becomes available, especially without having to classify the data in a computationally and memory-intensive manner.

[0077] Optionally, an approximately uniform distribution of flow vectors on the image can be achieved without reclassifying the flow vectors. According to an embodiment, this is achieved through two measures: an occupancy map for collision detection and a multi-indexing scheme.

[0078] According to one embodiment, the region where the linked flow vector begins can be optionally restricted to reduce computation time.

[0079] The linked flow vectors enable tracking of motion across intermediate images, not just between the starting and ending images. This allows for the selection of any temporal basis, starting from the current image or from previous images.

[0080] According to one embodiment, the quantization error that may occur when linking stream vectors is reduced to a minimum.

[0081] The flow graph on the input side can have spatial misalignment (offset). According to one embodiment, this misalignment is compensated for. This enables resource-efficient computation of optical flow.

[0082] Figure 5 A block diagram is shown for updating the current trajectory map 500 according to one embodiment. Step 306 is shown for updating the flow trajectory of trajectory map 500 to obtain an updated trajectory map 502. The current trajectory map 500 either relates to an initialized trajectory map or to a previous trajectory map, depending on whether the method is being implemented for the first time or is being implemented repeatedly.

[0083] By performing step 306, an updated trajectory graph 502 is determined using the current trajectory graph 500 and the current flow graph 200. The current flow graph 200 includes multiple flow vectors.

[0084] The current trajectory maps 500 and 502 are divided into multiple plots 510. Figure 5 Only one of the multiple tiles described herein has an attachment label. The trajectory diagram 500 is, by way of example only, divided into 36 tiles 510 arranged in six rows and six columns. According to the illustrated embodiment, all tiles 510 have the same size. Alternatively, at least some of the images in the tiles 510 have different sizes from each other. The arrangement of the tiles 510 in the columns and rows is also, by way of example only.

[0085] The current trajectory diagram 502 includes multiple flow trajectories 512, in Figure 4 Only one of the multiple flow tracks described herein is provided with a reference numeral. For example, each block in block 510 includes at least one flow track.

[0086] The updated trajectory map 502 includes multiple updated flow trajectories 514, in Figure 4 Of the multiple updated flow trajectories described herein, only one updated flow trajectory is provided with reference numerals. All updated flow trajectories 514, or at least a majority of updated flow trajectories 514, originate from an extension of the flow trajectory 512 included in the current trajectory diagram 500. Therefore, at least a majority of the updated flow trajectories 514 have a greater length than the current flow trajectory 512.

[0087] As long as a new flow graph exists, the updated trajectory graph 502 can be regarded as the new current flow graph and updated using the new flow graph, as described with respect to the illustrated flow graph 200 and the illustrated current trajectory graph 500. Therefore, step 306 can be performed continuously and repeatedly to update the corresponding existing current trajectory graph.

[0088] According to one embodiment, the flow graph 200 and trajectory graphs 500, 502 exist as lists, independent of the graphical representation shown in the accompanying drawings. For example, for each updated flow trajectory in the updated flow trajectory 514, the list representing the updated trajectory graph 502 includes values ​​defining a start point, an end point, and at least one intermediate point that may be generated by extension.

[0089] The described scheme enables real-time optical flow linking. According to... Figure 5 The following description is based on one embodiment and is updated in real time.

[0090] to this end, Figure 5 The update of flow trajectories 512 is illustrated using a flow graph 200 derived from optical flow. Each flow trajectory 512 is associated with a dedicated tile 510, in which the points of the corresponding flow trajectory 512 are stored. Each tile 510 contains M flow trajectories, where typically M = 2…3. For clarity, only one trajectory is shown for each tile 510 in the drawing. The size of the tile 510 is variable. In practice, a size of 6 to 24 pixels per dimension has proven suitable. The height and width can be the same, but are not required to be the same. In the update routine illustrated by step 306, each tile 510 is first examined individually. If a valid or active flow trajectory 512 exists in a tile 510 of the current trajectory graph 500, that flow trajectory 512 is extended. If no active flow trajectory 512 exists, a new flow trajectory is initiated, with its starting point located in tile 510 where no active flow trajectory 512 exists. Here, for example, the best starting point for the new flow trajectory is selected based on quality characteristics present in flow graph 200.

[0091] When extending one of the flow trajectories in flow trajectory 510, two situations can occur:

[0092] In the first case, in the flow graph 200 on the input side, the endpoint of the current flow trajectory 510 matches a valid entry (trifft...auf). The flow trajectory 510 is extended, and new data points are stored in memory, for example.

[0093] In the second case, there is no valid flow vector at the end of flow trajectory 510 in flow graph 200. In this case, it is possible to either use an adjacent flow vector from flow graph 200, which results in a correspondingly higher error, or mark the end of flow trajectory 510 as invalid.

[0094] The method is repeated for all flow trajectories 510 of the current trajectory graph 500. Thus, updates to the flow trajectories 510 are performed in the iterative steps, starting from an initialized or previous trajectory graph 500 with linked flow vectors (the linked flow vectors are referred to herein as flow trajectories 510).

[0095] If it is necessary to extend the corresponding flow trajectory 510, optimization of the updated trajectory map 502 is performed in an optional step to achieve a uniform distribution. This is achieved, on the one hand, through collision detection, and on the other hand, through a multi-indexing scheme, as illustrated in the following figures.

[0096] Figure 6 An updated trajectory map 502 according to one embodiment is shown. This, for example, relates to... Figure 5 The updated trajectory diagram 502 is described. For clarity, only one of the multiple tiles 510 has reference numerals, and only one of the multiple flow trajectories 510 has reference numerals.

[0097] from Figure 6 As can be seen, the block 510 of the flow trajectory diagram 502 can be occupied by multiple flow trajectory endpoints. Here, the endpoint of the flow trajectory 514 corresponds to the endpoint of a flow vector, and the flow trajectory 514 is ultimately extended by this flow vector. The flow trajectory 514, which is provided with reference numerals, is exemplary composed of three flow rates.

[0098] The reason for the multiple occupancy of tile 510 is that flow trajectory 514 is stored in its starting tile and the endpoints of flow trajectory 514 only virtually overlap. However, this is not a desirable characteristic for two reasons. First, the distribution of flow trajectory endpoints is no longer uniform, but concentrated at specific locations in the image represented by flow trajectory map 502. However, a uniform distribution is advantageous, for example, for self-motion estimation.

[0099] On the other hand, given a defined camera orientation or driving direction, it is possible for the flow trajectory endpoints to converge (compress) at a location within the image. This is particularly true when the so-called focus-of-compression (FOC) is located within the image. The focus-of-compression (FOC) is, to some extent, the same as the focus-of-expansion (FOE) in the opposite driving direction. This occurs, for example, when the driving direction is straight and the camera is pointing backward, or when the driving direction is backward and the camera is pointing forward.

[0100] exist Figure 6 In the diagram, the shaded portion of block 510 in the flow trajectory diagram 502 indicates the multiple occupancy of block 510, which passes through the endpoint of flow trajectory 514: multiple flow vectors 514 end in the block. In the case of a flow trajectory 514 composed of multiple flow vectors, the last segment extends toward the edge of flow trajectory diagram 502.

[0101] exist Figure 6 The image schematically shows the forward movement of the front-end camera, in which the flow trajectory 514 moves from the image center (expanded focus) to the edge and collides with the shorter flow trajectory 514 there.

[0102] Figure 7 An updated trajectory diagram 502 according to one embodiment is shown. Figure 6 Different, schematically shown in Figure 6 The image shows the front camera moving backward. In the case of backward movement, the flow trajectory 514 is compressed at a singularity at the center of the image. Even when moving forward again, only a small amount of flow trajectory 514 with a different origin remains.

[0103] Figure 8 An updated trajectory map 502 according to one embodiment is shown. This relates to... Figure 7 The trajectory diagram shown is now indexed. Thus, the tile 510 with the attached reference numerals is indexed "9".

[0104] In the case of this indexed flow trajectory diagram 502, each tile 510 has an index, such as an address.

[0105] Figure 9 Occupancy diagram 902 is shown according to one embodiment. Occupancy diagram 902 is based on... Figure 8 The indexed flow trajectory diagram is shown in the figure.

[0106] In the occupancy diagram 902, there exists an address for the flow trajectory to be occupied. In memory, for example, for storing a list of mapped flow trajectory diagrams, there is a separate address; multiple names only indicate collisions to be triggered. If tile 510 is not occupied, a "Null-Addresse" is used.

[0107] The use of occupancy diagram 902 illustrates a strategy for utilizing congestion patch 510 at the end of a flow trajectory. Here, in each, for example, according to... Figure 5 After the described trajectory expansion, it is checked whether the address of another flow trajectory already exists at the end of the current flow trajectory in the occupancy diagram 902. If no other flow trajectory address exists, the address of the current flow trajectory is saved. If an occupancy already exists, a collision strategy is applied to select a better flow trajectory. For example, this collision strategy can use the length of the flow trajectory in pixels, the number of segments (corresponding to the epoch of the flow trajectory), the magnitude of the quantization error, the origin of the flow trajectory, or other quality metrics of the flow trajectory.

[0108] In some cases, storing the occupancy of the endpoint in plot 902 can lead to significant sparsity in the generated flow trajectories. If the objective is not necessarily a uniform distribution of endpoints, but rather to avoid a degenerate configuration used for its own motion estimation, it makes more sense to store the starting point of the last trajectory segment instead of the endpoints, which corresponds to the penultimate point of the flow trajectory. Therefore, unlike the approach mentioned above, instead of preventing multiple occupancy of endpoints, it prevents multiple occupancy of plot 510 containing either the starting point of a flow trajectory or a trajectory segment.

[0109] A key feature of the described scheme is that the occupancy graph 902 does not store, for example, the number of flow trajectories occupying the occupancy or other metadata, but rather stores the address of the occupied flow trajectory, thus enabling direct memory access. Another feature closely related to the multi-index scheme described below (Hand in Hand) is that, for collision detection, memory-intensive and computationally expensive reclassification of flow trajectories is not required.

[0110] Another feature is the ability to turn collision detection on or off depending on the situation, and to activate or deactivate it based on a region. To reduce computation time, it might be meaningful to only activate collision detection under specific conditions. For example, this is the case when a vehicle with a front-end camera is moving backward. Collision detection is not needed in other scenarios. Similar behavior can be achieved when collision detection is activated only in one or more regions. Here, the region surrounding the expansion focus is also suitable.

[0111] According to one embodiment, a status signal is used to indicate a situation in which sorting of flow trajectories is desirable. For example, the status signal is provided by a control device for controlling the operation of an image detection apparatus. If the status signal indicates desirable sorting, then, for example, after or during each extension of the flow trajectory, it is checked whether the flow trajectory can be sorted from the trajectory diagram. If a flow trajectory to be sorted is found, then the flow trajectory is sorted from the trajectory diagram or, for example, marked as invalid or inactive. Conversely, if the status signal does not indicate desirable sorting, then, for example, such a check is not performed or flow trajectories that could potentially be sorted are not sorted.

[0112] Figure 10 A trajectory diagram 502 according to one embodiment is shown. Unlike the previous figures, each flow trajectory in the flow trajectory 514 is labeled with its endpoint 1010. For clarity, only one block in the diagram 510 is labeled, and only one new flow trajectory and its endpoint are labeled in the new flow trajectory shown.

[0113] according to Figure 10 and Figure 11 Describe the multi-index scheme. The multi-index scheme is another strategy used for homogenizing the flow trajectory map 502, and especially for homogenizing the distribution of the endpoint 1010 of the flow trajectory 514. For example... Figure 10 As shown, the endpoint 1010 of the flow trajectory 514 moves further away from its starting point over time, but this is actually the expected behavior. However, in many camera orientations, this causes the area in the image to become verwaisen rather quickly. This is especially true in areas of FOE.

[0114] To prevent sparsity around the endpoint 1010 of the FOE, a multi-indexing scheme is used in the illustrated approach. Here, when a flow trajectory 514 deviates from its original tile, it is identified. Then, when it exceeds a certain threshold, a new flow trajectory 514 is initiated, while the previous flow trajectories 514 continue. In this way, it is ensured that the distribution of the endpoint 1010 of the flow trajectories 514 is approximately uniform. At the data level, an attribute is added to each flow trajectory 514, describing whether the flow trajectory 514 is active or inactive. Alternatively, it is possible to query the state of the last point in the flow trajectory 514, which gives the validity of the flow trajectory 514. The reply.

[0115] The number of flow trajectories 514 for each tile is configurable and is primarily determined by the requirements of the abnehmenden algorithm and the rate of scene change. For self-motion estimation applications, two flow trajectories 514 per tile 510 are sufficient in practice. For object detection and tracking, a higher number is desirable. This may involve increasing or decreasing the number of flow trajectories 514 per tile 510 during runtime within a defined region of interest. This might be the case, for example, in regions of FOE or along horizontal lines, to detect objects at a distance. A range of strategies exist for determining when to activate another flow trajectory 514, described below. The metric for minimum computational intensity is the number of segments of the flow trajectory 514, and consequently, the duration of that flow trajectory. If two flow trajectories 514 collide, the longer flow trajectory is retained; if they are of equal length, the existing flow trajectory is acquired. This results in reduced memory access.

[0116] Another strategy that might be better suited algorithmically is to determine the geometric distance between the starting point and the ending point 1010. A new flow trajectory 514 is initiated as long as this distance is above a threshold. The advantage here is that in the case of very small movements, where the flow trajectory 514 does not, for example, leave its own tile 510, another flow trajectory 514 is not needed. Geometric distance is also suitable as an indicator in the case of flow trajectories 514 that initially move away from the starting point and then approach it. This saves computation time and memory access. Furthermore, both of the mentioned indicators can depend on the image region and therefore behave differently locally.

[0117] According to one embodiment, instead of geometric distance, the velocity variation of the flow trajectory 514 is used. If the flow trajectory 514 accelerates, i.e., if the segment becomes longer, it may be meaningful to begin a new flow trajectory 514. In this case, an increase in the velocity variation can be expected. A similar scheme at the global level is implemented using vehicle signals. Typically, velocity and acceleration, as well as changes in vehicle orientation (yaw rate, pitch rate, and roll rate), are provided via the vehicle bus. All these metrics can be used to change the threshold for initiating a new flow trajectory 514, where a large acceleration signal results in the earlier initiation of a new flow trajectory 514. Figure 11 This illustrates, according to one embodiment, the result after applying a multi-indexing scheme, based on... Figure 10 The described trajectory diagram 502. For example, a flow trajectory 514, marked with reference numerals, moves from the FOE in a direction toward the edge of the image. A new additional flow trajectory 1114 is created for this flow trajectory 514, the starting point of which is located in the same tile 510 as the starting point of the flow trajectory 514.

[0118] In this way, the endpoint 1010 can be offset. Figure 10 The uneven distribution is visible in the image. For each flow trajectory 514, a new additional flow trajectory 1114 is created where the endpoint 1010 of the flow trajectory is sufficiently far from its original tile. Thus, the distribution of endpoints 1010 is more uniform compared to the previous one.

[0119] For example, trajectory diagram 502 is supplemented with six additional flow trajectories 1114, each having an endpoint 1010 near the FOE.

[0120] Figure 12 An image 106 according to one embodiment is shown, which has a new flow trajectory 1214 determined using image 106. Furthermore, a cross is drawn, representing the expanded focus of interest (FOE) 1220. The new flow trajectory 1214 can be incorporated into the trajectory diagram as already described.

[0121] according to Figure 12 This describes possible limitations on the starting region used for the new flow trajectory 1214.

[0122] exist Figure 12 The new flow trajectory 1214 is shown in a single traversal (e.g., in the case of...). Figure 5 During the update of the described trajectory map, the new flow trajectory is reinitialized. According to one embodiment, this reinitialization is performed if a previous flow trajectory of a tile cannot find a successor or has been deleted during collision detection. As can be seen from image 106, vectors at the image edges are primarily reinitialized, especially those in regions far from FOE 1220. Additionally, it can be seen that vectors in the next image, i.e., the new flow trajectory 1214 supplemented with the flow vectors derived from the next image, will no longer be located in the image plane. That is, its long-term use is limited, while computational and memory access overhead is high.

[0123] To address this issue, according to one embodiment, region-based restrictions on execution trajectories are enabled to improve computational efficiency. In this context, understanding the number of memory accesses is crucial for selecting a starting point. If no active or valid flow trajectory exists within the tile, a suitable starting point must be found for the new flow trajectory 1214. Ideally, these starting points are provided externally. If this is not feasible, according to one embodiment, all possible starting points for the tile are examined for suitability. Typically, this not only implies high computational overhead but also increased overhead for accessing (potentially slow) working memory.

[0124] To reduce the overhead, according to one embodiment, region-based activation of the starting point is performed. Parameterization determines which tiles are allowed to initiate a new flow trajectory 1214. Typically, this is not meaningful in all regions of image 106. In the case of a front-end camera during forward travel, initializing a new flow trajectory 1214 at the image edges adds less value because the new flow trajectory 1214 will move out of the image region in the near future or be covered by a longer trajectory within the collision detection framework. In the case of a side-oriented fisheye camera, the flow vector is compressed at the center of the left or right image edge. According to one embodiment, this region is also omitted here to reduce computational and memory load.

[0125] According to one embodiment, an permission signal is generated that indicates which tiles are permitted to open a new flow trajectory 1214. For example, the permission signal is generated based on the operating mode of an image detection device (which has detected image 106). Alternatively, the corresponding permission signal is predetermined. When using the permission signal, it is possible to determine for each traversal whether opening a new flow trajectory 1214 is permitted in a tile.

[0126] Another approach is to restrict the starting region to objects in image 106. These objects are predefined externally, for example, by shape-based object detection or semantic segmentation. In this case, the new flow trajectory 1214 is restricted to the predefined objects and is suitable for tracking. For example, a vehicle mapped in image 106 can be used as such an object 1225.

[0127] According to one embodiment, an object signal is generated that indicates which tiles contain object 1225. This object signal is provided, for example, by an object recognition device configured to identify the associated object 1225 in image 106 and display the location of the associated object within image 106. When using the object signal, it is possible to determine for each traversal whether a tile covers an area of ​​image 106, for example, that is related to the tracking of object 1225. If it covers, then the opening of a new flow trajectory 1214 can be permitted for that tile. Otherwise, the tile can be blocked to reopen the trajectory.

[0128] Figure 13Schematic diagram showing the data layout 1330 of the trajectory graph 502 according to an embodiment. The trajectory graph 502 is also referred to as a trajectory map. Exemplarily, for each flow trajectory 514 of the trajectory graph 501, a starting point 1310, a first intermediate point 1312, a second intermediate point 1314, and an end point 1010 are shown. The flow trajectory 514 extends between the starting point 1310 and the end point 1010. The flow trajectory 514 consists of a first flow vector extending between the starting point 1310 and the first intermediate point 1312, a second flow vector extending between the first intermediate point 1312 and the second intermediate point 1314, and a third flow vector extending between the second intermediate point 1314 and the end point 1010.

[0129] For each of the points 1310, 1312, 1314, 1010 there is an x coordinate and a y coordinate, validity information "v", and quantization errors dx and dy generated when making a link. For the sake of clarity, each gallery 510 shows only one flow trajectory 514, where only one tile and one flow trajectory are provided with reference numerals. The scheme includes: each tile 510, M flow trajectories 514, in order to achieve a uniform distribution of the end points 1010 of the flow trajectories 514, as can be achieved, for example, by the described multi-index scheme.

[0130] In Figure 13 the structural construction of a single flow trajectory 514 is exemplarily shown. It can be seen that the trajectory consists not only of the starting point 1310 and the end point 1010, but also of the corresponding individual points (x and y), the validity attribute, and the quantization errors dx and dy. Thereby, it can be achieved that, starting from the last image or image point, arbitrarily long or arbitrarily short distances of the flow trajectory 514 can be reconstructed. This is particularly important in practice because the current image is always the reference parameter here. When the current image is labeled as T, then in practical applications, not only self-motion estimation between the image T and T-X is required, but also depth reconstruction of the scene and object detection are required. Here, X can be selected as X = 1…m, where m is a variable integer, and m < M-1, where M is the maximum number of points 1010, 1310, 1312, 1314 stored in the flow trajectory 514.

[0131] Through the data layout introduced here, although there is only one trajectory graph, different m values can be achieved. Therefore, self-motion estimation at multiple time distances or spatial distances can be achieved, and additionally, depth reconstruction of the scene at multiple time distances can be achieved. In practice, different time distances are used to identify different types of objects in the scene.

[0132] For fast-moving objects, such as a cyclist crossing or an oncoming vehicle, a short time basis is suitable because the object is either only briefly visible in the image or is limited by optical flow (as described below). Figure 15 As described, it is not possible to accurately identify the object over a long time period. On the other hand, a long time basis is advantageous for objects that are far away or moving slowly. For example, this could be a vehicle traveling ahead or a vehicle at the end of a traffic jam. The data scheme shown here allows for the simultaneous operation of two application scenarios.

[0133] Figure 14 Image 106 is shown according to one embodiment, having a flow trajectory 514 to be extended. The flow trajectory 514 is extended using the current flow vector 202, where quantization errors dx and dy are generated. Quantization error occurs when the flow vector corresponding to the flow trajectory 514 ends at a non-integer value. To read the next flow vector (here, flow vector 202), the endpoint must be rounded to the next integer pixel. This generates quantization error. Minimizing quantization error is advantageous.

[0134] Depend on Figure 13 It can be seen that each point in the flow trajectory 514 also contains information about the quantization error (dx and dy). This quantization error occurs when the flow trajectory 514 ends at a non-integer pixel, as is common in flow algorithms with sub-pixel estimation. If the flow trajectory 514 should be extended, it needs to be rounded to the next integer pixel from the end of the flow trajectory 514, such as... Figure 14 As shown in the figure. This results in quantization error.

[0135] Assuming that subpixel estimation always exists in the optical flow, the subpixel can be constrained to ±1px by subtracting the quantization error from the last point of the flow trajectory 514 when linking the flow vector 202. First, the calculation is performed to access the point with the marker x. idx and y idx The index of the flow graph. Here, the quantization error of the last traversal is subtracted from the endpoint of the last traversal (Durchlauf):

[0136] x idx =round(x t-1 -dx t-1 )

[0137] y idx =round(y t-1 -dy t-1 )

[0138] To link the flow vectors, specifically to link the flow trajectory 514 with flow vector 202, the optical flow (OF(x, y)) is now queried at the pre-calculated rounded coordinates and added as components to the last point. xt-1 y t-1 ):

[0139] (x t y t )=(x t-1 y t-1 )+OF t-1，t (x idx y idx )

[0140] The new quantization error is now determined as shown in the equation below by subtracting the rounded index of the previous endpoint x or y from the true value.

[0141] d xt =round(x t-1 -dx t-1 )-(x t-1 -d xt-1 )

[0142] d yt =round(y t-1 -dy t-1 )-y t-1 -d yt-1 )

[0143] This ensures that the quantization error does not accumulate only in the positive or negative direction over a long period of time, but does not exceed ±1px.

[0144] An alternative to this approach is to interpolate the neighborhood of the flow vector. The determination of the next trajectory segment observed so far corresponds to nearest-neighbor interpolation. In other interpolation cases, such as bilinear interpolation, no quantization error is introduced, but rather rounding error. However, this calculation is several times more computationally expensive and involves more access to slow working memory because the data is distributed across multiple image rows in memory. Cache-friendly access is not possible. Furthermore, there is no guarantee that the results will actually become more accurate through interpolation.

[0145] Figure 15 Flow graphs 200 and 1500 are shown according to one embodiment. Flow graph 200 from the current traversal and flow graph 1500 from the previous traversal are shown. Flow graphs 200 and 1500 can have spatial misalignment, such as... Figure 15 As shown schematically.

[0146] In systems that move freely in the world, not all regions of an image have the same correlation (Relevanz). For this reason, it is proposed to spatially restrict resource-intensive computations of optical flow, such as those with flow graphs of 200 or 1500. Figure 15 As shown. In many cases, it is meaningful that the region of optical flow follows a specific location in the image or is confined to a specific area. For example, this could be an object, or an expansion focal point. Conversely, the sky, for example, often has low correlation. To achieve this, according to one embodiment, spatial misalignment of the flow field is compensated for. The true dimensions of the flow trajectory map are determined separately.

[0147] Figures 16 to 19 Images 106, 1706, 1806, and 1906, with flow trajectories 514, are shown according to one embodiment. Here, flow trajectories 514 with representative encodings of individual flow vectors are shown. Each feature represents a flow vector from a single time step (Zeitschritt) between two images 106, 1706, 1806, and 1906. For example, a dotted line segment represents a new flow vector, and a dashed line flow vector represents an old flow vector. Flow trajectories 514 for self-motion estimation in a static world are shown, as well as flow trajectories 514 for use in object recognition on a moving vehicle.

[0148] Figure 20 A flowchart is shown for a method of associating optical flow across multiple images. For this purpose, for example, flow vectors can be determined from a series of temporally consecutive images, and the flow vectors can be sequentially strung together to form flow trajectories. This relates, for example, to one embodiment of the method, as described with reference to the preceding figures.

[0149] In step 2000, at least one tile is selected from the trajectory map. For example, in Figure 8 The diagram shows a corresponding trajectory graph with multiple tiles and flow trajectories. In step 2002, at least one flow trajectory, which is associated with a selected tile, is extended. Here, the flow trajectory is extended using the flow vector of the flow graph, such as, for example... Figure 2 As shown. According to one embodiment, steps 2000 and 2002 are repeatedly performed for all or selected tiles in the trajectory map.

[0150] If no trajectory graph is still available, then according to one embodiment, all flow vectors or a selection of flow vectors from the flow graph are used as the initial flow trajectory.

[0151] Optionally, in step 2002, the endpoint of the flow trajectory before extension is retained as the midpoint of the flow trajectory. Additionally or alternatively, a flow trajectory validity attribute is set, which indicates the validity of the flow trajectory. Additionally or alternatively, the quantization error of the flow trajectory is updated. According to one embodiment, in the extension step, the endpoint of the flow trajectory after extension is corrected with the quantization error associated with the flow trajectory.

[0152] According to one embodiment, when the end point of the flow trajectory can be assigned to the start point of the flow vector, the flow trajectory is extended in step 2002. If this is not the case, according to one embodiment, instead of step 2002, step 2004 is performed, in which a new flow trajectory is started when using the flow vector of the flow graph. According to one embodiment, step 2004 is also performed when the tile is not assigned to a flow trajectory.

[0153] According to one embodiment, in step 2004, a new flow trajectory with a starting point in the block is initiated when the permission signal 2030 indicates that the block is permitted to open a new flow trajectory, or when the object signal 2032 indicates that the associated object is mapped into an image segment of the current image (where the flow vector has been determined to be used as the new flow trajectory if the current image is used). The permission signal 2030 and—additionally or alternatively—the object signal 2032 are provided, for example, by an image analysis processing device.

[0154] Optionally, in step 2006, either the extended flow trajectory from step 2002 or another flow trajectory is sorted from the trajectory diagram. Sorting is performed when the endpoint of the extended flow trajectory is occupied by the endpoint of another flow trajectory. To determine which flow trajectory to sort, a comparison is performed, for example, between quality metrics associated with the flow trajectory, and the selection of the flow trajectory to be sorted is based on the result of the comparison. According to one embodiment, the sorting step is performed or skipped based on a status signal 2034. For example, the status signal 2034 is provided by a control device for controlling the operation of an image detection apparatus for detecting images. For example, the status signal 2034 is provided according to the operating mode of the image detection apparatus.

[0155] Optionally, when the flow trajectory meets the reopening criteria after the extended step 2002, an additional flow trajectory is opened in step 2008.

[0156] Optionally, the method includes step 2010, in which a flow graph is read in. Optionally, the method further includes step 2012, in which an image is detected, and the method additionally or alternatively includes step 2014, in which a flow graph is determined using at least one of the detected images.

[0157] According to one embodiment, the method is implemented in real time. In this way, an updated trajectory map can be determined, for example, in response to each newly detected image.

Claims

1. A method for correlating optical flow across multiple images in an image detection apparatus for a vehicle (100), wherein, The method includes the following steps: At least one tile (510) is selected (2000) from the trajectory graph (502), wherein the trajectory graph (502) comprises multiple flow trajectories and is divided into multiple tiles; and The extension (2002) is performed on at least one flow trajectory (514) included in the tile (510), the extension being performed using flow vectors of the flow graph (200) that can be associated with the flow trajectory (514), wherein the flow graph (200) includes a plurality of flow vectors based on the current image (106) among the plurality of images. In the extension step (2002), the quantization error of the flow trajectory (514) is updated, wherein the quantization error shows the deviation between the quantized endpoint of the flow trajectory for the extension step and the actual endpoint.

2. The method according to claim 1, wherein, When the endpoint of the flow trajectory (514) can be assigned to the starting point of the flow vector (202), the flow trajectory (514) is extended in the extension step (2002).

3. The method according to any one of claims 1 to 2, the method comprising the step of enabling (2004): when the block (510) does not include a flow trajectory of a flow vector (202) that can be associated with the flow graph (200), enabling a new flow trajectory (1214) while using the flow vector (202) of the flow graph (200).

4. The method according to claim 3, wherein, When the permission signal (2030) indicates that the tile (510) is permitted to open a new flow trajectory, or when the object signal (2032) indicates that the associated object (1225) is mapped into the image segment of the current image (106) belonging to the tile (510), a new flow trajectory (1214) with a starting point (1310) in the tile (510) is opened in the opening step (2004).

5. The method according to any one of claims 1 to 2, the method having a sorting step (2006): in the sorting step, when the end point (1010) of the flow trajectory (514) after the extension is occupied by the end point of another flow trajectory, either the flow trajectory (514) or the other flow trajectory is sorted from the trajectory diagram (502).

6. The method according to claim 5, wherein, In the sorting step (2006), in order to perform sorting, either the flow trajectory (514) or the other flow trajectory is selected based on a comparison between the quality metric of the flow trajectory (514) and the quality metric of the other flow trajectory.

7. The method according to claim 5, wherein, The sorting step (2006) can be performed or skipped based on the status signal (2034).

8. The method according to any one of claims 1 to 2, the method having a restart step (2008): in the restart step, when the flow trajectory (514) after the extension meets the restart criterion, an additional flow trajectory (1114) is started.

9. The method according to any one of claims 1 to 2, wherein, In the extension step (2002), the endpoint (1010) of the flow trajectory (514) prior to the extension is retained as the midpoint (1312, 1314) of the flow trajectory (514), and / or, the validity attribute of the flow trajectory (514) is set, wherein the validity attribute indicates the validity of the flow trajectory (514).

10. The method according to any one of claims 1 to 2, wherein, In the extended step (2002), the endpoint (1010) of the flow trajectory (514) after the extended step is corrected for the quantization error associated with the flow trajectory (514).

11. The method according to any one of claims 1 to 2, the method comprising the step (2010) of reading the flow graph (200): reading the flow graph (200) having a flow vector based on the current image (106) among the plurality of images, wherein, In response to the reading of the flow graph (200), at least the selected step and the extended step (2000, 2002) are performed for at least one block (510) of the trajectory graph (502).

12. The method according to any one of claims 1 to 2, wherein the method is implemented in real time.

13. The method according to any one of claims 1 to 2, the method having a detection step (2012) and / or a step of forming the flow graph (200) (2014): detecting the current image (106) among the plurality of images when using the image detection device, and forming the flow graph (200) when using the current image (106).

14. An apparatus (102) for associating optical flow on multiple images of an image detection device for a vehicle (100), the apparatus being configured to implement and / or manipulate the steps of the method according to any one of claims 1 to 13 in corresponding units.

15. A computer program product comprising instructions that, when executed by a processor, perform the steps of the method according to any one of claims 1 to 13.

Images