METHOD AND DEVICE FOR VALIDATING A FLOW VECTOR HYPOTHESIS

Description

State of the art

[0001] The present invention relates to a method and a device for the plausibility of a flow vector hypothesis

[0002] In optical flow (OF), correspondences are formed in the temporal direction by determining mappings between coordinates in a first image and coordinates in a second image. Such a correspondence then indicates how the projection of a point in the 3D scene into the 2D image has moved from an old coordinate to a new coordinate.

[0003] The movement in the image can be caused by the movement of the scene point or by the movement of the camera, or both at the same time.

[0004] Jeffrey W. McCandless discloses in "Detection of aircraft in video sequences using a predictive optical flow algorithm", OPTICAL ENGINEERING., Vol. 38, No. 3, March 1, 1999, pages 523-530, XP055503109 an image processing algorithm that determines the optical flow to detect moving objects.

[0005] Optical flow is already used in various products, for example in driver assistance systems.

[0006] The object of the present invention is to improve the assessment of flow vectors. Disclosure of the invention

[0007] The above object is achieved according to the invention according to independent claims 1 and 9.

[0008] The inventive method for checking the plausibility of a flow vector hypothesis comprises determining a prediction vector associated with a feature based on a movement of the feature between a first image and a second image, wherein the second image is an image following the first image in a sequence of images, generating a hypothesis vector associated with the feature, which describes a presumed movement of the feature between the second image and a third image, wherein the third image is an image following the second image in a sequence of images, calculating a degree of similarity between the prediction vector and the hypothesis vector based on a difference between the prediction vector and the hypothesis vector, and evaluating a plausibility of the hypothesis vector based on the calculated degree of similarity, wherein the evaluation is carried outwhether the hypothesis vector describes an actual movement of the feature between the second image and the third image.

[0009] The device according to the invention for checking the plausibility of a flow vector hypothesis comprises a computing unit which is configured to determine a prediction vector associated with a feature based on a movement of the feature between a first image and a second image, wherein the second image is an image following the first image in a sequence of images, to generate a hypothesis vector associated with the feature, which describes a presumed movement of the feature between the second image and a third image, wherein the third image is an image following the second image in a sequence of images, to calculate a degree of similarity between the prediction vector and the hypothesis vector based on a difference between the prediction vector and the hypothesis vector, and to evaluate a plausibility of the hypothesis vector based on the calculated degree of similarity, wherein the evaluation is carried outwhether the hypothesis vector describes an actual movement of the feature between the second image and the third image.

[0010] The first, second, and third images are images in a sequence of images. The sequence of images is, for example, a video sequence. The first image preferably appears first, followed by the second image, and then the third image in chronological order within the sequence of images.

[0011] The feature is a feature depicted in the image. The depiction of the feature is limited, in particular, to a single pixel or a group of pixels. The movement of the feature between images reflects the movement of an object in the environment depicted by the image and / or the movement of a camera used to capture the images.

[0012] When determining the prediction vector associated with the feature, a movement of the feature between the first image and the second image is detected. This records a change in the feature's position between two images, and this is used to determine an expected future change in the feature's position. This movement can be described by a vector that connects a starting point of the movement in the first image with an end point of the movement in the second image. Especially in sequences of images captured by a camera, objects typically move continuously between individual images. This means that an object's movement does not change abruptly in direction or speed. The movement of the feature between the first image and the second image can therefore be estimated how the feature will move between the second and third images.This movement is described by the prediction vector. In the simplest case, the prediction vector is a motion vector that describes the movement of the feature between the first image and the second image. Further information is preferably incorporated into the determination of the prediction vector. For example, a camera movement can be detected, and this movement can be appropriately added to the motion vector to determine the prediction vector. The predictor thus describes an expected movement of the feature between the second image and the third image.

[0013] A suitable addition of the movement of the camera can be understood, for example, as meaning that the movement of the feature between the first image and the second image is first decomposed into a first part caused by the continuous movement of the object and a second part caused by the movement of the camera.

[0014] A prediction can be made separately for both parts.

[0015] For example, it can be assumed that the object will continue to move according to the previously observed movement, e.g. with unchanged speed in space or with unchanged acceleration in space. Measurements of the camera's movement may even be available, which come from other sensors (e.g. vehicle odometry and / or inertial sensors) or are obtained from the camera images, whereby the movement is determined from a plurality of vectors. The prediction of the movement of the feature between the second image and the third image can then be formed by combining, e.g. adding, the two components, i.e. a first component for the movement prediction of the object and a second component for the movement of the camera. The second component can be measured, which can lead to greater accuracy than a prediction by simple extrapolation.

[0016] When generating the hypothesis vector associated with the feature, a guess is made as to how the feature moved between the second image and the third image. This can be done, for example, by analyzing the third image to determine where features are located in the third image that are similar to the sought-after feature for which the hypothesis vector is being generated. For example, it can be analyzed where pixels with similar color properties are located in the second and third images. In particular, the requirement that each feature of an image should be assigned a maximum of one flow vector can be taken into account. A hypothesis vector is a flow vector that is regarded as a correct flow vector associated with the feature. However, there is a possibility that the analysis underlying the hypothesis vector is incorrect.Therefore, we speak of a hypothesis vector, since initially only one hypothesis is formulated.

[0017] When calculating the degree of similarity between the prediction vector and the hypothesis vector, link costs can be calculated, which describe the degree of similarity. High link costs describe a low degree of similarity, and low link costs describe a high degree of similarity. The degree of similarity thus provides an indicator that allows an assessment of whether a movement of the feature between the first and second images corresponds, within a tolerance, to the movement of the feature between the second and third images, if the feature moves according to the hypothesis vector. Assuming that the movement of an object does not change abruptly in direction or speed, it can be assumed that the hypothesis vector corresponds to the prediction vector within the tolerance.In this way, a high degree of similarity and potentially low linking costs result if the degree of similarity is calculated for a hypothesis vector that describes an actual movement of the feature between the second and third images, i.e., if the hypothesis vector is based on a correct hypothesis. If an incorrect hypothesis is formulated, for example, because a pixel or combination of pixels in the third image is mistakenly considered to be the feature for which the hypothesis vector is generated due to a color similarity, a low degree of similarity and potentially high linking costs will result, since the prediction vector defines where in the third image the feature under consideration should be depicted within the tolerance.

[0018] The difference between the prediction vector and the hypothesis vector is defined in particular by a difference in their lengths and / or their orientations. The length describes the speed of movement of the feature in the sequence of images. The orientation describes the trajectory of movement of the feature in the sequence of images. Thus, a difference based on the length and / or orientation of the vectors makes it particularly easy to describe the actual movement of an object associated with the feature with respect to a camera that captures the images. In particular, separate consideration of lengths and orientation is advantageous because their changes over the course of the sequence of images can be based on different assumptions.

[0019] When assessing the plausibility of the hypothesis vector, the plausibility is rated higher if the calculation of the degree of similarity results in a greater similarity between the prediction vector and the hypothesis vector than if the calculation of the degree of similarity results in a comparatively low similarity between the prediction vector and the hypothesis vector. Plausibility describes whether the flow vector hypothesis underlying the hypothesis vector can be considered correct. It is assumed that the flow vector hypothesis is correct, i.e., the hypothesis vector has a high degree of plausibility, if the degree of similarity is high. Accordingly, it is assumed that the flow vector hypothesis is false, i.e., the hypothesis vector has a low degree of plausibility, if the degree of similarity is low.

[0020] In the simplest case, plausibility is assessed solely based on the degree of similarity. However, a combination with other validation methods is also possible, allowing an alternative approach to assess plausibility. For example, plausibility could be rated high by the method according to the invention, but could be lowered in parallel by a competing assessment.

[0021] If several competing hypothesis vectors are available for the movement of a feature point, the hypothesis vector that results in the highest degree of similarity or the lowest linkage costs with respect to the given prediction vector can be selected, provided that these values ​​are below or below a threshold. Alternatively, it is possible to rank the hypothesis vectors based on the determined degree of similarity or the determined linkage costs, e.g., to determine the N most plausible hypothesis vectors.

[0022] The method and device according to the invention enable the robustness of the evaluation of a flow vector hypothesis to be increased. This makes it possible to generate a flow field with particularly high reliability.

[0023] The device according to the invention is designed to carry out the method according to the invention.

[0024] The subclaims show preferred developments of the invention.

[0025] It is advantageous if the method is carried out in a loop, wherein in a current loop iteration, the hypothesis vector of the feature is assigned information that is based on information assigned to the hypothesis vector of the feature in a previous loop iteration. In this way, in addition to a movement of the feature in the sequence of images described by the hypothesis vector, further information can also be transported across the sequence of images. The information of an image linked to a hypothesis vector can be used in particular to evaluate the plausibility of the hypothesis vector of a subsequent image. Information that describes properties of an object depicted by the feature can also be assigned to a feature.In particular, it is advantageous if the hypothesis vector, which was classified as plausible in a first loop run, is transferred into a flow field as a flow vector together with the linked information and, in a subsequent second loop run, the hypothesis vector is read out from the flow field together with the linked information as a prediction vector of the feature for the second loop run.

[0026] Furthermore, it is advantageous if the information associated with the hypothesis vector is a result of a linkage check, wherein the linkage check is carried out in each loop iteration, wherein the prediction vector and the hypothesis vector of the feature are considered to be linked to one another in the linkage check if the degree of similarity between the prediction vector and the hypothesis vector exceeds a predetermined level, and in each loop iteration, the plausibility of the hypothesis vector is further assessed based on whether the prediction vector and the hypothesis vector of the feature were considered to be linked to one another in one or more previous loop iterations.Because the plausibility of the hypothesis vector is further assessed based on whether the prediction vector and the hypothesis vector of the feature were considered to be linked in one or more previous loop iterations, a history of the hypothesis vector is considered. This history of the hypothesis vector is also referred to as the age of the hypothesis vector. A hypothesis vector is considered to be very old if the prediction vector and the hypothesis vector of the feature have already been considered to be linked multiple times in consecutive loop iterations. In particular, a number of consecutive loop iterations in which the prediction vector and the hypothesis vector of the feature were considered to be linked can be assigned to the hypothesis vector as information, an age attribute.By considering the history of a hypothesis vector, the plausibility can be determined with high reliability. This additionally incorporates an evaluation of the movements of a feature, which results from several previous images in the sequence of images, into the plausibility assessment. Thus, the hypothesis vector is only considered particularly plausible if the corresponding feature for which the hypothesis was created consistently exhibits low linkage costs in successive loop iterations. This provides a historical evaluation of the hypothesis that is particularly reliable.

[0027] It is also advantageous if the information assigned to the hypothesis vector describes an object class of an object belonging to the feature. In particular, information is linked to the feature which indicates that the feature belonging to the hypothesis vector represents an object of a specific object class. For example, a probability value is linked to the feature which describes a probability that the feature represents an object of a specific object class. The probability value is preferably assigned to the feature in a flow field. The object class describes in particular what type of object is represented by the feature. In this case, an object class particularly groups together objects which have similar movement properties within the framework of predefined specifications.In particular, the object class is used to evaluate the plausibility of the hypothesis vector, for example, to define a degree of similarity between the prediction vector and the hypothesis vector, which must be present at least for the plausibility of the hypothesis vector to be positively assessed. In particular, a threshold for the degree of similarity is set to a first value if the object class assigned to the feature is a first object class, which, for example, includes objects of the type "person."

[0028] Furthermore, the threshold for the degree of similarity is set to a second value if the object class assigned to the feature is a second object class, which, for example, includes objects of the type "vehicle." The first value is lower than the second value. The higher plausibility rating only occurs if the threshold is not exceeded. In this way, for the example object classes "person" and "vehicle," for example, it is taken into account that a higher continuity in the movement of vehicles in the sequence of images is to be expected than for people.

[0029] When calculating the degree of similarity between the prediction vector and the hypothesis vector, a deviation in the orthogonal direction relative to the prediction vector is evaluated more strictly than a deviation in the longitudinal direction relative to the prediction vector, or a deviation in the orthogonal direction relative to the hypothesis vector is evaluated more strictly than a deviation in the longitudinal direction relative to the hypothesis vector. A stricter evaluation means a greater influence on the calculated degree of similarity.

[0030] This takes into account that a 3D point of an object moving in equal steps along a straight line generally does not move in equal steps after being projected into an image. Rather, the steps in the image are larger when the point is closer to the camera and smaller when the point is farther from the camera.

[0031] Furthermore, it is advantageous if the prediction vector is determined from a flow field that describes a backward flow and / or the hypothesis vector is determined from a flow field that describes a forward flow. When calculating the prediction vector and / or the hypothesis vector in this way, the prediction vector (as the backward flow) ends at the image position of the second image at which the hypothesis vector (as the forward flow) begins, whereby an association between the prediction vector and the hypothesis vector is already implicitly given and does not have to be determined first. Thus, such a calculation of a backward flow and a forward flow takes place as part of generating a flow field, since this is required for evaluating the flow field. Thus, the additional plausibility check of one or more flow vector hypotheses can be carried out with little additional computational effort based on available data.

[0032] It is also advantageous if, when calculating the degree of similarity, the prediction vector is adjusted to compensate for a camera movement between capturing the first image and the second image and / or a camera movement between capturing the second image and the third image. This prevents a change in the position of the feature in the images that does not result from the inherent movement of the associated object from being incorporated into the calculation of the degree of similarity and distorting it.

[0033] It is also advantageous if the degree of similarity is determined using a predefined function from the difference between the prediction vector and the hypothesis vector, wherein the predefined function in particular weights the difference between the prediction vector and the hypothesis vector. In this way, the degree of similarity between the prediction vector and the hypothesis vector can be calculated geometrically and converted into a simple numerical value for the degree of similarity. The difference is weighted in such a way that a small difference leads to a particularly high degree of similarity and possibly to particularly low linking costs, and a large difference leads to a particularly low degree of similarity and possibly to particularly high linking costs. The weighting is carried out in particular in a non-linear manner with respect to the degree of similarity.

[0034] According to the invention, when calculating the degree of similarity, the difference is determined using a difference vector, which is generated by forming a difference between the prediction vector and the hypothesis vector. Such a difference vector is an easily calculated parameter that describes the similarity between the prediction vector and the hypothesis vector. Preferably, the prediction vector and the hypothesis vector are each decomposed into a first component caused by the movement of the object and a second component caused by the movement of the camera. Thus, it is advantageous to determine the difference vector based only on the components caused by the movement of the object.

[0035] According to the invention, when calculating the degree of similarity, the difference is determined using two vector components of the difference vector, with the two vector components being weighted differently. This results in a different evaluation of the difference, which depends on the direction of a deviation between the prediction vector and the hypothesis vector. In particular, a deviation that is orthogonal to an orientation of the hypothesis vector, or alternatively orthogonal to an orientation of the prediction vector, is weighted more heavily and influences the degree of similarity than a deviation that is longitudinal to an orientation of the hypothesis vector, or alternatively longitudinal to an orientation of the prediction vector.This allows, for example, deviations in the longitudinal direction to be evaluated differently, in particular less strictly, than deviations in the orthogonal direction. This allows, for example, the fact that a 3D point moving in equal steps along a straight line does not generally move in equal steps within the image after projection into the image can be taken into account. Rather, the steps in the image are larger when the point is closer to the camera and smaller when the point is farther away.

[0036] Furthermore, it is advantageous if, when generating a hypothesis vector associated with the feature, several hypothesis vectors associated with the feature are generated, and the degree of similarity is calculated and the plausibility is assessed for each of the hypothesis vectors. This allows an advantageous selection of a preferred hypothesis vector. In particular, one of the hypothesis vectors is selected based on plausibility. Thus, the hypothesis vector with the highest plausibility is preferably regarded as a hypothesis vector that describes an actual movement of the feature between the second image and the third image. However, it is also possible for all of the hypothesis vectors to be rejected if their plausibility is below a predetermined threshold, for example, because the associated degree of similarity is too low. Short description of the drawings

[0037] Embodiments of the invention are described in detail below with reference to the accompanying drawings. In the drawing: Figure 1 a representation of an exemplary sequence of images, Figure 2 a representation of an exemplary prediction vector and an exemplary hypothesis vector, Figure 3 a representation of prediction vectors and hypothesis vectors, which lead to different linkage costs, Figure 4 an example function for weighting link costs, and Figure 5 a device for verifying the plausibility of a flow vector hypothesis. Embodiments of the invention

[0038] Fig. 1shows an exemplary sequence of images 20. The sequence of images 20 comprises a first image 1, a second image 2, and a third image 3. In the exemplary sequence of images 20 shown, the second image 2 immediately follows the first image 1, and the third image 3 immediately follows the second image 2. However, further images in the sequence of images 20 can also be located between the individual images. The sequence of images 20 is a video sequence captured by a camera 41.

[0039] An object was captured by camera 41, which is shown in images 1, 2, 3 as a feature 4 of the respective image 1, 2, 3. Thus, feature 4 moves in the sequence of images 20 shown from bottom left to top right through the image shown by the video sequence. Figure 1The feature is located only at the position in the respective image 1, 2, 3 where it is represented by a solid line. The dashed lines show a position of feature 4 in the remaining images. Thus, in the first image 1, feature 4 is located at a first position 5. In the second image 2, feature 4 is located at a second position 6. In the third image 3, feature 4 is located at a third position 7. Feature 4 is particularly limited to a small number of pixels.

[0040] Furthermore, Figure 1 Two flow vectors 10, 11 associated with feature 4 are shown. A first flow vector 10 describes a movement of feature 4 between the first image 1 and the second image 2. A second flow vector 11 describes a movement of feature 4 between the second image 2 and the third image 3.

[0041] The second flow vector 11 was determined by means of image analysis from the second image 2 and the third image 3. This is done by generating or establishing a flow vector hypothesis. Using the flow vector hypothesis, a hypothesis vector 13 is generated. The second flow vector 11 thus corresponds to the hypothesis vector 13. This can sometimes lead to incorrect results during image analysis. The problem with image analysis is that the feature 4 shown in the third image 3 could belong to a different object than the feature 4 shown in the second image 2. In this case, a hypothesis vector 13 would have been determined, but this is not a correct flow vector because it does not describe the movement of feature 4 in the sequence of images 20, but rather connects positions of different features. It is therefore not certain whether the second flow vector 11 is a correct flow vector.Therefore, we speak of a flow vector hypothesis. The flow vector calculated when generating the flow vector hypothesis is therefore referred to as hypothesis vector 13. Since it is not conclusively determined whether this is a correct flow vector, hypothesis vector 13 describes a presumed movement of feature 4 between the second image 2 and a third image 3.

[0042] A distinction is made between forward flow and backward flow, which expresses the image in which a reference pixel is located and the image in which a correspondence was sought (search direction). If the result is provided as a flow field (in a matrix structure), the search direction usually also determines the coordinate system in which a flow field is stored. A flow field is an array of flow vectors in which all, or at least all calculated, flow vectors resulting from a pair of two images are entered. Additionally, attributes can also be assigned to each flow vector.

[0043] For a forward flow, the reference pixel is searched for in a previous image and the corresponding pixel in a current image. The result is stored in the coordinates of the previous image. For a backward flow, the correspondence for the reference pixel in the current image is searched for in the previous image. The result is stored in the coordinates of the current image. In both cases, the signs of the flow vector components of the respective flow vectors 10, 11 are set identically, namely such that a horizontal component is positive when moving to the right in the image, and a vertical component is positive when moving downwards in the image.

[0044] Confidence in the flow vector hypothesis, and thus the plausibility of hypothesis vector 13, increases if hypothesis vector 13 already had a predecessor in the previous time step that "matches" the considered flow vector hypothesis in terms of direction and magnitude. This is based on the assumption that the object associated with feature 4 moves "inertly" due to its mass, and that camera 41 also moves inertly because it is rigidly connected to a mass-bearing object (e.g., a vehicle). Sudden changes in direction therefore practically do not occur in the image plane. A sufficiently high temporal sampling rate of camera 41 is assumed.

[0045] Therefore, a predictor is determined which describes a movement of the feature 4 between the first image 1 and the second image 2 by means of a prediction vector 12, wherein the second image 2 is an image following the first image 1 in the sequence of images 20. The prediction vector 12 therefore corresponds to the Figure 1 shown example the first flow vector 12.

[0046] The first to third images 1, 2, 3 were recorded by camera 41. The first image 1 was recorded at a first time t 1 , the second image 2 at a second time t 2 and the third image 3 at a third time t 3 . Images 1, 2, 3 were recorded consecutively with t 1 < t 2 < t 3 , but not necessarily immediately after one another (therefore, further images may have been recorded in between). A constant time interval between images 1, 2, 3 is not required. This means that the quotient q q = t 3 − t 2 t 2 − t 1

[0047] in general can also be unequal to 1.

[0048] For example, an existing optical flow field between times t 1 and t 2 can be used as a prediction. This is preferably present as a backward flow, as this has the advantage that the prediction vectors 12 are already present in the correct coordinate system from the common time t 2, thus no longer needing to be converted (warped). Thus, the prediction vector 12 is determined from a flow field that describes a backward flow.

[0049] The flow field used for prediction can also be generated in other ways, e.g., using a three-dimensional scene model and assumptions about the camera positions at the considered times, from which the expected flow field can then be calculated.

[0050] The following example assumes the simplest case, namely that no special effort is required to provide the prediction, and the prediction consists of or is generated from the old flow field. The prediction flow field and thus the flow vectors defined in the prediction flow field are optionally scaled by a factor to bring the flow vectors, and thus the prediction vector 12, to the expected lengths. It is advantageous if the factor for scaling the flow vectors is chosen equal to the quotient q.

[0051] This prediction supports the determination of the next flow field between the times t 2 and t 3 . The time t 2 is therefore common to both flow fields: the flow vectors of the first flow field end there, and those of the second start there. Therefore, it is further advantageous if the time to be determined

[0052] The flow field between t 2 and t 3 is initially a forward flow. This series of images can be continued by adding another image associated with time t 4 with t 4 > t 3 .

[0053] The flow field between t 2 and t 3 can then be used as a predictor and support the next determination of the flow field between times t 3 and t 4 (common time t 3 ), and so on. The corresponding prediction vectors 12 are read from the flow field between t 2 and t 3 .

[0054] For each flow vector hypothesis, and thus for each hypothesis vector 13, a linkage check is performed, whereby the hypothesis, i.e., hypothesis vector 13, is compared with the respective prediction, i.e., prediction vector 12, in terms of orientation and length. Linkage costs are thus calculated based on a difference between prediction vector 12 and hypothesis vector 13, with the linkage costs indicating a degree of similarity between prediction vector 12 and hypothesis vector 13. High linkage costs indicate a low degree of similarity, and low linkage costs indicate a high degree of similarity. Thus, a degree of similarity between prediction vector 12 and hypothesis vector 13 is calculated based on a difference between prediction vector 12 and hypothesis vector 13.

[0055] When calculating the linkage costs, the prediction vector 12 is optionally adjusted, which compensates for a camera movement between capturing the first image 1 and the second image 2 and / or compensates for a camera movement between capturing the second image 2 and the third image 3. For this purpose, for example, the change in camera movement is detected using a sensor or image analysis, and the resulting change in movement at the respective pixel is determined and added to the first flow vector 10 to determine the prediction vector 12.

[0056] Prediction vector 12 and hypothesis vector 13 are considered "linked" or "chained" because, in other words, they fit together. This property increases the confidence in the thus confirmed flow vector hypothesis and can be attached to the respective flow vector as a quality measure or quality attribute. This increases the plausibility of the flow vector hypothesis. During the linkage test, prediction vector 12 and hypothesis vector 13 of feature 4 are considered linked if the degree of similarity between prediction vector 12 and hypothesis vector 13 exceeds a predetermined level, i.e., if the linking costs are below a predetermined level. If prediction vector 12 and hypothesis vector 13 are considered linked, this information is assigned to hypothesis vector 13.

[0057] If the hypothesis vector 13 is used later in the process, for example, when generating a prediction vector 12 associated with feature 4 based on a movement of feature 4 between the second image 2 and the third image 3, this information can be used to evaluate the plausibility later in the process. In this case, the plausibility of the hypothesis vector 13 is evaluated based on whether the prediction vector 12 and the hypothesis vector 13 of feature 4 were considered linked in one or more previous loop iterations.

[0058] If this connection check is successful multiple times in succession, the confirmation extends back correspondingly far into the past. This is referred to as the age of the flow vector or hypothesis vector 13, which refers to the number of consecutively successful connection checks. The age thus formed, in turn, represents a quality attribute that can be represented as a number and attached to the respective flow vector.

[0059] The method is executed in a loop, which enables the aging of flow vectors associated with a feature 4. The hypothesis vector 13 and the prediction vector 12 for feature 4 are considered to be linked in a current loop iteration if the calculation of the linking costs indicates a degree of similarity between the prediction vector 12 and the hypothesis vector 13 that is greater than a predetermined degree and / or there is no alternative hypothesis vector for which an even higher degree of similarity exists. The plausibility of a flow vector hypothesis of a loop iteration following the current loop iteration is further increased if the hypothesis vector 13 and the prediction vector 12 for feature 4 are already considered to be linked in the current loop iteration.

[0060] One could therefore also speak of a concatenation (even if it is not explicitly stored as such). The age thus formed can be used as a quality criterion to filter out good flow vectors or eliminate outliers. Most outliers can be eliminated with a threshold of "age ≥ 1." With a higher threshold, the separation becomes even better; however, the higher the threshold, the higher the waiting time (in terms of the number of images) until a new hypothesis vector 13 (without predecessors) can reach the threshold for the first time.

[0061] Furthermore, a useful option arises to transport further information from image to image using the flow field if the linkage check for the respective feature 4 was successful. For example, the second image 2 from time t 2 may already be provided with label information, e.g., a semantic segmentation that indicates for each feature 4 the probability that the respective pixel belongs to a specific object class (e.g., pedestrian, car, bicycle, building, street, curb, sidewalk, guardrail, traffic sign, vegetation, etc.).

[0062] This information can be forwarded using the corresponding hypothesis vector 13 (from t 2 to t 3 ). This ensures that this information arrives at the correct coordinate. Thus, information is linked to feature 4, which indicates that the feature 4 associated with the flow vector hypothesis represents an object of a specific object class.

[0063] This passing of information or an attribute is particularly useful if the linkage check was successful, because then the behavior is temporally consistent and physically plausible. In this case, the probability is high that the passing was correct and that the additional information will subsequently be correctly assigned.

[0064] A preferred option for link testing and calculating the degree of similarity is described below. The link costs for a link are determined, with the link costs consisting of an orthogonal component and a longitudinal component. The difference underlying the link costs is determined using two vector components 22, 23 of a difference vector 21. A first vector component 22 describes the orthogonal component, and a second vector component 22 describes the longitudinal component. By definition, the vector components 22, 23 are perpendicular to each other.

[0065] The relationship between prediction vector 12 and hypothesis vector 13 is in Figure 2 shown as an example. For example, in Figure 2the first flow vector 10 is shown. This describes the movement of feature 4 between the first position 5 and the second position 6. This movement of feature 4 occurs between the first image 1 and the second image 2. From this movement between the first image 1 and the second image 2 and thus based on the first flow vector 10, the prediction vector 12 is determined. The prediction vector 12 is equal to the first flow vector 10. This is based on the assumption that feature 4, with a constant movement in the third image 3, should be at a theoretical position 8.

[0066] In the Figure 2 In the example shown, hypothesis vector 13 is further calculated through image analysis. Thus, the flow vector hypothesis is established that feature 4 moved to the third position 7 between the second image 2 and the third image 3. This is described by hypothesis vector 13.

[0067] If a flow field already exists, the hypothesis vector is determined from the flow field. The flow field preferably describes a forward flow. To evaluate whether this flow vector hypothesis, and thus the hypothesis vector 13, is plausible, the link costs are now calculated. For this purpose, the difference vector 21 is calculated from the difference between the hypothesis vector 13 and the prediction vector 12. In a simple case, the difference is equal to the absolute value of the difference vector 21, whereby a value of this absolute value is directly considered a value for the link costs.

[0068] However, in the embodiment described here, the difference underlying the linkage costs is determined using the vector components 22, 23 of the difference vector 21. The orthogonal component, i.e. the first vector component 22 perpendicular to the hypothesis vector 13, represents a measure of the deviation from an orientation, with the orientation of the hypothesis vector 13 being used as the reference orientation. Alternatively, the orientation of the prediction vector 12 could also serve as a reference. The longitudinal component, i.e. the second vector component 23 and thus the second vector component 23 parallel to the hypothesis vector 13, measures the deviation in the direction of the reference vector or in the opposite direction. In other words, the lengthening or shortening of the hypothesis vector 13 compared to the prediction vector 12 is measured, with different scales generally being applied for these two cases.

[0069] If the reference vector corresponds to the zero vector, no orientation is defined. Special treatment is appropriate for this case.

[0070] The orthogonal and longitudinal components of the link costs, both of which are non-negative, are then combined, e.g., in the sense of a norm. The components can also be weighted.

[0071] However, in this embodiment, a stricter standard should be applied to the evaluation of the orthogonal component than for the longitudinal component, because the orthogonal deviation can only be explained by a change in the object's direction or a rotation of the camera, whereas the longitudinal component also changes simply by the object approaching or moving away from the camera. Therefore, a longitudinal change within a given interval can be evaluated generously, e.g., by not incurring any linkage costs. This results in a different weighting of the two vector components 22, 23.

[0072] The linkage costs are a measure of the difference between the prediction vector 12 and the hypothesis vector 13. Example combinations of the prediction vector 12 and the hypothesis vector 13 are shown in Figure 3Thus, the linkage costs are zero if the prediction vector 12 and the hypothesis vector 13 are identical. In Figure 3 On the far left is an example in which the prediction vector 12 and the hypothesis vector 13 are similar and low linkage costs arise. In Figure 3 In the middle, an example is shown in which the prediction vector 12 and the hypothesis vector 13 are dissimilar to each other and high linkage costs arise. Furthermore, Figure 3 On the right, an example is shown in which there is no prediction vector 12 and therefore no linkage costs can be calculated.

[0073] To convert a determined difference between the prediction vector 12 and the hypothesis vector 13 into the linkage costs, the linkage costs are determined from the difference between the prediction vector and the hypothesis vector using a predefined function. The predefined function can be used to weight the difference between the prediction vector and the hypothesis vector. In particular, the function is a function that depends on the magnitude of the first vector component 22 and the magnitude of the second vector component 23.

[0074] The link costs can then be mapped to convert them into a more usable form, for example, with a non-linear function. Such a non-linear function is shown, for example, in Figure 4This is piecewise linear and maps non-negative link costs to signed values, which are subsequently called weights. The function can be continuous or discrete. The idea here is that "good flow vector hypotheses," i.e., those with low link costs, are enhanced by adding a positive weight, while flow vector hypotheses with high link costs are downweighted. The link costs are represented by a first axis 30, and the weighting is represented by a second axis 31. At a neutral point 32, the weighting transitions from a positive weight to a negative weight.

[0075] In Figure 4The two intercepts have different slopes and intersect at a single point. Here, the intersection point lies on the abscissa. This intersection point is called neutral point 32 and represents a special point: At the neutral point, the additive weight is exactly zero, and thus as large as for a flow vector hypothesis for which no prediction exists. With this concept of signed additive weights, it is therefore easily possible to account for the fact, important in practice, that no prediction exists for some, many, or sometimes all, flow vector hypotheses.

[0076] The additive weight can be added to an existing weight that the flow vector hypothesis may already have been assigned for reasons not described in detail here. Other approaches to considering weights instead of adding them are also possible.

[0077] The link costs or mappings thereof (e.g., after nonlinear mapping) can be compared with one or more thresholds to map the range of values ​​to a small number of replacement values. This step of linear or nonlinear quantization can produce a binary or multi-binary result.

[0078] In the case of a binary decision, this can directly correspond to rejecting or accepting a flow vector hypothesis. Thus, the plausibility of the flow vector hypothesis and the corresponding hypothesis vector 13 is assessed based on the calculated linkage costs and thus based on the calculated degree of similarity.

[0079] If more than 2 replacement values ​​are available, for example 4, they can have the following interpretation: 0: Prediction and flow vector hypothesis contradict each other; the flow vector hypothesis and thus hypothesis vector 13 are rejected. 1: Prediction and flow vector hypothesis do not agree well, but are still within the tolerance range. Caution is warranted when further processing the flow vector hypothesis and hypothesis vector 13. 2: Prediction and flow vector hypothesis agree well; the flow vector hypothesis and thus hypothesis vector 13 can continue to be used normally. 3: Prediction and flow vector hypothesis agree particularly well or are even identical; the flow vector hypothesis and thus hypothesis vector 13 can be used for special purposes.

[0080] The previously described values ​​1 to 4 describe the plausibility of the flow vector hypothesis and thus of the corresponding hypothesis vector 13. It is evident that the plausibility is rated higher when the linking costs indicate a first degree of similarity than when the linking costs indicate a second degree of similarity, with the first degree of similarity being greater than the second degree of similarity. Thus, the plausibility is rated higher when the prediction and the flow vector hypothesis do not contradict each other. The plausibility is rated lower when the prediction and the flow vector hypothesis contradict each other. Whether the prediction and the flow vector hypothesis contradict each other is determined by the linking costs.

[0081] Typically, when generating the hypothesis vector 13 associated with feature 4, several hypothesis vectors associated with feature 4 are generated. In this case, the degree of similarity and the plausibility assessment are carried out for each of the hypothesis vectors 13. Thus, for each of the hypothesis vectors 13 associated with feature 4, linkage costs are calculated, which describe the degree of similarity of the respective hypothesis vector 13 with the prediction vector 12. When evaluating the plausibility of the individual hypothesis vectors 13, each of the hypothesis vectors 13 is assigned one of the previously described values ​​1 to 4. In this case, the hypothesis vector 13 associated with feature 4 that has been assigned the highest value is preferably selected for further processing, for example, incorporated into a flow field.

[0082] Generally speaking, the hypothesis vector 13 associated with feature 4 that exhibits the highest plausibility when assessed is selected for further processing. Thus, the plausibility for each hypothesis vector 13 is increased or decreased, starting from a predetermined initial value, depending on the linking costs and thus on the associated degree of similarity. Plausibility is increased when the degree of similarity is high and the linking costs are low. Correspondingly, plausibility is decreased when the degree of similarity is low and the linking costs are high. The degree of change in plausibility can, for example, depend on the linking costs.Once the plausibility has been assessed, the hypothesis vector 13 associated with feature 4 is selected for further processing, for example, incorporated into a flow field, which has the highest plausibility after the plausibility assessment.

[0083] The result of comparing the flow vector hypothesis with the prediction can be a quality measure that can be encoded as additional information (here with 2 bits) and stored, passed on or further processed.

[0084] A temporal aggregation of such quality measures is particularly advantageous. For example, the quality measure determined during the linkage test is "calculated"—for example, added—with an attribute of the flow vector for each flow vector. Such an attribute could be the aforementioned age: In this case, the quality measure would be binary, and the addition could continue until a maximum representable age is reached. Aggregating signals with more than binary values ​​over time is also useful. However, this attribute would then no longer be directly interpretable as age.

[0085] According to the invention, robustness against outliers is thus significantly improved. Outliers in optical flow are incorrect correspondence assignments, which can occur, for example, when structures repeat in the scene.

[0086] The inventive comparison of flow vector hypotheses for prediction favors flow vector hypotheses that behave as expected and effectively suppresses outlier hypotheses that do not. Furthermore, a measure of temporal consistency is obtained as additional information and propagated temporally using the flow vectors.

[0087] This transport route can also be used for other information.

[0088] The idea of ​​flow prediction for evaluating flow vector hypotheses and for transporting attributes can be used both for sparse optical flow, where motion is detected only for a few distinct pixels, and for dense optical flow, where motion is detected for each pixel or a large proportion of the pixels.

[0089] An example of such an attribute is the "age" of a flow vector. This can be, for example, the number of consecutive successful confirmations based on the comparison with the respective prediction and thus represents a measure of quality or confidence.

[0090] Thus, optical flow prediction is used to verify or de-validate flow vector hypotheses, e.g., to eliminate false correspondences (outliers) or to enhance correspondences confirmed by the prediction or to downgrade correspondences not confirmed by the prediction.

[0091] In addition, a linkage test is carried out between the prediction vector and the hypothesis vector, generating an evaluation.

[0092] The linkage test is broken down into a test for deviation from an orientation and a test for deviation along the orientation. This is done, if necessary, with a weighted combination of the components.

[0093] Deviations along the orientation within a predetermined interval can be allowed. Different treatment occurs depending on whether the hypothesis vector 13 is longer or shorter (along the orientation) than the prediction vector 12.

[0094] Prediction is preferably performed as a backward flow in combination with flow hypotheses as a forward flow. Furthermore, a measure of the temporal consistency of a flow vector (in connection with the history input via the prediction) is created, e.g., as an age attribute. Attributes generated, confirmed, or updated by the linkage check are also made available at an interface, and any attribute is transported over time, e.g., attached to a flow vector, depending on a linkage check.

[0095] Preferably, parameters are used to adapt the prediction, e.g. to take into account different time intervals or route sections or to compensate for the camera's own movement.

[0096] In Figure 5A device for verifying the plausibility of a flow vector hypothesis is shown. This device comprises a computing unit 40 configured to carry out the method described above. Images 1, 2, 3 are provided by a camera 41. However, it should be noted that images 1, 2, 3 can also be provided by an alternative source, for example, by a storage medium.

[0097] In addition to the above written revelation, explicit reference is made to the revelation of Figures 1 to 4 referred to.

Claims

1. Computer-implemented method for checking the plausibility of a flow-vector hypothesis, comprising: - determining a prediction vector (12), associated with a feature (4), based on a movement of the feature (4) between a first image (1) and a second image (2), wherein the second image (2) is an image following the first image (1) in a sequence of images (20), - generating a hypothesis vector (13), which is associated with the feature (4) and describes a probable movement of the feature (4) between the second image (2) and a third image (3), wherein the third image (3) is an image following the second image (2) in a sequence of images (20), - calculating a degree of similarity between the prediction vector (12) and the hypothesis vector (13) based on a difference between the prediction vector (12) and the hypothesis vector (13), wherein, when calculating the degree of similarity between the prediction vector (12) and the hypothesis vector (13), a deviation in the orthogonal direction with respect to the prediction vector (12) is evaluated more strictly than a deviation in the longitudinal direction with respect to the prediction vector (12), or a deviation in the orthogonal direction with respect to the hypothesis vector (13) is evaluated more strictly than a deviation in the longitudinal direction with respect to the hypothesis vector (13), and - evaluating a plausibility of the hypothesis vector (13) based on the calculated degree of similarity, wherein it is evaluated whether the hypothesis vector (13) describes an actual movement of the feature (4) between the second image (2) and the third image (3), characterized in that - when calculating the degree of similarity, the difference is determined by means of a difference vector (21) which is generated by forming a difference between the prediction vector (12) and the hypothesis vector (13), wherein the difference is determined by means of two vector components (22, 23) of the difference vector (21), which represent a measure of the deviation in the orthogonal direction and a measure of the deviation in the longitudinal direction, wherein the two vector components (22, 23) are differently weighted.

2. Method according to Claim 1, characterized in that the method is performed in a loop, wherein, in a current pass through the loop, the hypothesis vector (13) of the feature (4) is assigned information which is based on information that was assigned to the hypothesis vector (13) of the feature (4) in a previous pass through the loop.

3. Method according to Claim 2, characterized in that the information assigned to the hypothesis vector (13) is a result of a linkage check, wherein - the linkage check is performed in each pass through the loop, wherein the prediction vector (12) and the hypothesis vector (13) of the feature (4) are considered to be linked to one another in the linkage check if the degree of similarity between the prediction vector (12) and the hypothesis vector (13) exceeds a specified amount, and - in each pass through the loop, the evaluation of the plausibility of the hypothesis vector (13) is also based on whether the prediction vector (12) and the hypothesis vector (13) of the feature (4) were considered to be linked to one another in one or more previous passes through the loop.

4. Method according to Claim 2, characterized in that the information assigned to the hypothesis vector (13) describes an object class of an object associated with the feature (4).

5. Method according to one of the preceding claims, characterized in that the prediction vector (12) is determined from a flow field which describes a backward flow and / or the hypothesis vector (13) is determined from a flow field which describes a forward flow.

6. Method according to one of the preceding claims, characterized in that, when calculating the degree of similarity, the prediction vector (12) is adapted to compensate for a camera motion between a recording of the first image (1) and the second image (2) and / or to compensate for a camera motion between a recording of the second image (2) and the third image (3).

7. Method according to one of the preceding claims, characterized in that the degree of similarity is determined from the difference between the prediction vector (12) and the hypothesis vector (13) by means of a specified function, wherein the specified function is used in particular for weighting the difference between the prediction vector (12) and the hypothesis vector (13).

8. Method according to one of the preceding claims, characterized in that, when generating a hypothesis vector (13) associated with the feature (4), a number of hypothesis vectors associated with the feature (4) are generated and the degree of similarity is calculated for each of the hypothesis vectors (13) and the plausibility is evaluated for each of the hypothesis vectors (13).

9. Device for checking the plausibility of a flow-vector hypothesis, comprising a computing unit (40), which is configured so as: - to determine a prediction vector (12), associated with a feature (4), based on a movement of the feature (4) between a first image (1) and a second image (2), wherein the second image (2) is an image following the first image (1) in a sequence of images (20), - to generate a hypothesis vector (13), which is associated with the feature (4) and describes a probable movement of the feature (4) between the second image (2) and a third image (3), wherein the third image (3) is an image following the second image (2) in a sequence of images (20), - to calculate a degree of similarity between the prediction vector (12) and the hypothesis vector (13) based on a difference between the prediction vector (12) and the hypothesis vector (13), wherein, when calculating the degree of similarity between the prediction vector (12) and the hypothesis vector (13), a deviation in the orthogonal direction with respect to the prediction vector (12) is evaluated more strictly than a deviation in the longitudinal direction with respect to the prediction vector (12), or a deviation in the orthogonal direction with respect to the hypothesis vector (13) is evaluated more strictly than a deviation in the longitudinal direction with respect to the hypothesis vector (13), and - to evaluate a plausibility of the hypothesis vector (13) based on the calculated degree of similarity, wherein it is evaluated whether the hypothesis vector (13) describes an actual movement of the feature (4) between the second image (2) and the third image (3), characterized in that the computing unit (40) is configured so as, when calculating the degree of similarity, to determine the difference by means of a difference vector (21) which is generated by forming a difference between the prediction vector (12) and the hypothesis vector (13), wherein the difference is determined by means of two vector components (22, 23) of the difference vector (21), which represent a measure of the deviation in the orthogonal direction and a measure of the deviation in the longitudinal direction, wherein the two vector components (22, 23) are differently weighted.