DETECTION AND VALIDATION OF OBJECTS FROM SEQUENTIAL CAMERA IMAGES USING HOMOGRAPHS

DE502017016551D1Active Publication Date: 2024-11-28AUMOVIO AUTONOMOUS MOBILITY GERMANY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502017016551
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2016-09-29
Filing Date
2017-09-28
Publication Date
2024-11-28
Estimated Expiration
2037-09-28

AI Technical Summary

Technical Problem

Current vehicle detection systems based on classification methods struggle with new vehicle designs and changing structures, leading to reduced performance, especially in recognizing objects near the epipole, where triangulation is imprecise and relative speed is unknown, causing errors in detecting collisions with stationary or moving vehicles.

Method used

A method that records a sequence of images from a monocular vehicle camera, determines corresponding features, assigns them to levels in space based on orientations (floor, rear wall, side wall planes), calculates homographies to determine movement, and detects dynamic objects by analyzing the movement of these levels, allowing for accurate segmentation and differentiation between static and dynamic objects.

Benefits of technology

This approach enables accurate detection of dynamic objects, improves system performance by segmenting objects based on speed, scaling, and deformation, and provides real-time object recognition, even near the epipole, by compensating for camera rotation and determining absolute velocities, thus enhancing collision detection and free space analysis.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for detecting objects from images of a camera and for tracking them and can be used in particular in camera-based driver assistance systems.

[0002] State-of-the-art vehicle recognition systems are mostly classification-based. Classification-based systems can recognize vehicles or vehicle components that they have seen in their training data. However, new vehicle designs and changing bodywork can lead to significantly reduced system performance and require generic approaches to object recognition.

[0003] US 2014 / 0161323 A1 shows a method for generating dense three-dimensional structures in a road environment from images taken with a mono camera.

[0004] EP 2 993 654 A1 describes a method for forward collision warning (FCW) based on camera images. This method analyzes an image section to determine where the host vehicle will travel within a specified time interval. If an object is detected there, a collision warning is issued.

[0005] In "Stereo-Based Vision System for Automotive Imminent Collision Detection," Intelligent Vehicles Symposium, 2004 IEEE Parma, Italy, June 14-17, Piscataway, NJ, USA, IEEE, pp. 274-279, DOI: 10.1109 / IVS.2004.1226394, P. Chang et al. demonstrate a method for detecting an impending collision based on image data from a stereo camera. Using image segmentation based on the depth images, a collision object can be detected and its speed determined.

[0006] P. Singhal et al., in "Top Down Approach to Detect Multiple Planes in Images", Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing, ICVGIP '14, pp. 1-8, New York, New York, USA, DOI: 10.1145 / 2683483.2683536, present a method for detecting multiple planes in images during 3D reconstruction.

[0007] US 2015 / 086080 A1 shows a driver assistance system with a camera and a detection of a vertical deviation in the road contour while driving. First, second, and third images of the road are captured with the camera. By matching pixels of the road in the first image with corresponding pixels of the road in the second image, a first homography is calculated that converts the first image of the road into the second image. A second homography is calculated that converts the second image of the road into the third image of the road. A concatenated homography is calculated by concatenating the first and second homographies. The concatenated homography is used as an initial estimate to calculate a third homography that converts the first image of the road into the third image of the road.

[0008] J. Prankl et al., in "Interactive object modeling based on piecewise planar surface patches," Computer Vision and Image Understanding, Vol. 117, No. 6, 2013, pp. 718-731, DOI: 10.1016 / j.cviu.2013.01.010, present interactive object modeling based on piecewise planar surface patches. Multiple planes are detected in image sequences by forming hypotheses for planes based on tracked image points. The validity of the hypotheses is verified. A two-step approach is proposed, in which, in the first step, an affine homography with five parameters is computed as an input check. Only if this check is successful is the full homography computed using Hartley's Direct Linear Transform. Planar patches are tracked, and objects are modeled from the planes.

[0009] It is an object of the present invention to provide a method for detecting objects.

[0010] The invention is based on the following considerations: If the camera positions of two frames (individual images) are known, point correspondences (corresponding feature points) can be triangulated, but no objects are generated because the triangulation does not have any model knowledge that could cluster a point cloud into meaningful objects. Disadvantages of monocular systems are that objects close to the epipole can only be triangulated imprecisely and that even the smallest errors in egomotion (the camera's own movement) become noticeable there. The epipole is the image point in a first camera image at which the center of the camera is imaged at a second time. During straight-line driving, for example, the vanishing point corresponds to the epipole. However, this is the relevant area for detecting collisions with stationary vehicles or vehicles driving ahead. Dynamic objects can be triangulated if they move according to the epipolar geometry.However, they are estimated to be too close or too far away due to the unknown relative speed.

[0011] If, instead of individual correspondences, several neighboring correspondences (corresponding features) are considered and assigned to one of a plurality of levels in space, objects can be segmented based on different speeds, scaling and deformation.

[0012] A method according to the invention for detecting objects from a sequence of images from a vehicle camera is the subject of claim 1. The method comprises the steps: a) Recording a sequence of images using a monocular vehicle camera, b1) Determining corresponding features in at least two consecutive images of the sequence of images, b2) Determining planes in space, wherein the planes are determined based on an assignment of adjacent, corresponding features to a plane with one of the following predetermined orientations: a rear wall plane normal to the vehicle's longitudinal direction, a side wall plane normal to the lateral direction or a floor plane normal to the vertical direction, c) Calculating homographies for adjacent corresponding features to determine the plurality of planes and to assign the adjacent, corresponding features to the plurality of planes, d) Determining a movement of the determined planes in space, wherein t / d = ( tx , ty , tz ) is determined as a measure of movement for the assigned levels, where tthe translation of the vehicle camera and d the distance to this plane (perpendicular to this plane), and e) detection of dynamic objects based on the determined movement of at least one plane that can be assigned to an object.

[0013] The vehicle camera is preferably designed to record the surroundings of a vehicle. The surroundings are, in particular, the surroundings in front of the vehicle. The vehicle camera can preferably be integrated into or connected to a driver assistance device, wherein the driver assistance device is designed, in particular, for object recognition from the image data provided by the vehicle camera device. The vehicle camera device is preferably a camera arranged in the interior of the motor vehicle behind the windshield and directed in the direction of travel. According to the invention, the vehicle camera is a monocular camera.

[0014] Preferably, individual images are taken with the vehicle camera at specific or known times, resulting in a sequence of images.

[0015] Correspondence refers to the correspondence of a feature in a first image to the same feature in a second image. Corresponding features in two images can also be described as a flow vector, which indicates how the feature has moved within the image. A feature can be an image section (or patch), a pixel, an edge, or a corner.

[0016] Step b) can alternatively be described as follows: Determination of the movement of several levels, in each of which a large number of neighboring corresponding features are located or will be located.

[0017] Step b) also includes the fact that a plurality of levels in space is specified, and an assignment of adjacent corresponding features to each of the specified levels is made and then the movement of this level is determined (see step d) below).

[0018] Based on the movement of a plane that can be assigned to an object, dynamic objects can be distinguished from static objects in step e).

[0019] In the context of the present invention, the term "plane" describes the following relationships: on the one hand, a criterion for the accumulation of neighboring corresponding features. This means that these are considered to belong together if they lie in a common plane in space and develop over time according to the movement of the plane.

[0020] Corresponding features accumulated in this way are subsequently referred to as, for example, the "ground plane," since they all lie in the plane corresponding to the roadway plane. However, such a ground plane does not extend to infinity, but rather refers to a sub-area of ​​the plane, namely the area in which corresponding features are actually located.

[0021] The multiple planes determined in this way can be taken into account for object detection. This can be done, for example, by deriving a roadway hypothesis from a detected ground plane, and generating an object hypothesis for a raised object from a rear wall plane or a side wall plane. From a roadway hypothesis and object hypothesis(es) for raised objects, open space detection can be performed, which indicates which open space in the vehicle's surroundings is currently passable. One advantageous application of open space detection is, for example, road edge detection, which is not dependent on the detection of lane markings.

[0022] The phrase "detection of objects" can, for example, mean the generation of object hypotheses or objects.

[0023] A homography describes the correspondence of points on a plane between two camera positions or the correspondence of two points in two consecutive images from the vehicle camera. By calculating homographies for neighboring corresponding features, the assignment of neighboring corresponding features to a plane in space can be achieved (see step b).

[0024] Advantageously, the corresponding features can be divided into different segments based on the calculated homographies, with each segment corresponding to a plane in space.

[0025] Finally, objects can be detected taking into account the segmented features.

[0026] In the case of a coordinate system in which the x-direction is horizontal or lateral, the y-direction is vertical and the z-direction is in the longitudinal direction of the vehicle, a floor plane normal to the y-direction, a rear wall plane normal to the z-direction and a side wall plane normal to the x-direction can be specified.

[0027] By calculating homographies of a floor plane, a back wall plane and a side wall plane, an assignment to one of these planes can be made for adjacent corresponding features or for each corresponding feature.

[0028] Preferably, the homographies for the backplane can be calculated according to equation (10) or for the bottomplane according to equation (9) or for the sideplane according to equation (11). Here, a, b, c are constants, x 0 , y 0 , x 1 , y 1 denote correspondences in the first image (index 0), taken at a time t-0, and second image (index 1), taken at an earlier time t-1, and tx , ty , tz are the components of the vector t / d. t describes the translation of the vehicle camera, and d the distance to a plane (perpendicular to this plane), i.e., along the normal vector of this plane. The components tx, ty, and tz are also referred to below as "inverse TTC." TTC stands for "time to collision" and is calculated in one spatial direction as the distance divided by the translational velocity.

[0029] Preferably, if multiple planes with identical orientations occur, the planes with identical orientations can be separated based on the corresponding tx, ty, and tz values. For example, two rear planes that are at different distances from the vehicle camera in the z-direction can be differentiated from each other using different tz values.

[0030] According to an advantageous development, the intrinsic rotation of the vehicle camera can be determined between at least two consecutive images by determining one or more planes (areas) in the image that do not exhibit absolute translational movement. Very distant or infinitely distant objects exhibit no (calculable) translational movement (tx = ty = tz ≈ 0). The movement of these planes in the image is therefore only determined by the camera's intrinsic rotation. Conversely, the camera's intrinsic rotation can be reconstructed by determining the movement of corresponding features or planes.

[0031] Preferably, the image coordinates of the epipole x e,t 0 at time t by the homography of static planes H st the image coordinates of the epipole x e,t 1 at time t-1 are predicted: x e,t 0 = H st * x e,t 1

[0032] According to a preferred embodiment, the pitch rate of the camera can be determined as atan ( ( x e 0 - x e 1 ) / f ) with the focal length f relative to one pixel.

[0033] Alternatively or cumulatively, the yaw rate of the camera can be advantageously determined as atan ( ( y e 0 - y e 1 ) / f ) with the focal length f relative to one pixel.

[0034] Preferably, the rotation of the vehicle camera can be determined or derived from data from the vehicle sensor cluster. In particular, the rolling motion can be determined from this data, which can be provided to the vehicle camera via the vehicle's CAN bus.

[0035] Preferably, after compensation of the self-rotation of the vehicle camera, overtaking vehicles can be detected based on the negative tz component of the movement of the corresponding plane.

[0036] Advantageously, after compensation of the vehicle camera's rotation, approaching objects can be detected as potential collision objects based on the positive tz component of the movement of the corresponding plane.

[0037] According to an advantageous further development, after compensation of the self-rotation of the vehicle camera, vehicles driving in a curve can be detected by a lateral tx component of the movement of the corresponding plane that is not equal to zero.

[0038] Preferably, absolute velocities of objects can be determined from the inverse TTC / the components tx, ty, tz of the movement of the corresponding planes if a disparity map is available in a vehicle stereo camera. If the vehicle camera itself is designed as a stereo camera, a disparity map can be generated by an analysis familiar to those skilled in the art of a pair of images recorded simultaneously by the two camera modules. This means that the distance to an object for one pixel of a camera module is known. This is also possible if a disparity map is available from an additional stereo camera or other 3D camera in the vehicle, and the vehicle camera is designed as a mono camera.

[0039] Advantageously, if a homography with all degrees of freedom is calculated for each plane, this can also be used to reconstruct the 3D environment, especially near the epipole, by using the measured position x t 0 the predicted position H * x t 1 used for triangulation.

[0040] According to a preferred embodiment, an object that has once "outed" itself as a moving object (where a movement of the corresponding plane has been detected) can subsequently be tracked with the label or additional information "is a moving object".

[0041] Advantageously, geometric tracking of a moving object can be performed, i.e., assuming a fixed width, height, and / or length of the object. Using the homography and previous movement of a plane, the movement in subsequent images can be excellently predicted, and a dense optical flow field can be generated.

[0042] Preferably, an image can be divided into similar cells using a grid, and for each cell, a homography can be calculated from the corresponding features identified therein. Cells with matching homographies can then be clustered.

[0043] Preferably, if the calculated homography of a first cell does not sufficiently match the homography of a neighboring cell, a so-called backprojection error of individual corresponding features can advantageously be considered to determine a plane boundary. Corresponding features can be evaluated using the backprojection error. The backprojection error indicates the difference between the measured flow and the flow predicted from the calculated homography. In other words, the backprojection error of a plane describes the difference between a point x at time t-0 and the corresponding point mapped according to the homography of this plane at the previous time t-1 (see below: Equation 4).

[0044] By comparing the backprojection error of a corresponding feature in a first cell with the backprojection errors of the homographies of neighboring cells, and assigning this corresponding feature to the homography with the lowest error, the level boundary (or segment boundary, or cluster boundary) within the first cell can be refined. In this way, different corresponding features of a cell can be assigned to different levels.

[0045] Preferably, the assignment of planes to adjacent corresponding features can be determined essentially in the entire image of the vehicle camera (e.g., in at least 80% of the image area, preferably at least 90%). Since the method according to the invention can be implemented very quickly, generic object detection or scene integration is possible for almost the entire image in real time.

[0046] The invention further relates to a device for detecting objects from a sequence of images of a monocular vehicle camera, the subject of claim 15. The device comprises a camera control unit and an evaluation electronics, wherein the camera control unit is configured or designed to a) to record a sequence of images with the vehicle camera; and wherein the evaluation electronics are configured / designed to, b) determine a movement of planes in space, wherein the planes each correspond to a floor plane, a rear wall plane or a side wall plane and the planes are determined based on an assignment of adjacent corresponding features in at least two consecutive images, d) as a measure of movement t / d = ( tx , ty , tz ) for the assigned level, where t the translation of the vehicle camera and d the distance to this plane, and e) to detect dynamic objects based on the determined movement of a plane that can be assigned to an object.

[0047] The means mentioned may in particular be a camera control unit, a digital signal processor (DSP), a microcontroller or processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) and the like, as well as software for carrying out the corresponding processing steps.

[0048] The present invention can thus be implemented in digital electronic circuits, computer hardware, firmware or software.

[0049] Further features, advantages, and effects of the invention will become apparent from the following description of preferred embodiments of the invention. These show: Fig. 1 schematically shows a typical deformation of an approaching back wall plane; Fig. 2 schematically shows a typical deformation of an approaching ground plane; Fig. 3 schematically shows a typical deformation of a) a rapidly approaching and b) a slowly approaching or more distant back wall plane; Fig. 4 schematically shows a subdivision of an image with two different segments into cells; Fig. 5 segmentation results after a third iteration step; Fig. 6 plane orientation for target validation (validation of potential collision objects); Fig. 7 time to collision analysis; and Fig. 8 projection (or warping) of the guardrail segment at time t-0 (right) to t-1 (left).

[0050] Corresponding parts are generally provided with the same reference numerals in all figures.

[0051] In Fig. 1 A backplane is schematically shown, which at a first time t-1 occupies the hatched area (20, dotted line). At a subsequent time t, the distance between the vehicle camera and the backplane has decreased, resulting in the deformation of the area (21, solid line) of the backplane in the image indicated by the arrows (d1). The area (20; 21) scales or enlarges due to the relative movement of the vehicle camera to the backplane.

[0052] In Fig. 2 A ground plane is schematically depicted, which at a first time t-1 occupies the hatched area (30, dotted line). This could be a section of a road surface on which the vehicle is traveling. Due to the vehicle camera's own movement, the area (in the image) changes at a subsequent time t, leading to the deformation of the area (32) of the ground plane outlined by the arrows (d2). At time t, the lines labeled 32 delimit the area of ​​the ground plane. The "ground plane" is therefore understood here to be a delimited area on the road surface. The edge area results, for example, from signatures (or edge points) on the road surface that can be tracked in the image sequence.

[0053] Fig. 3 illustrates the difference between a rapidly ( Fig. 3a : 20, 21; deformation d1) and a slowly ( Fig. 3b ) approaching rear wall plane (20, 23; deformation d3), if at time t-1 the rear wall plane (20) is in Fig. 3a has the same distance to the vehicle camera as the rear wall plane (20) in Fig. 3b .

[0054] Alternatively, Fig. 3 the difference between a close rear wall plane ( Fig. 3a : 20, 21; deformation d1) and a more distant backplane (20, 23; deformation d3), which move, for example, with the same (relative) speed, then this would be Fig. 3b represented object (20, 21) in real space is larger than the one in Fig. 3a represented object (20, 23).

[0055] If several neighboring correspondences are considered instead of individual correspondences, objects can be segmented based on different speeds, scaling and deformation.

[0056] Assuming that the world consists of planes, these can be described by homographies and separated by their distance, speed and orientation, as shown below.

[0057] A homography describes the correspondence of points on a plane between two camera positions or the correspondence of two points in two consecutive frames: x t 0 = H ∗ x t 1 mit x t 0 = x 0 y 0 1 , x t 1 = x 1 y 1 1 .

[0058] Here, the vector x t 0 the 3D correspondence at time t-0 of the vector x t1 at time t-1. A homography can be calculated image-based by knowing four point correspondences (see Tutorial: Multiple View Geometry, Hartley, R. and Zisserman, A., CVPR June 1999: https: / / de.scribd.com / document / 96810936 / Hartley-Tut-4up accessed on September 26, 2016). The relationships given on page 6 of the tutorial, top left (slide 21), can be formulated using the notation of Equation 1 as follows: − x 0 − y 0 − 1 0 0 0 x 1 x 0 x 1 y 0 x 1 0 0 0 − x 0 − y 0 − 1 y 1 x 0 y 1 y 0 y 1 ⋮ ∗ h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 = 0 with H = h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 .

[0059] Alternatively, if you know the camera translation t , the rotation R and the distance d along the normal vector n of the plane, the homography can be calculated according to equation 3. Equation 3 illustrates that for an inverse TTC t / d non-zero, planes with different orientations n can be modeled and that planes with identical orientation n can be separated via their inverse TTC. H = R − t ∗ n ′ d

[0060] A homography can theoretically be divided into the normal vector n , the rotation matrix R and the inverse TTC t / d Unfortunately, this decomposition is numerically extremely unstable and sensitive to measurement errors.

[0061] If you describe a scene using layers, you can segment it as follows.

[0062] Fig. 4 schematically shows a subdivision into cells (grids / lines). The scene is divided into NxM initial cells, and each point correspondence is assigned a unique ID. This ID initially indicates the cell affiliation. Later, the ID can indicate the affiliation to a cluster or an object.

[0063] An object (especially a back wall plane) in the foreground is shown hatched. The background is shown in white. If a cell contains only one object (cells B3, D3), a homography will describe this cell very well. However, if a cell contains more than one object (cell C3), the homography will not describe either object well. If the point correspondences (black dot or black cross or x) are assigned to the clusters (or segments) of the neighboring cells (B3 or D3) using their backprojection errors, the black dot is assigned to the segment of cell B3 and the black cross to the segment of cell D3, since the homography for cell C3 describes neither the foreground nor the background well.

[0064] If prior knowledge of a scene is available, the segment sizes can be adapted to the scene, for example, by generating larger regions close to the vehicle or in areas with positive classification responses. For each segment, a dedicated back / ground and side-plane homography is calculated, as shown in equations 5 to 10.

[0065] The calculation of the back- / ground- and side-plane homography increases the selectivity, since a homography with fewer degrees of freedom can only poorly model areas that contain different planes and thus corresponding points will have a higher backprojection error, see Fig. 4 . The backprojection error ei is therefore a measure of how well a point x at time t-0 is described by the homography of a plane i of the corresponding point at time t-1: e i = x t 0 − H i x t 1 .

[0066] If the static installation position of the camera and camera rotation in two different views are assumed to be given (e.g. by knowledge of the camera calibration and by the calculation of the fundamental matrix in a monocular system or by rotation values ​​of a yaw rate sensor cluster), the inverse TTC t / d using the flow vectors compensated for the static camera rotation, as shown below for a ground plane n' = [0 1 0] is shown. If the rotation is unknown, it can be approximately replaced by an identity matrix.

[0067] If you replace the quotient t / d in equation 3 by the inverse time to collision t x t y t z follows: x 0 y 0 1 = 0 t x 0 R − 0 t y 0 0 t z 0 x 1 y 1 1 ⇒ x 0 y 0 1 − R x 1 y 1 1 = − 0 t x 0 0 t y 0 0 t z 0 x 1 y 1 1 .

[0068] By introducing the constants a, b, c, where a b c : = R x 1 y 1 1 Equation 5 gives the simplified form: x 0 y 0 1 − a b c = − t x y 1 t y y 1 t z y 1 ⇒ x 0 y 0 1 = a b c − t x y 1 t y y 1 t z y 1 .

[0069] By normalizing the homogeneous coordinates we get: x 0 c − t z y 1 = a − t x y 1 y 0 c − t z y 1 = b − t y y 1

[0070] For more than one measurement, a system of equations of the form Mx = v with a vector x to be determined, a matrix M and a vector v (see equation 9), which can be solved for at least three image correspondences as support points by, for example, a singular value decomposition (singular value decomposition of the matrix) or a least square method: x 0 c − a y 0 c − b ⋮ = − y 1 0 y 1 x 0 0 − y 1 y 1 y 0 ⋮ ∗ t x t y t z .

[0071] The derivation of the back- and side-plane homographies is analogous and results in: x 0 c − a y 0 c − b ⋮ = − x 1 0 x 1 x 0 0 − x 1 x 1 y 0 ⋮ t x t y t z bzw. x 0 c − a y 0 c − b ⋮ = − 1 0 x 0 0 − 1 y 0 ⋮ t x t y t z .

[0072] In order to segment larger objects consisting of several cells, neighboring cells can be combined in a further step by calculating the backprojection errors ∑ x t 0 i − H j x t 1 i or ∑ x t 0 j − H i x t 1 j via support points (see below point 1: RANSAC) of the neighboring segmentsj and i and their homographies are calculated. Two neighboring clusters are combined if ∑ x t 0 i − H j x t 1 i smaller ∑ x t 0 i − H i x t 1 i or, for example, the backprojection error normalized to the predicted flow length is below an adjustable threshold. In particular, two neighboring clusters can be combined if ∑ x t 0 i − H j x t 1 i smaller ∑ x t 0 i − H i x t 1 i and the two backprojection errors ∑ x t 0 i − H j x t 1 i and ∑ x t 0 i − H i x t 1 i falls below a threshold normalized to the flow length. Alternatively, backprojection errors can be used as potentials in a graph and a global solution can be calculated. The compactness of the clusters can be determined using the edge potentials in the graph.

[0073] Once the segments have been merged, the homographies are recalculated, and the point correspondences are assigned to the clusters with the lowest backprojection error. Considering only directly adjacent clusters allows for the generation of very compact objects. If the minimum error exceeds a configurable threshold, new (cluster / object) IDs are assigned to the correspondences to detect partially occluded objects or objects with slightly different TTC. By adjusting the threshold, the resolution of (slightly) different objects can be adjusted.

[0074] The backprojection errors can be biased to reduce costs for contiguous regions or to increase the costs for an ID change if point correspondences have had the same ID affiliation for a longer period of time.

[0075] Fig. 5 shows an example of scene segmentation: Fig. 5a shows an image taken by a vehicle camera located inside the vehicle and capturing the surroundings ahead through the windshield. A three-lane roadway (51), e.g., a motorway, is visible. The lanes are separated by lane markings. Vehicles are traveling in all three lanes. The vehicle ahead in its own lane (53) may obscure other vehicles ahead in its own lane. To the left of the three-lane roadway, there is a raised structural barrier (52) separating the oncoming lane.

[0076] To the right of the three-lane carriageway (51) is a shoulder or hard shoulder, bordered to the right by a guardrail, behind which lies a wooded area. Some distance ahead of your vehicle, you can see sign gantries (54), one of which spans the three-lane carriageway (51).

[0077] Analogous to the Fig. 4 described procedure, this scene can be segmented. In Fig. 5b bis 5d Cells (56) can be seen. Point correspondences (55) are shown in the cells. The assignment of a cell (56) to a segment is indicated by the color of the cell border or the point correspondences (55).

[0078] Fig. 5b shows the red channel of the segmented image, Fig. 5c the green channel and Fig. 5d the blue channel.

[0079] Different segments were given different colors. One segment, which is green in the original, extends over the bottom five to six lines (in Fig. 5b und 5d (shown in white and without cell frames). This segment corresponds to the ground plane, i.e. the surface of the road (51) on which the car is driving.

[0080] Another segment can be seen in the middle of the image, which is pink in the original. Therefore, it has Fig. 5b high red values, in Fig. 5d weaker blue values ​​and in Fig. 5c No green values. This segment corresponds to the rear wall plane of the (transporter) vehicle (53) driving ahead in the same lane.

[0081] The segmentation result shown was determined in just three iterations without any prior knowledge of the scene. This demonstrates the enormous speed and performance of an embodiment of the invention using temporal integration.

[0082] Fig. 6 shows a determination of the orientation of planes in the already Fig. 5 described scene. Fig. 6a shows the surrounding situation for orientation purposes according to Fig. 5a .

[0083] All correspondences assigned to a side wall level show Fig. 6b The correspondences on the left edge were assigned to a right sidewall plane, which is correct, since the right side of the structural boundary (52) facing the opposite lane is located there. The correspondences in the right half of the image were assigned to left sidewall planes, which is also correct, since the "left side" of the roadside development or planting is located there. Fig. 6c shows which correspondences are assigned to a ground plane, which is correct since the surface of the roadway (51) can be seen there in the image.

[0084] Fig. 6d shows which correspondences are assigned to a rear wall plane. This is largely correct. This determination alone cannot sufficiently distinguish between different rear wall planes, e.g., those of the delivery truck (53) traveling ahead in the same lane from the signs of the sign gantry (54) located above it in the image. However, this representation can already provide important clues as to where raised objects occur in the vehicle's surroundings.

[0085] As in Fig. 7 As illustrated, the inverse TTC (tx , ty , tz ) can be used to detect dynamic objects. Fig. 7a again shows the image of the vehicle situation (identical to Fig. 6a ). The vehicle ahead in the same lane (73) is a delivery van. Two vehicles (71 and 72) are traveling in the left lane, and two more vehicles (74 and 75) are traveling in the right lane.

[0086] Fig. 7b shows correspondences that again correspond to the ground plane (purple in the original) and are the only ones with a red component.

[0087] Fig. 7c shows correspondences assigned to moving objects. These are originally green when they are moving away from the vehicle (i.e., moving faster) and turquoise when they are moving slower.

[0088] Fig. 7d shows correspondences with a blue component, i.e. those that correspond to the ground plane (cf. Fig. 7b ), moving objects approaching your own vehicle (cf. Fig. 7c ) and those that correspond to static raised objects, these are only in Fig. 7d such as forest areas to the left and right of the motorway and the sign gantries. Fig. 7c und 7d together, it can be seen that the vehicle in its own lane (73) is approaching. The same applies to the vehicle in front in the right lane (75). In contrast, the other vehicles (71, 72, and 74) are moving away.

[0089] The area that corresponds to the sky in the image does not lead to any correspondences due to the lack of structure in the image (white in Fig. 7b bis 7d ) .

[0090] If the self-rotation is taken into account in the correspondences before calculating the homography, or if the self-rotation is taken into account in the rotation matrix R Considered, overtaking vehicles can be detected based on their negative tz component, and vehicles swerving or cornering can be detected based on a non-zero lateral tx component. If the dynamic segments are predicted based on their homographies (see below "Optical Flow Densification Based on Homographies"), a dynamic map can be constructed over time.

[0091] Considering equation 3, we see that segments with an inverse TTC equal to zero describe the rotation matrix and it can be obtained by calculating a homography with full degrees of freedom (equation 2) from segments with t / d equal to zero. Assuming that the translational components are not noticeable near the epipole, the pitch and yaw rates can also be determined by predicting the coordinates of the epipole (xe , ye ) by the homography of static segments and the atan ( ( x e 0 - x e 1 ) / f ) or atan ( ( y e 0 - y e 1 ) / f ) with the focal length per pixel f is calculated.

[0092] If a homography with all degrees of freedom is calculated for each cluster, these can also be used to reconstruct the 3D environment by using the measured position x t 0 , the predicted position H*x t 1 used for triangulation. This not only reduces the influence of measurement errors but also allows for the reconstruction of objects near the epipole.

[0093] In the following, an embodiment for densifying the optical flow based on homographies is described.

[0094] Is the segmentation at the time t-1As is known, it can be used both for predicting objects and for generating a dense flow field. Signature-based flow methods generate signatures and attempt to uniquely assign them in successive frames. Signatures are usually calculated from a patch (image section or image area) of a defined size. However, if the size and shape of a patch change, finding a correspondence with a fixed template (template, pattern, meaning, for example, an image section of an image in the image sequence that corresponds to an object - for example, a vehicle template) is no longer possible. When approaching a back plane, for example, the size of a patch changes. Or when moving over a ground plane or parallel to a side plane, both the size and shape of a patch change, see Fig. 1 und 2 ). If the segmentation is at the time t-1present, the homographies can be recalculated over already found flow vectors and used to determine the position and shape of already established correspondences of t-1 on t-0 to predicate.

[0095] Alternatively, the current frame at the time t-0 on the time t-1 transform to compensate for scale and shape changes.

[0096] Fig. 8 illustrates such an approach.

[0097] Fig. 8a shows an image of a different driving situation, captured by the vehicle camera at time t-1. It shows a highway with three lanes in each direction. To the left of the three-lane highway, there is a guardrail (81) that forms a raised barrier to the oncoming lane. To the right of the highway, there is a noise barrier (82).

[0098] Fig. 8b shows an image that was taken at the following time t and was transformed ("warped") via the homography of the guardrail in such a way that the changes in the image in the area of ​​the guardrail that occur as a result of the movement of the vehicle and thus of the vehicle camera between the two recording times are compensated. The forward movement of the own vehicle leads to Fig. 8b This means that the nearest lane marking line is closer to your own vehicle than in Fig. 8a The transformation leads to the trapezoidal displacement of the image, which is Fig. 8f is illustrated by a dashed line.

[0099] Fig. 8c now shows corresponding features (85) that are in the area of ​​the guardrail (81, cf. Fig. 8a ) were determined as white dots.

[0100] Fig. 8d shows where these corresponding features are to be expected in the next image (86), after this one has been Fig. 8b has been transformed as described.

[0101] In Fig. 8e und 8f This situation is shown again in a black and white representation, with the corresponding features (85) now corresponding to the black dots on the guardrail (81) in the left half of the image.

[0102] To generate a dense flow field, the current image can be warped to the previous image for each segment in order to find existing correspondences that have changed in scale or shape, or to establish new correspondences using congruent templates.

[0103] If there are not enough flow vectors in a current frame to recalculate a homography, the homography from the last frame can be used as an approximation to make the correspondence finding more robust against shape and scale changes.

[0104] The following design forms or aspects are advantageous and can be provided individually or in combination: 1. The image is divided into NxM cells, and each point correspondence within a cell is assigned a unique cell ID. RANSAC is used to calculate the back-ground and side-plane homographies (equations 9, 10, and 10) from the correspondences with identical IDs. The homography with the lowest backprojection error, as well as the sampling points used to calculate the homography, are stored. RANSAC (RAndom SAmple Consensus) typically uses a minimum number of randomly selected correspondences in each iteration to form a hypothesis. For each corresponding feature, a value is then calculated indicating whether the corresponding feature supports the hypothesis. If the hypothesis is sufficiently supported by the corresponding features, the non-supporting corresponding features can be discarded as outliers.Otherwise, a minimum number of correspondences is randomly selected again. 2. For neighboring cells . i , j the backprojection errors ∑ x t 0 i − H j x t 1 i bzw . ∑ x t 0 i − H i x t 1 i calculated using the support points of the neighboring homography. If the backprojection error ∑ x t 0 i − H j x t 1 i smaller ∑ x t 0 i − H i x t 1 i or if the errors fall below a threshold normalized to the flow length, the IDs are combined and the homographies are recalculated. In particular, two neighboring cells can be clustered as belonging to the same plane (or to the same segment or to the same object) if the backprojection error ∑ i x t 0 i − H j x t 1 i less than ∑ i x t 0 i − H i x t 1 i and if both backprojection errors ∑ i x t 0 i − H j x t 1 i and ∑ i x t 0 i − H i x t 1 i fall below a threshold normalized to the river length. 3. The backprojection errors x t 0 - H i x t1 of all point correspondences are calculated for the adjacent segments, and a point correspondence is assigned to the segment with the smallest backprojection error. If the minimum error exceeds a threshold, the correspondences are assigned a new object ID to detect smaller or partially occluded objects. 4. The homographies of the points at time t-1Extracted segments are recalculated at the beginning of a new frame (t-0) using the already found image correspondences, and the existing segment IDs are predicted into the current frame. If there are not enough flow vectors in the current frame to recalculate a homography, the homographies from the last frame can be used as an approximation. 5. To generate a dense flow field, the current frame (t-0) is warped to the last frame (t-1) for each segment in order to rediscover existing correspondences that have changed in scale or shape, or to establish new correspondences. 6. The backprojection errors of the back / ground and side planes can be used to validate elevated targets, see Fig. 6 7. If, for example, a vehicle stereo camera has a disparity map, the absolute speeds can be determined from the inverse TTC t / dbe calculated, since then the absolute distances d for individual pixels in the disparity map. 8. If a complete homography with all degrees of freedom is calculated for each segment, the rotation matrix R can be determined from segments with a TTC close to infinity (or inverse TTCs close to zero). 9. The 3D environment can be calculated from the predicted position ( Hx t 1 , x t 1 ) instead of the measured position ( x t 0 , x t 1 ) can be reconstructed and also allows objects at the epipole to be reconstructed.

Claims

1. Method for detecting objects, comprising the following steps: a) recording a sequence of images by means of a monocular vehicle camera, b1) determining corresponding features in at least two successive images of the image sequence, b2) ascertaining planes in space, wherein the planes are established on the basis of an assignment of adjacent, corresponding features to a respective plane with one of the following predefined orientations: a back plane normal to the longitudinal direction of the vehicle, a side plane normal to the lateral direction or a ground plane normal to the vertical direction, c) calculating homographies for adjacent corresponding features for the purpose of determining the plurality of planes and for the purpose of assigning the adjacent, corresponding features to the plurality of planes, d) ascertaining a motion of the ascertained planes in space, wherein tld = ( tx, ty, tz) is ascertained as a measure of motion for the ascertained planes, wherein t describes the translation of the vehicle camera and d describes the distance from this plane, and e) recognizing dynamic objects on the basis of the ascertained measure of motion of at least one ascertained back or side plane which can be assigned to an object.

2. Method according to Claim 1, wherein a coordinate system is used in which the x-direction runs laterally, the y-direction runs vertically and the z-direction runs in the longitudinal direction of the vehicle.

3. Method according to either of the preceding claims, wherein an object hypothesis for a raised object is generated from an ascertained back plane and / or a side plane.

4. Method according to any of the preceding claims, wherein an own rotation of the vehicle camera between at least two successive images is ascertained on the basis of ascertaining one or more static planes which have no translational motion.

5. Method according to Claim 4, wherein the image coordinates of the epipole xe,t0 at the time t are predicted by the homography of static planes Hst of the image coordinates of the epipole xe,t1 at the time t-1: xe,t0 = Hst * xe,t16. Method according to Claim 5, wherein the pitch rate of the vehicle camera is determined as atan((xe0 - xe1) / f) with the focal length f with respect to one pixel.

7. Method according to Claim 5 or 6, wherein the yaw rate of the vehicle camera is determined as atan((ye0 - ye1) / f) with the focal length f with respect to one pixel.

8. Method according to any of the preceding claims, wherein an own rotation of the vehicle camera is determined from data or taking account of data of a sensor cluster of the vehicle.

9. Method according to any of Claims 4 to 8, wherein, after compensation of the own rotation of the vehicle camera, overtaking vehicles are recognized on account of their negative tz component of the motion of the corresponding plane.

10. Method according to any of Claims 4 to 9, wherein, after compensation of the own rotation of the vehicle camera, approaching objects are recognized as potential collision objects on account of the positive tz component of the motion of the corresponding plane.

11. Method according to any of Claims 4 to 10, wherein, after compensation of the own rotation of the vehicle camera, cornering vehicles are recognized by virtue of a lateral tx component of the motion of the corresponding plane that is not equal to zero.

12. Method according to any of the preceding claims, wherein an object for which a motion of the corresponding plane has been established is subsequently tracked using the label "mobile object".

13. Method according to any of the preceding claims, wherein geometric tracking of an object is carried out by assuming a fixed width, height and / or length of the object in space.

14. Device for detecting objects from a sequence of images from a monocular vehicle camera comprising a camera controller and evaluation electronics, wherein the camera controller is designed a) to record a sequence of images using the monocular vehicle camera; and wherein the evaluation electronics are designed b1) to determine corresponding features in at least two successive images of the image sequence, b2) to ascertain planes in space, wherein the planes are established on the basis of an assignment of adjacent, corresponding features to a respective plane with one of the following predefined orientations: a back plane normal to the longitudinal direction of the vehicle, a side plane normal to the lateral direction or a ground plane normal to the vertical direction, c) to calculate homographies for adjacent corresponding features for the purpose of determining the plurality of planes and for the purpose of assigning the adjacent, corresponding features to the plurality of planes, d) to ascertain a motion of the ascertained planes in space, wherein tld = ( tx, ty, tz ) is ascertained as a measure of motion for the ascertained planes, wherein t describes the translation of the vehicle camera and d describes the distance from this plane, and e) to recognize dynamic objects on the basis of the ascertained measure of motion of at least one ascertained back or side plane which can be assigned to an object.