Method and apparatus for determining the orientation of an object surface

Abstract

The present invention relates to a method for determining the orientation of the surface (26) of an object (12) by means of a camera (16) of a vehicle (14), the object being located in a detection area in front of or behind the vehicle (14), particularly a road vehicle, comprising the following steps: acquiring a first image (22) of the detection area by means of the camera (16); acquiring a second image (24) of the detection area by means of the camera (16) after the acquisition of the first image (22); generating first image data corresponding to the first image (22) and second image data corresponding to the second image (24); determining a total of eight image coordinates of four pixels in each of the first image (22) and the second image (24) by means of the first image data and the second image data, the four pixels corresponding to four points (a to d) on the surface (26) of the object (12); determining a normal vector (n) of the surface (26) of the object (12) by means of the eight image coordinates; and determining the orientation of the surface (26) of the object (12) by means of the normal vector (n).

Description

Technical Field

[0001] The present invention relates to a method and apparatus for determining the orientation of an object surface by means of a camera in a vehicle, the object being located in a detection area in front of or behind the vehicle, particularly a road vehicle. Background Technology

[0002] In the fields of autonomous driving and driver assistance systems, image data captured by cameras fixed to vehicles needs to be processed to determine the orientation of object surfaces. If the orientation of such objects is incorrectly determined, dangerous situations may arise. For example, if a lane marking that looks like the rear of a vehicle is mistaken for a vehicle, sudden emergency braking may occur. Conversely, if the rear of a vehicle is incorrectly identified as a lane marking, a rear-end collision may occur.

[0003] Document EP 2 092 270 B1 discloses a method and apparatus for identifying and determining the position of planar objects in an image using a stereo camera.

[0004] The paper "Deeper understanding of the homography decomposition for vision-based control. RR-6303, INRIA. 2007, pp. 90" by Malis et al. discloses a method for calculating the so-called homography matrix / transfer matrix, which can be used to estimate the motion of a calibrated camera between two images of a planar object. The paper also discloses an analytical method for calculating normal vectors based on the homography matrix. Summary of the Invention

[0005] The object of the present invention is to provide a method and apparatus in which the orientation of an object surface in a detection area in front of or behind a vehicle can be determined simply by means of a camera of a vehicle.

[0006] This objective is achieved by a method having the features of claim 1 and an apparatus having the features of the independent method claim. Advantageous improvements are provided in the dependent claims.

[0007] A method for determining the orientation of an object surface in a detection area in front of or behind a vehicle using a camera from the vehicle includes the following steps:

[0008] - Acquire the first image of the detection area using a camera;

[0009] - In terms of time, after the first image is acquired, a second image of the detection area is acquired by means of a camera;

[0010] - Generate first image data corresponding to the first image and second image data corresponding to the second image;

[0011] - Based on the first image data and the second image data, determine a total of eight image coordinates corresponding to four image points in each of the first and second images, wherein the four image points correspond to four points on the surface of the object.

[0012] - The normal vector of the object's surface is determined by using eight image coordinates.

[0013] - The orientation of an object's surface is determined by using normal vectors.

[0014] The camera is preferably fixedly connected to the vehicle. The detection area can be, in particular, a portion of the driving lane in front of and / or behind the vehicle. The vehicle is especially a road vehicle.

[0015] To determine the orientation of a surface, a first image is first acquired. Then, a second image is acquired in time. If the vehicle moves during this time period, the object's position relative to the vehicle changes between the acquisition of the first and second images. With a few exceptions, such as when the object is another vehicle moving at the exact same speed as the vehicle equipped with the camera, the four image coordinates of four image points in the first image differ from the four image coordinates of four image points in the second image. Based on this movement of the image points, the normal vector can be determined geometrically, and thus the orientation of the object's surface relative to the vehicle can be determined. Here, it is sufficient for the first and second images to be non-stereoscopic images to determine the surface orientation. Therefore, to perform the method, only a monocular camera is needed, i.e., a camera with a single objective lens that can only acquire non-stereoscopic images. Thus, the method according to the invention can be performed simply.

[0016] In a preferred embodiment, the homography transformation / transfer function is determined using eight image coordinates. The homography transformation converts four image points in the first image into four image points in the second image. The homography transformation is determined in the form of a homography matrix, and the normal vector is determined using this homography matrix. Homography is a geometric mapping that converts the four image coordinates of four image points in the first image into the four image coordinates of four image points in the second image. This mapping can be represented in the form of a homography matrix, which can be determined based on eight image coordinates using a suitable method. The normal vector can be quickly and robustly determined from the elements of the homography matrix. Therefore, reliable determination of surface orientation can be achieved using the homography matrix.

[0017] In another preferred embodiment, the distance between the camera and the object surface is determined using a homography matrix. Specifically, the distance between the camera's aperture and the object surface is determined. Preferably, the distance along the normal vector is determined. Similar to the normal vector, the distance between the camera and the surface can also be determined from the elements of the homography matrix. Therefore, a reliable determination of the distance between the camera and the object surface can be achieved using a homography matrix.

[0018] Preferably, the homography matrix is ​​determined numerically or analytically. This allows for a simple calculation of the homography matrix.

[0019] In another preferred embodiment, the pitch angle is determined by means of a normal vector, which corresponds to the inclination of the normal vector about the lateral axis of the vehicle relative to the vertical axis of a coordinate system fixed to the vehicle. The pitch angle determines the orientation of the object's surface. A 90° pitch angle means the surface is perpendicular to the vehicle's direction of travel. A 0° pitch angle means the object's surface is parallel to the roadway on which the vehicle travels. Therefore, the pitch angle can, for example, reliably determine whether an object can be driven over without danger.

[0020] In another preferred embodiment, an image region encompassing the object surface of the first image is determined using first image data. Image coordinates corresponding to four pixels in the first image are determined using a subset of the first image data corresponding to this image region. The image region of the first image is determined, in particular, by an image recognition method. Preferably, an image recognition method is used that can identify objects of a specific category, such as lane markings, vehicles, traffic signs, and other road users. Preferably, the image region is a rectangle that minimally encloses the object surface. Determining the image region reduces the first image to the relevant area. This simplifies further processing. More than one image region of the first image can also be determined, each corresponding to an object whose surface orientation is to be determined.

[0021] In another preferred embodiment, image coordinates corresponding to four image points in the first image are determined using the first image data and employing an image recognition method, particularly a machine learning method. The image recognition method is specifically designed such that three of the four points are not collinear. This ensures the successful determination of the homography matrix. Therefore, the robustness and reliability of surface orientation determination can be improved by using an image recognition method.

[0022] In another preferred embodiment, using first image data, second image data, and image coordinates corresponding to four image points in the first image, additional image coordinates corresponding to the four image points in the second image are determined using image processing methods, particularly optical flow-based methods or machine learning methods. The positions of the four image points in the camera's image frames change between capturing the first and second images. The four image points move as if traversing the image frames. The image processing method tracks this movement, thereby quickly locating the four image points in the second image. Therefore, this particularly accelerates the determination of surface orientation. Tracking methods are particularly useful for tracking movement and for locating image points in the second image.

[0023] In another preferred embodiment, the ability of a vehicle to drive over an object without risk is determined based on the orientation of the object's surface. This is particularly true for objects whose surfaces lie in the plane of a driving lane, such as lane markings. Determining the surface orientation provides a easily determinable criterion by which to determine whether a vehicle can drive over an object without risk.

[0024] In another preferred embodiment, the orientation of the object surface is further processed as an input parameter to the object recognition method. The object recognition method may, for example, include distinguishing whether an object is the rear of a vehicle, or whether a lane marking has been incorrectly identified as the rear of a vehicle. This can be used to make the object recognition method more reliable.

[0025] In another preferred embodiment, whether an object is a lane marking is determined based on the orientation of its surface. If the object's surface lies in the plane of the driving lane, it can be assumed with high probability that the object is a lane marking. This information can be used, for example, by a driver assistance system to highlight the lane marking for the vehicle's driver. Similarly, information about whether an object is a lane marking can be used by the vehicle's autonomous driving system to determine a trajectory, i.e., to perform evasive or braking maneuvers.

[0026] The present invention also relates to an apparatus for determining the orientation of an object surface by means of a camera in a vehicle, the object being located in a detection area in front of or behind the vehicle, particularly a road vehicle. The apparatus comprises: a camera configured to acquire a first image of the detection area, and, temporally after acquiring the first image, acquire a second image of the detection area, and generate first image data corresponding to the first image and second image data corresponding to the second image; and a control unit designed to determine, by means of the first and second image data, eight image coordinates of four pixels in each of the first and second images, the four pixels corresponding to four points on the object surface. The control unit determines a normal vector of the object surface by means of the eight image coordinates, and determines the orientation of the object surface by means of the normal vector.

[0027] The device has the same advantages as the claimed method and can be improved in the same way, especially by the features of the appended claims.

[0028] Advantageously, the camera is a monocular camera. Monocular cameras are cheaper than comparable stereo cameras. Furthermore, monocular cameras do not require complex calibration. Therefore, by using a monocular camera, the device can be manufactured at a particularly cost-effective cost and has low maintenance costs.

[0029] Preferably, the camera is fixedly connected to the vehicle. Attached Figure Description

[0030] Other features and advantages are described below, with further elaboration of the embodiments in conjunction with the accompanying drawings. Wherein:

[0031] Figure 1 A schematic diagram of a device for determining the orientation of an object surface in a detection area in front of or behind a vehicle is shown.

[0032] Figure 2 It shows that according to Figure 1 A schematic diagram of the device's camera and the object;

[0033] Figure 3a A schematic diagram of the first image of the detection area is shown;

[0034] Figure 3b A schematic diagram of the second image of the detection area is shown;

[0035] Figure 4 A flowchart is shown for the process of determining surface orientation. Detailed Implementation

[0036] Figure 1 A schematic diagram of a device 10 for determining the orientation of the surface of an object 12 in a detection area in front of or behind a vehicle 14 is shown.

[0037] Object 12 in Figure 1 The object 12 is exemplarily shown as a lane marking, which looks like the front or rear of another vehicle. Alternatively, object 12 could also be another vehicle, for example.

[0038] The device 10 includes a camera 16, configured as a monocular camera and fixedly connected to the vehicle 14. The camera 16 is oriented along the direction of travel of the vehicle 14, enabling it to detect a detection area on the lane 18 preceding the vehicle 14. The device 10 also includes an image analysis and processing unit 20, which is connected to the camera 16 via a cable 21 and forms the control unit of the device. The image analysis and processing unit 20 is configured to analyze images 22, 24 (see image 24) captured by the camera 16. Figure 3a and3b The image data generated in the image determines the orientation of surface 26 of object 12 (see image data generated in the ... Figure 2 Here, a first image 22 of the detection area is acquired using camera 16, and a second image 24 of the detection area is acquired using camera 16 after the acquisition of the first image 22. First image data corresponding to the first image 22 and second image data corresponding to the second image 24 are generated. The image analysis and processing unit 20 determines eight image coordinates of four pixel points in each of the first image 22 and the second image 24 based on the image data. These four pixel points correspond to four points a to d on the surface 26 of the object 12 (see...). Figure 2 The image analysis and processing unit 20 determines the normal vector n of the surface 26 of the object 12 using eight image coordinates. Furthermore, the image analysis and processing unit 20 determines the orientation of the surface 26 of the object 12 using the normal vector n. The determination method will also be described below using... Figures 2 to 4 Further explanation is needed.

[0039] Figure 2 It shows that according to Figure 1 A schematic diagram of the camera 16 and object 12 of the device 10. Figure 2 Coordinate system 28, which is fixed to the camera's coordinate system, is also shown. Because the camera 16 is fixedly connected to the vehicle 14, the coordinate system fixed to the camera is also the coordinate system fixed to the vehicle. Here, the first coordinate direction X is the driving direction of the vehicle 14, the second coordinate direction Y is the lateral axis of the vehicle 14, and the third coordinate direction Z is the vertical axis of the vehicle 14.

[0040] The positions of the four points a to d on surface 26 of object 12 are... Figure 2 The image is illustrated by four arrows P1 to P4, originating from the camera lens 30 of camera 16. The positions of the four points a to d relative to camera 16 change due to the relative motion between vehicle 14 and object 12 from the moment the first image 22 is captured to the moment the second image 24 is captured. Therefore, the image coordinates of the image points corresponding to the four points a to d on the surface 26 of object 12 also change. In other words, the image coordinates of the image points corresponding to the four points a to d on the surface 26 of object 12 in the first image 22 are different from their image coordinates in the second image 24. This will also be explained below with the aid of… Figure 3a and Figure 3b Further explanation is needed.

[0041] How the image coordinates of the four points a to d on the surface 26 of object 12 change depends on the relative motion between vehicle 14 and object 12, and also on the orientation of surface 26. Therefore, for the image analysis and processing unit 20, the orientation of surface 26 can be determined using image data in the form of a normal vector n, as shown in... Figure 2 As shown by the arrow, it originates from surface 26 of object 12 and points towards the camera lens 30 of camera 16. How to determine the normal vector n will be explained further below. Figure 4 To further explain, the distance between vehicle 14 and object 12 is defined by the normal vector n as the distance between the camera lens 30 and the point of contact between the normal vector n and the surface 26 of object 12. This distance... Figure 2 The number 32 is displayed as a dashed line in the middle.

[0042] Figure 3a A schematic diagram of the detection area, shown in the first image 22 captured by camera 16, is presented. The first image shows the detection area containing object 12 on the lane 18 in front of vehicle 14. The positions of the image points corresponding to four points a to d on the surface 26 of object 12 are shown in the diagram. Figure 3a and Figure 3b It is indicated by a cross-shaped symbol.

[0043] Figure 3b A schematic diagram of a second image 24 captured by camera 16 shows the detection area. The second image 24 shows the detection area containing object 12 on the lane 18 ahead of vehicle 14, where the image was captured later than the first image 22. The positions of the image points corresponding to four points a to d on the surface 26 of object 12 are compared to... Figure 3a The situation has changed because the relative positions between vehicle 14 and object 12 have changed between taking the first image 22 and taking the second image 24.

[0044] Figure 4 A flowchart is shown for the process of determining the orientation of surface 26.

[0045] The process begins in step S10. In step S12, a first image 22 of the detection area is acquired using camera 16. Next, in step S14, using an image recognition method, particularly a machine learning method, it is determined whether the relevant object 12 exists in the first image 22. If the object 12 cannot be determined in the first image 22, the process ends in step S24. Otherwise, the image coordinates of four pixels in the first image 22, corresponding to four points a to d on the surface 26 of the object 12, are determined using a suitable method. Here, the points are selected such that the four points a to d are not collinear. Then, temporally after step S12, a second image 24 of the detection area is acquired in step S16. In step S18, it is determined whether the object 12 exists in the second image 24 using an image recognition method. If the object 12 cannot be determined again in the second image 24, the process ends in step S24. Otherwise, the image coordinates of four pixels in the second image 24, corresponding to the four points a to d on the surface 26 of the object 12, are determined using a suitable method, particularly one based on optical flow.

[0046] In step S20, a homography matrix H is determined using eight image coordinates, which transforms the image coordinates of four pixels in the first image 22 into the image coordinates of four pixels in the second image 24. Next, in step S22, the normal vector n is calculated based on the elements of the homography matrix H. For this purpose, matrix S is first determined by symmetrizing the homography matrix H:

[0047] S = H T HI

[0048] Here, I is a 3x3 identity matrix. In the following text, the elements of the symmetric matrix S are referred to as si. ij Let represent element s ij The corresponding sub-determinant is assigned by M. sij Let's represent it. Therefore, the normal vector n can be expressed as:

[0049]

[0050]

[0051] Here, is the symbol of the sub-determinant corresponding to element s13 of S.

[0052] Next, in step S24, the pitch angle is determined based on the normal vector n, which corresponds to the inclination of the normal vector n around the y-axis (lateral axis) of the vehicle 14, which is fixed in the vehicle coordinate system, relative to the z-axis (vertical axis). Then, the orientation of the surface 26 of the object 12 is determined using the pitch angle. The method then ends in step S26.

[0053] With the help of Figures 1 to 4The method and apparatus according to the invention are illustrated by way of example through embodiments. In the illustrated embodiments, the detection area is located particularly in front of the vehicle 14. It goes without saying that embodiments of the illustrated method can also be applied accordingly to the area behind the vehicle 14.

[0054] List of reference numerals

[0055] 10 devices

[0056] 12 objects

[0057] 14 vehicles

[0058] 16 cameras

[0059] 18 lanes

[0060] 20 Image Analysis and Processing Units

[0061] 21 Cables

[0062] Images 22 and 24

[0063] 26 Surface

[0064] 28 Coordinate System

[0065] 30 camera lenses

[0066] 32 Distance

[0067] X, U, Z coordinate axes

[0068] Points a, b, c, and d

[0069] n is the normal vector.

Claims

1. A method for determining the orientation of a surface (26) of an object (12) by means of a camera (16) of a vehicle (14), the object being located in a detection area in front of or behind the vehicle (14), the vehicle being a road vehicle, wherein in the method: A first image (22) of the detection area is acquired by means of a camera (16). After the first image (22) is acquired, a second image (24) of the detection area is acquired by means of a camera (16). Generate first image data corresponding to the first image (22) and second image data corresponding to the second image (24). Using the first image data and the second image data, eight image coordinates are determined for each of four image points in the first image (22) and the second image (24), the four image points corresponding to four points a to d on the surface (26) of the object (12). The normal vector n of the surface (26) of the object (12) is determined by means of the eight image coordinates. The orientation of the surface (26) of the object (12) is determined by means of the normal vector n. Based on the orientation of the surface (26) of the object (12), it is determined whether the object (12) is a lane mark. The information about whether the object is a lane mark is used by the vehicle's automatic driving system to determine the trajectory, that is, to perform avoidance or braking operations.

2. The method according to claim 1, characterized in that, Using the eight image coordinates, the homography transformation of four image points in the first image (22) into four image points in the second image (24) is determined, the homography transformation is determined in the form of a homography matrix, and the normal vector n is determined using the homography matrix.

3. The method according to claim 2, characterized in that, The distance between the camera (16) and the surface (26) of the object (12) is determined by means of the homography matrix.

4. The method according to claim 2 or 3, characterized in that, The homography matrix is ​​determined using numerical or analytical methods.

5. The method according to any one of claims 1 to 3, characterized in that, The pitch angle is determined by means of the normal vector n, which corresponds to the inclination of the normal vector n around the lateral axis Y of the vehicle (14) relative to the vertical axis Z of the coordinate system fixed to the vehicle, and the orientation of the surface (26) of the object (12) is determined by means of the pitch angle.

6. The method according to any one of claims 1 to 3, characterized in that, The image region of the first image (22) including the surface (26) of the object (12) is determined by means of the first image data, and the image coordinates corresponding to the four image points in the first image (22) are determined by means of a subset of the first image data corresponding to the image region.

7. The method according to claim 6, characterized in that, The image region is a rectangle that minimally encloses the surface (26) of the object (12).

8. The method according to any one of claims 1 to 3, characterized in that, Using the first image data, image coordinates corresponding to four image points in the first image (22) are determined using an image recognition method, which is a machine learning method.

9. The method according to any one of claims 1 to 3, characterized in that, Using the first image data, the second image data, and the image coordinates corresponding to the four image points in the first image (22), additional image coordinates corresponding to the four image points in the second image (24) are determined using an image processing method, which is based on optical flow or machine learning.

10. The method according to any one of claims 1 to 3, characterized in that, Whether the vehicle (14) can drive over the object (12) without risk is determined based on the orientation of the surface (26) of the object (12).

11. The method according to any one of claims 1 to 3, characterized in that, The orientation of the surface (26) of the object (12) is further processed as an input parameter of the object recognition method.

12. A device (10) for determining the orientation of a surface (26) of an object (12) located in a detection area in front of or behind a vehicle (14), the vehicle being a road vehicle, the device comprising: A camera (16) is configured to acquire a first image (22) of a detection area, acquire a second image (24) of the detection area after the acquisition of the first image (22) in time, and generate first image data corresponding to the first image (22) and second image data corresponding to the second image (24); The image analysis and processing unit (20) is configured to determine, using the first image data and the second image data, a total of eight image coordinates for four image points in each of the first image (22) and the second image (24), the four image points corresponding to four points a to d on the surface (26) of the object (12), and to determine the normal vector n of the surface (26) of the object (12) using the eight image coordinates, and to determine the orientation of the surface (26) of the object (12) using the normal vector n. Based on the orientation of the surface (26) of the object (12), it is determined whether the object (12) is a lane marking. Information about whether the object is a lane marking is used by the vehicle's automatic driving system to determine the trajectory, i.e., to perform avoidance or braking operations.

13. The apparatus (10) according to claim 12, characterized in that, The camera (16) is a monocular camera.

14. The apparatus (10) according to claim 12 or 13, characterized in that, The camera (16) is fixedly connected to the vehicle (14).

Images