Method for determining the location of an object relative to road line markings of a road

Abstract

The invention relates to a method for determining the position of an object relative to road line markings of a road from a camera mounted on the rear of a motor vehicle, comprising the steps of generating (E1) a series of images of the environment behind the vehicle, selecting at least one point of the road line markings at the bottom of at least a first image of the series and tracking the forward movement of the at least one point in the images of the series of images (E2), transforming the coordinates of the at least one selected and tracked point in the images into a set of coordinates corresponding to a set of points represented in a 3D reference frame associated with the vehicle to form at least one road line marking in said 3D reference frame (E3), detecting and tracking (E4) at least one object in the environment behind the vehicle from an image of the series of images, and determining (E5) the position of the detected and tracked object relative to the at least one formed line.

Description

[Technical field]

[0001] The present invention relates to the field of motor vehicles, and more particularly to a method for determining the position of an object relative to road line markings of a lane. [Background technology]

[0002] Recently, it has been known to equip motor vehicles with cameras for driver assistance purposes: for example, a camera located at the front of the vehicle may enable the detection of obstacles or road markings, while a camera located at the rear of the vehicle may enable the driver to be supported with backing up.

[0003] Road line markings may be detected in particular to prevent the vehicle from straying from the lane in which it is driving, or to monitor the driver's attention level during actual driving, or to guide the vehicle in an assisted or autonomous manner. In the latter case, it may be important or even necessary to identify the lane in which other vehicles are driving. In particular, when an emergency vehicle is approaching from behind, for example, it may be advantageous for a third party vehicle behind the vehicle to identify the lane in which the driver is driving in order to regain control.

[0004] In one known solution, line markings are detected in an image by identifying pixels that are the color of the line markings. However, this solution is unable to detect road line markings beyond a distance of about 100 meters, as the line markings are difficult to see beyond this distance. Thus, it is not possible to use this solution to identify the lane of a distant approaching vehicle in the image, for example a vehicle that is about 200 meters in the camera's field of view, since the road line markings are not visible at this distance.

[0005] Therefore, a solution is needed to at least partially remedy these shortcomings. Summary of the Invention [Problem to be solved by the invention]

[0006] One object of the present invention is to provide a reliable method for identifying the lane of another vehicle. Another object of the present invention is to provide a method for detecting an emergency vehicle coming from behind a vehicle. [Means for solving the problem]

[0007] To this end, one subject of the invention is a method for locating an object relative to road line markings of a road using a camera mounted on the rear of a motor vehicle, said vehicle including an electronic control unit, said method comprising the steps of: - generating, by a camera, a series of images of an environment behind the vehicle, said environment including at least one road line marking; - selecting, by an electronic control unit, at least one point on the road line marking in a lower portion of at least a first image of the sequence, said point being characterized by its coordinates in a reference frame of the images; - tracking, by an electronic control unit, the advancement of at least one selected point in an image of the sequence of images; - transforming, by the electronic control unit, coordinates of at least one selected point tracked in the image into a set of coordinates corresponding to a set of points represented in a three-dimensional reference frame associated with the vehicle to instantiate at least one road line marking in the three-dimensional reference system; - detecting and tracking, by an electronic control unit, at least one object located in the rear environment of the vehicle based on an image from the sequence of images; - determining, by the electronic control unit, a position of the detected and tracked object relative to the at least one materialized line marking. Includes.

[0008] Through the use of the rear camera, the invention in particular allows to track points as they progress through the images, i.e. as they pass from the bottom to the top of the images as the images of the sequence proceed. In contrast to the prior art solution of detecting points on the road line markings in each image, which entails the risk of errors at points far away (at the top of the rear camera's images), in order to materialize the road line markings, by tracking selected points close to the vehicle, i.e. at the bottom of the first image of the sequence, it is possible to ensure that the transformation between the image's reference frame and the three-dimensional frame is accurate from the start. In particular, by selecting points close to the vehicle and then tracking these points in the images, it is possible to ensure that the image's reference frame and the three-dimensional reference frame associated with the vehicle are in the same plane of the road, which is not necessarily the case when the imaged points are far from the vehicle. By selecting nearby points for tracking, it is possible to materialize the line markings accurately, correctly, reliably and in real time. This therefore makes it possible to ensure that objects at the rear of the vehicle are located with a high degree of certainty relative to the road line markings. The detected object may in particular be an emergency vehicle approaching the vehicle from behind. In this case, the present invention can identify and therefore act upon the lane that the emergency vehicle is driving in. Thus, when a vehicle implementing the present invention is in an autonomous driving mode, the on-board computer can change lanes if necessary to allow the emergency vehicle to pass.

[0009] In one embodiment, an object of the environment is detected and tracked in the series of images, and the method includes a step of converting, by an electronic control unit, at least one point on the tracked object of the environment in the images into at least one point represented in a three-dimensional reference frame to materialize the object of the environment in the three-dimensional reference frame.

[0010] In another embodiment, objects in the environment are detected and tracked directly in the 3D reference frame.

[0011] According to one aspect of the invention, the method further comprises the steps of detecting at least one line marking in the image, projecting the at least one realized line marking into a reference frame of the image, and determining an error between the detected line marking and the projected line marking.

[0012] The invention also relates to a computer program product, characterized in that it comprises a set of program code instructions which, when executed by one or more processors, configure the one or more processors to carry out the method described above.

[0013] The invention also relates to an electronic control unit for a motor vehicle, said electronic control unit comprising: - receiving a series of images of an environment rearward of the vehicle, the environment including at least one road line marking, the series of images being generated by a camera mounted on a rear of the vehicle; - selecting at least one point on a road line marking in the lower part of at least one image of the sequence, said point being characterized in each image by its coordinates in the reference frame of the image; - tracking the progress of at least one point in an image of the sequence of images; - transforming coordinates of at least one selected point tracked in the image into a set of coordinates corresponding to a set of points represented in a three-dimensional reference frame associated with the vehicle to instantiate at least one road line marking in the three-dimensional reference system; - detecting and tracking at least one object located in a rear environment of the vehicle based on an image of the sequence of images; - determining the location of the detected and tracked object relative to at least one materialized line marking; The present invention is configured to perform the following steps.

[0014] In one embodiment, the electronic control unit is configured to detect and track an environmental object in the sequence of images and transform at least one point on the environmental object tracked in the images into at least one point represented in a three-dimensional reference frame to materialize the environmental object in the three-dimensional reference frame.

[0015] As a variant or in addition, the electronic control unit is configured to directly detect and track objects in the environment in the three-dimensional reference frame.

[0016] Advantageously, the electronic control unit is configured to detect at least one line indication in the image, project at least one realized line indication into a reference frame of the image, and determine an error between the detected line indication and the projected line indication.

[0017] The invention also relates to a motor vehicle comprising a camera mounted at the rear of the vehicle and configured to generate a series of images of the rear environment of the vehicle, and an electronic control unit as presented above.

[0018] Further characteristics and advantages of the invention will appear more clearly on reading the following description, which is purely exemplary and should be read in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0019] [Figure 1] 1 illustrates diagrammatically an embodiment of a vehicle according to the invention; [Diagram 2] 2 shows a schematic diagram of an example of a first image in a sequence of images. [Diagram 3] 2 shows a schematic example of a second image of a sequence of images; [Figure 4] 2 shows a schematic example of a third image of the sequence of images. [Diagram 5] 2 shows a schematic example of a fourth image in a sequence of images. [Figure 6] 1 shows a schematic diagram of an example of a 3D map of roads. [Figure 7] 1 illustrates a schematic representation of an embodiment of the method according to the invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] 1 shows a schematic diagram of an example of a motor vehicle 1 according to the invention. The vehicle 1 comprises a camera 10 and an electronic control unit 20.

[0021] The camera 10 is mounted at the rear 1A of the vehicle 1, for example between the rear bumper 1B and the rear trunk 1C, and is configured to generate a series of images of the rear environment of the vehicle 1, in particular the road 100. As a variant, the camera 10 can be mounted in any other suitable place on the vehicle 1, for example behind the rear windshield.

[0022] The electronic control unit 20 is mounted on the vehicle 1 and is connected in a manner known per se via a bus or a data communication network (not shown) to the camera 10. As a variant, the electronic control unit 20 can be mounted in the same housing as the camera 10.

[0023] The electronic control unit 20 is arranged to receive, via a bus or a data communication network, a sequence of images of the rear environment of the vehicle 1 produced by the camera 10 .

[0024] The electronic control unit 20 is configured to select one or preferably two or more points on one or more road line markings in the received sequence of images and to track their progress, each point being characterized in each image of the sequence by its coordinates in a reference frame accompanying the image, the reference plane being two-dimensional and called the reference frame of the image. In particular, the electronic control unit 20 is configured to implement a detection algorithm stored in a memory area and to track the selected point between the images. There is no direct correspondence between the points detected on the road line markings of the two images. In contrast, its position in the subsequent image can be determined in a manner known per se, for example by a motion vector, which also allows to estimate the deviation from the detected line marking, in a manner known per se.

[0025] The detection algorithm is configured to identify areas of the first image I1 as road line markings, and where the paved road is greyer between the road line markings L1, L2, L3, L4, for example, white pixels are identified as road line markings.

[0026] The electronic control unit 20 is configured to convert the coordinates of each point tracked in the images into a set of coordinates corresponding to the points represented in a three-dimensional reference frame associated with the vehicle. Thus, a given point on the road line marking allows the road line marking to be materialized in said three-dimensional reference frame (also called "3D reference frame") over time, i.e. for the duration of the series of images, as it moves from one image to the next in the series. The 3D reference frame of the vehicle 1 is preferably associated with the rear of the vehicle 1. For example, the 3D reference frame of the vehicle 1 may have one of the rear axles of the vehicle 1 or the rear wheels of the vehicle 1 as an origin.

[0027] The electronic control unit 20 is configured to detect and track at least one object in the rear environment of the vehicle 1. This object may in particular be another vehicle, such as an emergency vehicle (ambulance, fire engine, police car, etc.), or a fixed element, such as a bridge, a tower, a sign or any structure whose position relative to one or more road line markings is relevant for the operation of the vehicle 1.

[0028] The electronic control unit 20 is configured to determine the location of objects in the environment relative to one or more materialized line markings.

[0029] In one embodiment, the electronic control unit 20 is configured to detect and track an object in the sequence of images and transform at least one point on the tracked object in the images into at least one point represented in a three-dimensional reference frame to materialize the object in the environment in said three-dimensional reference frame.

[0030] In another embodiment, the electronic control unit 20 is configured to directly detect and track objects in the environment in a three-dimensional reference frame. For example, it may use the information that the detected object is on the ground. More precisely, considering the ground to be flat, it is possible to convert a 2D point on the ground into a 3D point with vertical distance, horizontal distance and zero altitude. It is also possible to estimate the dimensions of the object to infer the distance given its size in the image.

[0031] The electronic control unit 20 is configured to detect one or more line indications in the image, project one or more materialized line indications from a three-dimensional reference frame into a reference frame of the image, and determine an error between the detected line indications and the projected line indications.

[0032] The electronic control unit 20 comprises a processor capable of executing an instruction set enabling these functions to be carried out, and a memory area enabling said instruction set to be stored.

[0033] Implementation Here, an example of a mounting form will be described with reference to FIGS.

[0034] 2-5 show one image I1, I2, I3, I4 of a series of images I1, I2, I3, I4, respectively, in chronological order. In the example described below with reference to Figs. 2-5, a vehicle 1 is driving on a road 100 including four road line markings L1, L2, L3, L4 that define three lanes in the direction of vehicle travel: a lane called "right" lane C1, a lane called "center" lane C2 and a lane called "left" lane C3. The direction of travel and overtaking are those of so-called right-hand traffic, i.e. those of most countries (France, Germany, USA, etc.). Thus, depending on the case, the vehicle usually drives in the right lane C1 or the center lane C2 and usually overtakes using the center lane C2 or the left lane C3.

[0035] 2 to 5, the vehicle 1 is an autonomous vehicle driving in the left lane C3. The first object OB1 is a third party vehicle driving in the center lane C2. The second object OB2 is an ambulance approaching the vehicle 1 from behind in the left lane C3.

[0036] In step E1, the camera 10 generates a sequence of images I1, I2, I3, I4 of the rear environment of the vehicle 1, in particular of the road line markings L1, L2, L3, L4 present in the images I1, I2, I3, I4, which are transmitted one by one periodically and in real time to the electronic control unit 20 via the data communication network of the vehicle 1.

[0037] Next, in step E2, the electronic control unit 20 first selects in the first image I1 first points P1-L2, P1-L3, P1-L4 in some or all of the road line markings depending on the desired accuracy and completeness. In this example, the first points P1-L2, P1-L3, P1-L4 are selected and determined in each of the three road line markings L2, L3, L4 that are furthest towards the right in the image I1, the line marking L1 being less relevant to the objects OB1, OB2 for which it is desired to identify the diagonal line. "Selection of a point in each of the three road line markings L2, L3, L4" means selection of a group of pixels in the image I1 that represent the road line markings L2, L3, L4.

[0038] The first points P1-L2, P1-L3, P1-L4 of the road line markings L2, L3, L4 are selected in the lower part of the first image Il, which represents the area of ​​the road 100 that is closest to the vehicle 1 and therefore can be considered to be on the same plane (flat area) as the vehicle 1.

[0039] To select points at the road line markings L1, L2, L3, L4, the electronic control unit 20 is configured to implement a detection algorithm that is stored in a memory area and configured to identify areas of the image Il as road line markings, for example if the paved road is greyer between the road line markings L1, L2, L3, L4 then white pixels are identified as road line markings.

[0040] Then, again with regard to step E2, the electronic control unit 20 tracks the points detected in the first image I1 in the subsequent images I2, I3, I4 of the sequence, preferably in each image I2, I3, I4. Thus, in the second image I2, the electronic control unit 20 identifies three points P2-L2, P2-L3, P2-L4 that correspond to the three points P1-L2, P1-L3, P1-L4 initially detected in the first image I1 and that correspond to the same physical points on the road 100. The electronic control unit 20 then identifies three points P3-L2, P3-L3, P3-L4 in the third image I3 that correspond to the three points P2-L2, P2-L3, P2-L4 identified in the second image I2 and that correspond to the same physical points on the road 100. The electronic control unit 20 identifies three points P4-L2, P4-L3, P4-L4 in the fourth image I4 that correspond to the three points P3-L2, P3-L3, P3-L4 identified in the third image I3 and that correspond to the same physical points on the road 100.

[0041] The points P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4, which are tracked between two successive intervals I1, I2, I3, I4 of the series and which represent the same physical point on the road 100, are identified in the images in a known manner, for example using motion vectors, in a manner which is known and will not be described in further detail here.

[0042] Each point P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4 is characterized in the images I1, I2, I3, I4 in which it appears by its coordinates in the reference frame of images I1, I2, I3, I4.

[0043] In step E3, the electronic control unit 20 transforms the coordinates of the points detected and tracked in the image into a plurality of coordinate sets corresponding to a plurality of points P1-L2-3D, P1-L3-3D, P1-L4-3D, P2-L2-3D, P2-L3-3D, P2-L4-3D, P3-L2-3D, P3-L3-3D, P3-L4-3D, P4-L2-3D, P4-L3-3D, P4-L4-3D, represented in a 3D reference frame associated with the vehicle 1 to materialize the road line markings L2, L3, L4 in said 3D reference frame, then denoted as L2-3D, L3-3D, L4-3D as shown in FIG. 6.

[0044] This 2D-3D transformation is made possible by using matrix-based translation and rotation transformations as described below, and by a planar assumption being made for the points on the road 100 close to the vehicle 1. In other words, the points are defined relative to the same plane of the road 100, which plane is in the same plane as the rear of the vehicle 1 in at least the first image I1, and therefore the assumption that the two points are in the same plane as the vehicle 1 can be made to ensure that the coordinate transformation of the detected points between the image (2D) reference frame and the (3D) reference frame associated with the vehicle 1 is mathematically accurate by reducing errors, as detailed in step E3.

[0045] The transformation of the 2D points to 3D points is performed in a known manner using the position of the camera 10 of the vehicle 1 relative to a reference point associated with the vehicle 1 (e.g. the ground below the center of the rear axle), which is characterized by a first rotation matrix and a second translation matrix. Then, to transform the 3D points between two given points in time (two images), it is necessary to know the movement of the reference frame associated with the vehicle 1, which is characterized between these two points in time by a second rotation matrix and a second translation matrix.

[0046] The transformation of step E3 can be carried out each time a point is selected or identified during tracking, ie each time the electronic control unit 20 receives each image.

[0047] In step E4, the electronic control unit 20 detects and tracks the first object OB1 and the second object OB2 in each image I1, I2, I3, I4 in parallel with steps E1, E2 and E3.

[0048] For example, the electronic control unit 20 transforms all or some of the pixels corresponding to the objects OB1 and OB2 in the images I1, I2, I3, I4 into points represented in the 3D reference frame and positions them relative to the road line markings L2, L3, L4. As a variant, the objects OB1, OB2 can be detected and tracked directly in the 3D reference frame, for example as described above.

[0049] In step E5, continuing from step E4 and in parallel with steps E1, E2 and E3, the electronic control unit 20 determines the positions of the objects OB1 and OB2 relative to the three line representations L2, L3, L4 realized during steps E1, E2 and E3, for example each time an image I1, I2, I3, I4 is received, and determines the lane in which the first object OB1 is moving and the lane in which the second object OB2 is moving.

[0050] Advantageously, in optional step E6, the road line markings L2, L3, L4 may be detected in the images I1, I2, I3, I4 by the electronic control unit 20, for example using the detection algorithm described above, which then projects the road line markings L2, L3, L4 realized in the 3D reference frame into the 2D reference frame of the image, then determines an error between each detected line marking and each corresponding projected line marking and determines a confidence index of the object's position relative to the road line markings L2, L3, L4.

[0051] In the above example, only the point detected in the first image I1 is tracked in the subsequent images I2, I3, I4, however, other initial points can also be selected in the images I2, I3, I4 following the first image I1 and the method repeated for these points in parallel with the method for the point selected in the first image I1, thereby allowing a better localization of the object relative to the road line markings L2, L3, L4.

Claims

1. A method for identifying the position of an object (OB1, OB2) relative to road markings (L1, L2, L3, L4) of a road (100) using a camera (10) mounted at the rear of a motor vehicle (1), wherein the vehicle (1) includes an electronic control unit (20), and the method comprises: - A step (E1) of generating, by the camera (10), a series of images of the rear environment of the vehicle (1), the environment including at least one road marking (L1, L2, L3, L4), step (E1), - By the electronic control unit (20), at least one point (P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4) on the road markings (L1, L2, L3, L4) at the lower part of at least a first image (I1) of the series, wherein in each image (I1, I2, I3, I4), at least one point (P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4) characterized by its coordinates in the reference frame of the image (I1, I2, I3, I4) is selected, and by the electronic control unit (20), a step (E2) of tracking the advancement of the at least one point (P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4) in the series of images (I1, I2, I3, I4), - By the electronic control unit (20), converting the coordinates of the at least one selected point (P1-L2, P1-L3, P1-L4, P2-L2, P2-L3, P2-L4, P3-L2, P3-L3, P3-L4, P4-L2, P4-L3, P4-L4) tracked in the images (I1, I2, I3, I4) into a plurality of coordinate sets corresponding to a plurality of points represented in a three-dimensional reference frame associated with the vehicle (1), to materialize the at least one road marking (L1, L2, L3, L4) in the three-dimensional reference system, step (E3), - A step (E4) of detecting and tracking at least one object (OB1, OB2) located in the rear environment of the vehicle (1) based on the images (I1, I2, I3, I4) among the series of images by the electronic control unit (20). - A step (E5) of specifying the positions of the detected and tracked objects (OB1, OB2) with respect to the at least one embodied line marking by the electronic control unit (20). A method comprising the above steps. **Claim 2** The objects (OB1, OB2) of the environment are detected and tracked in the images (I1, I2, I3, I4) of the series. The method includes a step of converting at least one point on the objects (OB1, OB2) of the environment tracked in the images (I1, I2, I3, I4) into at least one point represented in the three-dimensional reference frame by the electronic control unit (20) to embody the objects (OB1, OB2) of the environment in the three-dimensional reference frame. The method according to claim 1. **Claim 3** The objects (OB1, OB2) of the environment are directly detected and tracked in the three-dimensional reference frame. The method according to claim 1. **Claim 4** The method according to claim 1 further includes a step of detecting at least one line marking (L1, L2, L3, L4) in the images (I1, I2, I3, I4), a step of projecting the at least one embodied line marking onto the reference frame of the images (I1, I2, I3, I4), and a step of specifying the error between the detected line marking (L1, L2, L3, L4) and the projected line marking. **Claim 5** A computer program product, characterized by including a set of program code instructions that, when executed by one or more processors, configure the one or more processors to implement the method according to any one of claims 1 to 4. **Claim 6** An electronic control unit (20) for an automotive vehicle (1), - Receiving a series of images (I1, I2, I3, I4) of the rear environment of the vehicle (1) generated by a camera (10) mounted at the rear of the vehicle (1), wherein the environment includes at least one road line marking (L1, L2, L3, L4). - Selecting at least one point on the road line markings (L1, L2, L3, L4) at the lower part of at least one of the series of images (I1, I2, I3, I4), wherein the point is characterized by its coordinates in the reference frame of the image (I1, I2, I3, I4) in each image (I1, I2, I3, I4). - Tracking the movement of the at least one point in the images (I1, I2, I3, I4) of the series of images (I1, I2, I3, I4). - Converting the coordinates of the at least one selected point tracked in the images (I1, I2, I3, I4) into a plurality of coordinate sets corresponding to a plurality of points represented in a three-dimensional reference frame associated with the vehicle (1), thereby embodying the at least one road line marking (L1, L2, L3, L4) in the three-dimensional reference system. - Detecting and tracking at least one object (OB1, OB2) located in the rear environment of the vehicle (1) based on the images (I1, I2, I3, I4) of the series of images (I1, I2, I3, I4). - Identifying the positions of the detected and tracked objects (OB1, OB2) relative to the at least one embodied line marking. An electronic control unit (20) configured to perform the above.

7. The electronic control unit (20) according to claim 6, configured to detect and track the objects (OB1, OB2) in the environment in the images (I1, I2, I3, I4) of the series, and convert at least one point on the objects (OB1, OB2) in the environment tracked in the images (I1, I2, I3, I4) into at least one point represented in the three-dimensional reference frame, thereby embodying the objects (OB1, OB2) in the environment in the three-dimensional reference frame.

8. The electronic control unit (20) according to claim 6, configured to directly detect and track the objects (OB1, OB2) in the environment in the three-dimensional reference frame.

9. The electronic control unit (20) according to claim 6, configured to detect the at least one line marking (L1, L2, L3, L4) in the image (I1, I2, I3, I4), project the at least one embodied line marking onto the reference frame of the image (I1, I2, I3, I4), and identify an error between the detected line marking (L1, L2, L3, L4) and the projected line marking.

10. A motor vehicle (1) comprising a camera (10) mounted at the rear of the vehicle (1) and configured to generate a series of images (I1, I2, I3, I4) of the rear environment of the vehicle (1), and an electronic control unit (20) according to any one of claims 6 to 9.

Images