Apparatus and method for representing the yaw orientation of a vehicle as seen on an image - Patents.com

Abstract

An apparatus and method for representing the yaw direction of a vehicle as seen on an image includes the steps of obtaining a position within the image of a rectangular bounding box (200) surrounding the vehicle (201); obtaining a position of a vertical dividing line (202) of the bounding box configured to be located at a corner of the vehicle separating a first portion of the vehicle and a second portion of the vehicle, the first portion and the second portion being the front, rear, or side of the vehicle; determining a signed horizontal shift of a horizontal position associated with the vertical dividing line relative to a horizontal reference position associated with the bounding box; and normalizing the signed horizontal shift to obtain a value θ used to represent the attitude of the vehicle.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more precisely, to the detection of vehicles visible in an image.

Background Art

[0002] The detection of vehicles on an image acquired by a camera mounted on a vehicle has become an important problem for improving a semi-autonomous or autonomous method for controlling a vehicle such as an automobile. The detection of vehicles is often completed by tracking the vehicles detected on a plurality of consecutive image frames acquired by these cameras.

[0003] Currently, a reliable technology for detecting and tracking vehicles has become essential.

[0004] In recent years, the use of machine learning techniques (i.e., deep neural networks) has facilitated the detection and tracking of objects.

[0005] In order to detect and track vehicles, it is necessary to use representations of the positions and orientations of these vehicles, and the positions and orientations form what is called the pose of these vehicles. These representations include a set of values such as coordinates, angles, etc. For example, the representation is a computer representation that can be used in a computer program to represent the position and orientation of a real-world vehicle.

[0006] Generally, in order to limit the computational load, it is desirable to use a representation having a limited number of variables.

[0007] In computer vision, the representation of objects in 3D space is often done using a 3D bounding box (3DBB), which is a convenient representation of the 3D bounding volume containing the object. Generally, a 3DBB has nine degrees of freedom (DOF): three for dimensions / range (width, height, and depth), three for position (X, Y, Z), and three for direction.

[0008] In automotive applications, since the target object (e.g., a vehicle) is likely located on the ground, the vehicle's roll and pitch angles relative to the ground can be considered zero, reducing the number of DOFs by two. Furthermore, the bottom of the vehicle's 3D bounding box can be considered to belong to the ground, allowing another DOF to be ignored. For this reason, considering only six DOFs for 3DBBs in the context of vehicle detection is useful.

[0009] The paper "Monocular Object Direction Estimation Using Riemann Regression and Classification Networks" (Mahendran S. et al., archive preprint, archive: 1807.07226(2018)) discloses the use of only 6DOF for 3DBB.

[0010] Furthermore, the 3D orientation can be represented in several different ways, namely by Euler angles (azimuth, elevation, and camera tilt, or yaw, pitch, and roll), by axis angles, for example, by quaternions (a four-dimensional representation with additional unit norm constraints, but avoiding unpleasant transitions between 0° and 360°), as disclosed in the literature "Posenet: A Convolutional Network for Real-Time 6-DoF Camera Repositioning" (Kendall A., Grimes M., Proceedings of the IEEE International Conference on Computer Vision, 2015), "Modeling Uncertainty in Deep Learning for Camera Repositioning" (Kendall A., Cipolla R., IEEE International Conference on Robotics and Automation 2016 (ICRA), IEEE, 2016), or "Geometric Loss Function for Camera Pose Regression Using Deep Learning" (Kendall A., Cipolla R., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017).

[0011] Typically, as described in the aforementioned literature "Monocular Object Orientation Estimation Using Riemann Regression and Classification Networks," "3D Bounding Box Estimation Using Deep Learning and Geometry" (Mousavian A. et al., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017 (pp. 7074-7082)), "Integrated Framework for Multi-View, Multi-Class Object Pose Estimation" (Li Ch., Bai J., Hager GD, Proceedings of the European Conference on Computer Vision (ECCV), 2018), "Densereg: Fully Convolved Dense Shape Regression in Real-World Environments" (Alp Guler R. et al., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017), and "Fusion of Vision and Robotics: KITTI Dataset" (Geiger A. et al., International Journal of Robotics Research 32.11(2013):1231-1237), the orientation of a vehicle is usually defined using only the yaw angle.

[0012] It has also been proposed to use a projection of 3DBB from 3D space to 2D space as a representation. These projections are often called 2.5D bounding boxes or 2.5DBB. For example, in section 3.3 of the paper "Vehicle Detection and Attitude Estimation for Autonomous Driving" (Novak L. Diss., Master's Thesis, Czech Technical University in Prague, 2017, citation, 2017), 2.5DBB is represented by seven numbers, namely the three boundary lines of the "bottom of the car" (X, Y for the left front, right front, and left rear).

[0013] In the paper "Densebox: Integration of Landmark Location and End-to-End Object Detection" (Lichao H. et al., archive preprint, archive: 1509.04874(2015)), 2.5 DBB is used for eight landmarks (specific parts) of a car.

[0014] Another 2.5 DBB method is disclosed in the paper "Multi-view 3D Object Detection Network for Autonomous Driving" (Chen X. et al., IEEE CVPR, 2017). This solution proposes using a dividing line to separate a single rectangle (or bounding box) into two between the visible front / rear and visible sides of the vehicle. A binary value is used to indicate which side of the dividing line is the side or front / rear of the vehicle, but this solution has been observed by the inventors to be unsatisfactory as it causes instability in tracking. In fact, using a binary value can cause the dividing line to jump from one side to the other in situations where the vehicle is viewed almost entirely from one side (e.g., from the rear with the side barely visible), as is typically the case in standard driving situations.

[0015] The above solutions are too complex or unstable for performing tracking and therefore do not provide satisfactory results. [Overview of the project]

[0016] This disclosure overcomes one or more defects of the prior art by proposing a method for representing the yaw direction of a vehicle as seen in an image, wherein the yaw direction of the vehicle is configured to be used, for example, in a subsequent tracking step (e.g., tracking of the yaw direction of the vehicle), the method comprising: obtaining the position in an image of a rectangular bounding box surrounding the vehicle; obtaining the position of a vertical dividing line of the bounding box, configured such that the vertical dividing line is located at a corner of the vehicle separating a first part of the vehicle from a second part of the vehicle, wherein the first and second parts are the front, rear, or side of the vehicle, and at least one of the first and second parts is visible in the image; determining a signed horizontal shift of a horizontal position associated with the vertical dividing line (e.g., the center of the visible first part of the vehicle, or one-third of the part measured from any side) with respect to a horizontal reference position associated with the bounding box (e.g., the center of the bounding box); and normalizing the signed horizontal shift to obtain a value θ used to represent the yaw direction of the vehicle.

[0017] To be associated means that the horizontal position can be defined using the horizontal position of the dividing line and, for example, the position of the bounding box (e.g., the position of the corners of the bounding box or the position of the left and right edges of the bounding box). Furthermore, the horizontal reference position can be defined using the position of the bounding box (e.g., the position of the corners of the bounding box or the position of the left and right edges of the bounding box).

[0018] Furthermore, when a vehicle is visible in an image, two parts, such as the front and side, or the rear and side, are visible. Sometimes only one part is visible, in which case that part is either the front, rear, or side.

[0019] The yaw direction can be used to represent the attitude of a vehicle. This representation of attitude is a set of data that describes the vehicle's position in space around the camera used to acquire the image. It is common to represent the vehicle as if it were placed on a plane (e.g., a road surface), in which case the vehicle's roll and pitch directions are ignored, and the yaw direction becomes the only direction necessary to describe the vehicle's attitude.

[0020] The inventors of the present invention have observed that a single bounding box with dividing lines indicating the location of a visible corner of a vehicle (front right, front left, rear right, or rear left corner) can be used to perform vehicle tracking.

[0021] Instead of using binary values ​​as described above in the literature "Multi-view 3D Object Detection Network for Autonomous Driving," the present invention proposes using a signed shift between a reference and a position associated with a dividing line. This allows for more information to be obtained to stabilize the detection performed in subsequent frames (i.e., when tracking is performed).

[0022] The value θ can be used as input in subsequent tracking steps. In fact, the value θ facilitates the use of any temporal process, such as any temporal filtering, as it has been observed that temporal filtering improves tracking using 2.5 DBB.

[0023] Note that the normalization step makes it possible to obtain a value of θ that is independent of, for example, the size of the vehicle or the size of the bounding box.

[0024] In addition, normalization makes the value θ dimensionless.

[0025] According to a particular embodiment, the horizontal reference position associated with the bounding box is either the horizontal center of the bounding box or one side of the bounding box.

[0026] This position has been observed to be a highly reliable position that does not change much in a critical situation such as when the vehicle is viewed almost from the front or rear.

[0027] According to a particular embodiment, the position associated with the vertical dividing line is the horizontal center of the first part or the second part.

[0028] According to a particular embodiment, the value θ used to represent the vehicle's attitude is calculated as θ = d / W, where d is a signed shift and W is a normalization factor associated with the bounding box (defined, for example, using the positions of the corners of the bounding box or the positions of the left and right ends of the bounding box).

[0029] According to a particular embodiment, W is the width of the bounding box.

[0030] Since both d and W are measured in the same unit, θ is dimensionless.

[0031] According to a particular embodiment, the method includes the step of determining the vehicle's attitude using a representation of the yaw direction.

[0032] According to a particular embodiment, the method further includes the step of determining the vehicle's yaw angle using the value θ.

[0033] This yaw angle can be used in any vehicle's deep learning-based detection method to improve its training.

[0034] Also, the value θ is converted to a value of the yaw angle based on knowledge of the vehicle's aspect ratio (width / length) or based on a predetermined aspect ratio that is satisfied for a plurality of vehicles.

[0035] According to a particular embodiment, the step of determining the yaw angle includes using a given aspect ratio associated with the vehicle.

[0036] According to a particular embodiment, the step of determining the yaw angle includes determining a function that approximates the yaw angle, taking a value θ and an aspect ratio associated with the vehicle as input, the function being a piecewise linear function or a heuristically constructed function fitted to a given set of points relating the value of the yaw angle to the value θ and the aspect ratio.

[0037] The present invention also provides a method for tracking the yaw direction of a vehicle in a series of images, the method comprising determining a value θ obtained by the above method for each image in the series of images.

[0038] The present invention also proposes an apparatus for representing the yaw direction of a vehicle as seen in an image, wherein the yaw direction of the vehicle is configured to be used, for example, in a subsequent tracking step, and the apparatus comprises: a module for obtaining the position in an image of a rectangular bounding box surrounding the vehicle; a module for obtaining the position of a vertical dividing line of the bounding box, configured to be positioned at the corners of the vehicle separating a first part of the vehicle from a second part of the vehicle, wherein the first and second parts are the front, rear, or side of the vehicle; a module for determining a signed horizontal shift of a horizontal position associated with the vertical dividing line (e.g., the center of the visible first part of the vehicle) with respect to a horizontal reference position associated with the bounding box (e.g., the center of the bounding box); and a module for normalizing the horizontal shift to obtain a value θ used to represent the yaw direction of the vehicle.

[0039] This device is configured to perform any embodiment of the method described above.

[0040] The present invention also provides a vehicle equipped with the above-described device.

[0041] In one particular embodiment, the steps of the method are determined by instructions in a computer program.

[0042] Therefore, the present invention also covers computer programs for performing the steps of the above-described method when the program is executed by a computer.

[0043] This program can use any programming language and can take the form of source code, object code, code that is an intermediate between source code and object code, such as a partially compiled form, or any other desired form.

[0044] The present invention also covers computer-readable information media containing computer program instructions as described above.

[0045] An information medium is any entity or device capable of storing a program. For example, the medium may include ROM, such as a CD-ROM or microelectronic circuit ROM, or magnetic storage means, such as a diskette (floppy disk) or hard disk.

[0046] Alternatively, the information medium may be an integrated circuit with a program embedded in it, and the integrated circuit is configured to perform or be used to perform the method. [Brief explanation of the drawing]

[0047] [Figure 1] Figure 1 illustrates the difficulties associated with using dividing lines in the conventional method. [Figure 2] Figure 2 shows the shift measurement used in one example. [Figure 3] Figure 3 is a schematic diagram of the steps of an example method. [Figure 4] Figure 4 shows an example vehicle. [Modes for carrying out the invention]

[0048] The features and advantages of this disclosure will become apparent from the following description of specific embodiments of this disclosure, which are given for illustrative purposes only and not limitation, with reference to the accompanying drawings.

[0049] The following describes how to obtain a representation of the yaw direction of a vehicle visible in an image, which can be used, for example, to represent or express the attitude of the vehicle. This attitude is relative to the camera that acquired the image and includes information about the vehicle's position and orientation in the image, or the vehicle's position and orientation in space relative to the camera.

[0050] Typically, the camera that acquires the images is embedded in another vehicle.

[0051] Furthermore, determining the yaw direction enables vehicle tracking. This tracking includes determining the evolution of this attitude over time.

[0052] Figure 1 illustrates the operation of a conventional method, particularly the method described in the literature "Multi-view 3D Object Detection Network for Autonomous Driving." This figure shows two different detection times, T1 and T2.

[0053] At a first detection time T1, the bounding box 100 is determined to surround the vehicle 101 as seen in the image acquired by the camera. The dimensions and position of the bounding box 100 relative to the vehicle 101 can be determined using machine learning methods such as deep neural networks in a manner known to those skilled in the art.

[0054] Furthermore, using the method described in the document "Multi-view 3D Object Detection Network for Autonomous Driving," the position (in this case, horizontal position) of the dividing line 102 that separates the visible front / rear and visible sides of the vehicle is determined.

[0055] At the initial detection line T1, the dividing line is determined to be on the left side of the bounding box. However, considering the position of the vehicles, typically the position of the vehicle in front of the camera-equipped vehicle, it is difficult to determine the position of the dividing line.

[0056] When binary values ​​are used to indicate which side of the dividing line is the front / rear, it was observed that the dividing line jumps between the left and right sides when the vehicle is viewed from the rear / front. For example, at time T2, dividing line 102 moves to the right side of the bounding box.

[0057] This instability is detrimental when performing vehicle attitude tracking.

[0058] Instead, this disclosure proposes introducing a value calculated based on a measured signed shift, as shown in Figure 2. In this figure, the position of a rectangular bounding box 200 is determined to surround the vehicle 201 visible in the image. The position of the bounding box means its dimensions and placement in the image, or the position of each corner in the image.

[0059] The position of the vertical dividing line 202 is obtained and configured to be at the corner of the vehicle separating the first part of the vehicle from the second part of the vehicle, and the first and second parts of the vehicle are either the front, rear, or side of the vehicle. In the illustrated example, the first part is the front of the vehicle and is shown in gray, while the second part is the left side of the vehicle.

[0060] Preferably, the first part is the front or rear of the vehicle, and the second part is the left or right side of the vehicle. In practice, one or two parts are normally visible, and if two are visible, they are always pairs selected from the group consisting of the front and right side, the front and left side, the rear and right side, and the rear and left side.

[0061] For example, the proposed method further obtains the position of the reference horizontal position within the bounding box, i.e., the center position 203, and the center 204 of the first part (in this case, the part where the front part of the vehicle is visible).

[0062] The positions of bounding boxes and dividing lines can be obtained using a deep network trained to detect vehicles in an image (as bounding boxes). It is also possible to train a deep neural network to predict the positions of dividing lines.

[0063] Please note that the positions mentioned above are horizontal positions related to the vertical line.

[0064] This makes it possible to determine the signed shift d of the center position 203 of the bounding box relative to the center position of the first part.

[0065] Next, this signed shift can be normalized to obtain a value θ calculated as θ = d / W using the width W of the bounding box 200.

[0066] This value θ can then be used in representing the vehicle's yaw direction, in the subsequent tracking step, and even in the time filtering step.

[0067] It was observed that using this value θ prevents the dividing line from jumping from one side to the other when the vehicle is viewed from the front or rear.

[0068] Figure 3 schematically shows the steps of a method for representing the yaw direction of a vehicle visible in an image.

[0069] These steps can be performed by a device such as the one described below with reference to Figure 4. Typically, a device equipped with a processor can perform this method.

[0070] These steps are performed on images acquired, for example, by a camera embedded in another vehicle. This image shows a vehicle like the one in Figure 2.

[0071] In the first step S01, the position of a bounding box, such as bounding box 200, is obtained. This step can be performed using machine learning techniques such as neural networks.

[0072] In the second step S02, the position of the dividing line 202 is obtained. This step can also be performed using machine learning techniques.

[0073] Next, in step S03, the signed shift d is determined. This step is a calculation step.

[0074] Next, in step S04, the value θ can be calculated using the equation θ = d / W described above.

[0075] It should be noted that this method may further include temporal filtering of θ and subsequent tracking steps of the vehicle's yaw direction or attitude (using θ, the position of the bounding box, and the position of the dividing line). In addition, linear yaw (or angular yaw if the vehicle's aspect ratio is known) can be determined.

[0076] For example, the value θ can be used to estimate the actual yaw angle φ of the vehicle (measured, for example, in radians). To accurately estimate φ, the aspect ratio α = w of the vehicle is used. v / l v This is necessary, and here, w v This is the actual width of the vehicle, l vθ,α is the actual length of the vehicle (all measured in meters, not pixels). The proposed method is not sensitive to the exact value of the aspect ratio α, so it can provide an approximate aspect ratio. For example, α can be "estimated" from the type of vehicle (i.e., long / short passenger cars, trucks, buses, etc.) using the same vehicle detector that provides the bounding box and dividing lines. Two alternatives are proposed to estimate / approximate the yaw angle function φ(θ,α). The first alternative is based on linearization. The desired function φ(θ,α) is calculated using a set of three-dimensional points calculated using bounding boxes of vehicles that are synthetically and randomly generated using a fixed camera configuration (camera resolution and field of view).

number

[0077] The second alternative is based on an analytical approximation. The desired function φ(θ,α) is an analytical function

number

number

[0078] Please note that parameters c1 and c2 may vary depending on the camera.

[0079] Figure 4 shows a vehicle 400 equipped with a device 401 configured to carry out a method such as the one disclosed with reference to Figure 3.

[0080] The vehicle further includes a camera 402 configured to acquire images of the road on which the vehicle is traveling, and the camera 402 typically acquires images of other vehicles present on the road, such as vehicle 201 shown in Figure 2.

[0081] The device 401 includes a processor 403 and a non-volatile memory 404 containing computer instructions for performing the above-described method on images acquired by the camera 402.

[0082] More precisely, the computer instruction includes an instruction 405 that executes step S01 when executed by processor 403, an instruction 406 that executes step S02 when executed by processor 403, an instruction 407 that executes step S03 when executed by processor 403, and an instruction 408 that executes step S04 when executed by processor 403.

[0083] Instructions 405-408, in combination with processor 403, form a module of device 401.

[0084] Although the present invention has been described above with reference to specific embodiments, it will be understood that the present invention is not limited to the details of specific embodiments. Many variations, modifications, and developments can be made to the embodiments described above within the scope of the appended claims.

Claims

1. 1. A method for representing the yaw direction of a vehicle as seen on an image, comprising: the yaw direction of the vehicle is adapted to be used in a subsequent tracking step, for example; The method comprises: A step (S01) of obtaining the position in the image of a rectangular bounding box (200) surrounding the vehicle (201); obtaining (S02) a position of a vertical dividing line (202) of the bounding box configured to be located at a corner of the vehicle separating a first portion of the vehicle and a second portion of the vehicle, the first portion and the second portion being the front, rear or side of the vehicle, and at least one of the first portion and the second portion being visible on the image; - determining a signed horizontal shift of the horizontal position associated with said vertical dividing line relative to a horizontal reference position associated with said bounding box (S03); normalizing the signed horizontal shift to obtain a value θ used to represent the yaw direction of the vehicle (S04); A method comprising:

2. The method of claim 1 , wherein the horizontal reference position associated with the bounding box is the horizontal center (203) of the bounding box or one side of the bounding box.

3. The method of claim 1 or 2, wherein the location associated with the vertical dividing line is the horizontal center (204) of the first portion or the second portion.

4. 4. The method according to claim 1, wherein the value θ used to represent the pose of the vehicle is calculated as θ=d / W, where d is the signed shift and W is, for example, a normalization factor associated with the bounding box.

5. The method of claim 5 , wherein W is the width of the bounding box.

6. A method according to any one of claims 1 to 5, comprising determining the attitude of the vehicle using the representation of yaw orientation.

7. The method of claim 1 , further comprising the step of using the value θ to determine a yaw angle of the vehicle.

8. The method of claim 7 , wherein determining the yaw angle includes using a given aspect ratio associated with the vehicle.

9. determining the yaw angle includes determining a function that approximates the yaw angle, the function taking as input the value θ and the aspect ratio associated with the vehicle; 9. The method of claim 8, wherein the function is a plane fitted to a set of given points or a heuristically constructed function relating values ​​of yaw angle to values ​​θ and aspect ratios.

10. 1. A method for tracking the yaw orientation of a vehicle in a sequence of images, comprising: A method comprising determining, for each image of the sequence of images, the value θ obtained by the method of any one of claims 1 to 9.

11. 1. An apparatus for representing the yaw direction of a vehicle as seen on an image, comprising: the yaw direction of the vehicle is adapted to be used in a subsequent tracking step, for example; The device, a module (403, 405) for obtaining the position in the image of a rectangular bounding box surrounding the vehicle; a module (403, 406) for obtaining the position of a vertical dividing line of the bounding box, the module (403, 406) being configured to be placed at a corner of the vehicle separating a first portion of the vehicle and a second portion of the vehicle, the first portion and the second portion being the front, rear or side of the vehicle; a module (403, 407) for determining a signed horizontal shift of a horizontal position associated with said vertical dividing line relative to a horizontal reference position associated with said bounding box; a module (403, 408) for normalizing the signed horizontal shift to obtain a value θ used to represent the yaw direction of the vehicle; An apparatus comprising:

12. A vehicle comprising the device according to claim 11.

13. A computer program comprising: A computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 10 when said program is executed by a computer.

14. A computer-readable recording medium, A recording medium having stored thereon a computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 10.

Images