Method and system for three-dimensional image reconstruction of a vehicle and vehicle with the system

Abstract

Method for three-dimensional image reconstruction of a vehicle (600), comprising the steps - capturing (401) an image of at least one vehicle with a camera (501), - detecting (402) a first rectangular border of the entire vehicle on the image to obtain a first rectangle (203), - detecting (403) a second rectangular border of a side of a vehicle (600) on the image to obtain a second rectangle (202), - determining (404) whether the first and second rectangles are boundaries that concern the same vehicle, - determining (405) whether a lateral orientation of the vehicle (600) can be assigned from the two rectangles if the first and second rectangles are borders relating to the same vehicle, - Determination (406) of the side orientation, if a side orientation can be determined, - performing (407) a three-dimensional reconstruction of the vehicle from the first rectangle, the second rectangle and the side orientation.

Description

[0001] The invention relates to a method for three-dimensional image reconstruction of a vehicle. Furthermore, the invention relates to a system configured to carry out the method and to a vehicle.

[0002] A key aspect in the field of autonomous driving and driver assistance systems is the recognition of traffic scenes. This requires the detection of vehicles in images captured by cameras in or on the vehicle.

[0003] Vehicle detection in traffic scenes based on camera data typically uses two-dimensional detection of so-called bounding boxes, i.e., rectangles that encompass the object. There are also approaches for pixel-precise vehicle detection. However, to capture the entire street scene, three-dimensional structures are required.

[0004] Current approaches therefore focus on detecting such 3D structures of objects. These include approaches based on other sensors, such as LIDAR (Light Detection and Ranging), or stereo approaches based on the use of more than one camera.

[0005] The approaches to estimate a three-dimensional structure from image data require a considerable additional annotation effort of the data in order to annotate, for example, an accurate 3-D structure, such as tires, window panes, etc., in the image, as in the so-called “DeepManta” approach.

[0006] Simple and fast approaches with bounding boxes are based either on a single 2-D bounding box and a so-called “ground plane assumption”, where the vehicle is on a non-inclined plane, or on multiple bounding boxes for the different side views.

[0007] From the article “A statistical method for 3D object detection applied to faces and cars” by H. Schneiderman et al., published in 2000 in the Proceedings IEEE Conference on Computervision and Pattern Recognition, CVPR 2000 (Cat. No. PR0662), IEEE, 2000, pp. 746-751, a statistical method for 3D object detection based on a product of histograms has become known, where each histogram represents a common statistic of a subset of wavelet coefficients and their position on the object.

[0008] The object of the invention is to provide a simple and effective detection of three-dimensional objects in the field of autonomous driving or driver assistance systems.

[0009] The object is achieved by the subject matter of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims, the following description, and the figures.

[0010] The present invention addresses the reconstruction of the three-dimensional structure of vehicles based on the use of a single camera and individual images. In contrast to stereo approaches or the use of LIDAR sensors, this approach is cost-effective.

[0011] According to one aspect, a method for three-dimensional image reconstruction of a vehicle is provided, comprising the steps of: - Capturing an image of at least one vehicle with a camera, - capturing a first rectangular outline of the entire vehicle on the image to obtain a first rectangle, - detecting a second rectangular border of a side of a vehicle, e.g. the front or rear side or side surface of a vehicle from the camera's perspective, in the image to obtain a second rectangle, - Determine whether the first and second rectangles are boundaries that concern the same vehicle, - Determine whether a side orientation of the vehicle can be assigned from the two rectangles if the first and second rectangles are borders that concern the same vehicle, - Determination of the side orientation, if a side orientation can be determined, - Performing a three-dimensional reconstruction of the vehicle from the first rectangle, the second rectangle and the side orientation.

[0012] The method is therefore based on an image that was preferably taken with just one camera and is then evaluated. The visible sides of objects such as vehicles are detected, and a rectangle is constructed around these surfaces. The visible sides are, for example, the entire front of an oncoming vehicle, including the wheels and windshield, or the entire rear of a vehicle in front if the camera is facing forward. Another rectangle is placed around the entire vehicle. This includes the front and rear sides visible in the image, as well as the visible side surfaces.

[0013] However, it is also possible to form a rectangle over a non-visible side from, for example, a visible edge.

[0014] The detection of a second rectangular border of a side of a vehicle can, for example, relate to a) the front side of a vehicle from the camera's perspective, b) the rear side of a vehicle from the camera's perspective, c) the right side or side surface of a vehicle (600) from the camera's perspective, or / and d) the left side or side surface of a vehicle (600) from the camera's perspective (600) in the image, in order to obtain a second rectangle (202).

[0015] Thus, other or even further, i.e. more than two, rectangles can be detected, such as one that borders one side of the vehicle or one that borders the rear side of the vehicle, which is detected, for example, from the rear visible, vertical edge of the vehicle.

[0016] Since the image may show several vehicles, which may also overlap, the rectangles should first be assigned to each other, so that, for example, a rectangle for the front or rear side is assigned to the rectangle of the entire, same vehicle.

[0017] Vehicles here include, for example, two-wheelers, trucks, trams, passenger cars, mobile homes, etc.

[0018] According to one embodiment, determining whether the first and second rectangles are boundaries relating to the same vehicle comprises calculating a first value indicating the ratio of the common area of ​​the first and second rectangles to the area of ​​the second rectangle, and then determining the first and second rectangles as boundaries relating to the same vehicle if the first value exceeds a first threshold.

[0019] Thus, the quotient is calculated from the intersection of the areas of the two assigned rectangles and the area of ​​the rectangle of the front or rear of the vehicle. This quotient is used as a measure for the assignment. The higher this value, the higher the probability that the two rectangles refer to the same vehicle. If this value exceeds a predetermined threshold, the rectangles are assigned to each other, and the next step of the process can be executed.

[0020] In this next step, which is carried out when the threshold for rectangle assignment has been exceeded, a check is carried out to determine whether the assigned rectangles are positioned in such a way that it is possible to clearly determine which side of the vehicle is depicted in the image. "Side" refers to the left or right side of the vehicle, e.g., from the camera's perspective. Due to errors and inaccuracies in image analysis, this is not possible for rectangle determination, for example, if the side surfaces are too small due to the perspective; e.g., if the vehicle is viewed from an almost frontal perspective.

[0021] According to one embodiment, determining whether a side of the vehicle can be assigned from the two rectangles comprises determining a second value that indicates the ratio of the common area of ​​the first and second rectangles to the sum of the areas of the first and second rectangles, and a side orientation is considered determinable if the second value falls below a second threshold value.

[0022] Here, the quotient of the intersection of the two rectangles to the total area is calculated, which thus represents a relative measure of the size of, for example, the visible side area. In other words, the further a vehicle is depicted from the side in an image, the smaller the front rectangle becomes and thus the intersection, and the larger the rectangle of the entire vehicle becomes, assuming the vehicle is longer than it is wide. If this ratio ultimately falls below a threshold, it is assumed that one side can be reliably determined. Which side of the vehicle is visible in the image—i.e., the side orientation—can be determined based on the relative position of the two rectangles.

[0023] After determining that a lateral orientation can be determined and once this has been determined, the three-dimensional reconstruction of the vehicle can be carried out.

[0024] According to one embodiment, performing a three-dimensional reconstruction of the vehicle comprises determining the rear or front surface from the camera's perspective by projecting the front or rear surface, respectively, wherein the direction of the projection takes into account the lateral orientation; and comprises determining the side surfaces by connecting the corners of the front or rear surface to the corners of the rear or front surface.

[0025] For example, the vehicle in the image exhibits a perspective alignment, which can be accounted for by projecting the front rectangle, as seen from the camera, onto an invisible rear rectangle to be constructed. The alignment is determined in particular by the size of the objects in the image and the camera's focal length, as explained in more detail below. Once the rear rectangle, which is invisible in this example, is constructed, the side, top, and bottom surfaces can be constructed by connecting the corresponding corners of the front and rear rectangles.

[0026] According to one embodiment, determining the back or front surface comprises determining an angle between the bottom edge of the front or back rectangle of an adjacent line from a bottom corner of the front or back rectangle and the corresponding corner of the back or front rectangle.

[0027] The angle can be determined trigonometrically, for example, from knowledge or assumptions about the vehicle length or width and the rectangle dimensions. The vehicle dimensions can be obtained, for example, through vehicle type recognition from these or other camera data, or via a car-to-car connection. This allows additional parameters to be calculated, such as the longitudinal distance difference between the rear and front surfaces of the vehicle.

[0028] According to one embodiment, determining the rear or front surface comprises calculating the height of the rear or front rectangle, respectively. The height is calculated from the same ratio of the focal length of the camera to the length of a lateral edge of the first or second rectangle, respectively, and the distance of the camera from the vehicle and the estimated actual front or rear height of the vehicle, respectively. This allows all corners of the rear rectangle to be determined, the corresponding corner points to be connected, and the 3D reconstruction to be completed.

[0029] If more than two rectangles are used, appropriate criteria can be applied. For example, the rectangles of the rear area and the total area, or the rectangles of the front area and the total area, can be assigned to a vehicle in pairs, and pairs with a common total area rectangle can be searched for. Similarly, the orientation can be estimated in pairs. With more than two rectangles, plausibility checks are possible, and errors can be reduced, for example, by averaging.

[0030] According to a second aspect, a system for three-dimensional image reconstruction of a vehicle is provided, which system comprises a camera for capturing an image with at least one vehicle and a computing unit configured to carry out the method described above.

[0031] According to a third aspect, a vehicle is provided having a system as described above.

[0032] In the following, embodiments of the invention are explained in more detail with reference to the schematic drawing. Fig. 1 a diagram relating to the generation of the rectangles and the 3-D reconstruction according to an embodiment, Fig. 2 a criterion for assigning two rectangles to each other according to an embodiment, Fig. 3 a criterion for determining whether a side orientation is determinable, according to an embodiment, Fig. 4 a method according to an embodiment, Fig. 5 a system according to an embodiment, Fig. 6 a vehicle according to an embodiment, Fig. 7 a front view of the geometry of the rectangles according to an embodiment, Fig. 8 a plan view of the geometry of the rectangles according to an embodiment, Fig. 9 the relationship between the actual heights of the vehicle and the heights of the vehicle depicted in the image according to one embodiment.

[0033] In the following, the rectangles spanning the front, rear or the entire vehicle are also referred to by the technical term “bounding box”.

[0034] The reconstruction of the 3-D structure of vehicles based on a bounding box that encompasses the entire car and a second bounding box that encompasses the rear or front of a car goes beyond previous approaches, since - minimal annotation effort is required, - the number of two boxes for reconstructing the 3-D view is minimal, - so-called “IoA” and “IoU” criteria are introduced to link the bounding boxes in the image, and - an estimation of the vehicle orientation based on the 3-D reconstruction is possible.

[0035] This procedure can be extended by detecting side boxes: The detection of side boxes can be used as an alternative calculation instead of the front / back fronts, e.g., when the side view is more visible in the image than the front / back front or as an additional estimation to improve the accuracy of the calculation.

[0036] This method can be implemented, for example, using convolutional neural networks (CNNs) as detectors or with classical feature-based detectors such as HOG (Histogram of Oriented Gradients). This enables a computationally efficient implementation of 3D and orientation estimation.

[0037] The majority of object detectors only estimate the 2D structure of objects in images. This is represented either by 2D bounding boxes or by pixel-precise object annotation. No 3D orientation can be derived from this. Initial methods based on detailed 3D structures, such as car models, tires, doors, etc., require considerable annotation effort to train machine learning algorithms that estimate the 3D boxes. Furthermore, these methods are often computationally intensive. The proposed invention requires little annotation effort and is computationally efficient.

[0038] The invention can be combined with any object detector. It was prototyped using CNNs, but can also be used for other object detectors, such as HOG-based detectors. It is therefore usable for various object detectors and is not limited to, for example, a fixed CNN architecture, as is the case with other methods.

[0039] 3D bounding box estimation and orientation estimation for vehicles based on images from a camera in road traffic for individual images.

[0040] The invention is explained below using an example in which a front side of the vehicle, as seen from the camera, and a visible side are outlined with a first rectangle, and the front side, as seen from the camera, is outlined with a second rectangle. Detection and processing of other or further sides would be performed analogously.

[0041] Fig. Figure 1 provides an overview of the reconstruction of 3D boxes based on 2D detectors. The procedure is described below: In a first step 101, two detectors are trained to detect the entire box and the front or rear box (back / front box) in image 101. For this purpose, annotations for the training data are required which indicate that vehicles surround the entire area in image 102 occupied by the vehicle with a bounding box (total box 106), as well as the visible area of ​​the front and / or rear of a vehicle with a bounding box (back / front box 107 in image 103).

[0042] For a signed orientation estimation, it is necessary to distinguish between the front and rear boxes. The orientation then indicates the type of side (front, rear, left, or right) from the vehicle's perspective. This information may be missing for a 3D representation.

[0043] The method can be used with any detector, for example a ResNet50, i.e. a 50-layer residual network detector or a so-called “Region-based Fully Convolutional Networks” (RFCN) object detector, which can predict bounding boxes in image data and output attributes for them.

[0044] Since the detectors provide the x, y coordinates (i.e., height and width) of the bounding boxes 602, 603 independently of each other, a calculation is first necessary to determine which boxes belong to a car. Here, front / rear boxes 103 are assigned to the overall box 102. For this purpose, as in Fig. 2, the metric “IoA” is calculated: IoA=(Area_Back_Front_Box 202 AND Area_Total_Box 203) 201 / Area_Back_Front_Box 204.

[0045] If this metric is greater than a threshold (e.g., 0.7), two boxes are associated. This step is necessary because multiple detections for both total boxes and back / front boxes can occur in an image. For boxes without associations, no 3D reconstruction is performed. The orientation of the boxes is either side, front, or back.

[0046] After associating two boxes, it is determined whether a reconstruction of the 3-D box is useful. For this purpose, as in Fig. 3, the metric “IoU” Intersection over Union is calculated: (Area_Back_Front_Box 302 AND Area_Total_Box 303) 301 / (Area_Back_Front_Box 302 OR Area_Total_Box 303) 304.

[0047] If this value is smaller than a specified threshold, for example, 0.7, a 3D box is reconstructed. Otherwise, the 3D reconstruction is error-prone, as the box detectors may be subject to small measurement errors, which can lead to reconstruction of the wrong side of the vehicle.

[0048] Since both the back_front_box detector and the total_box detector can provide erroneous measurements for the bounding boxes, a correction for the front / back view is first calculated. The corrected front / back view is the following area: (Area_Back_Front_Box 302 AND Area_Total_Box 303) 304.

[0049] The 3D view is then reconstructed. The estimated side view is then connected to the front / rear view and extended until the front / rear view and the front / rear view occupy the entire box. Using a ground plane assumption—that is, that the vehicle is on a non-inclined plane—the offset of the lower trailing edge to the front trailing edge in the z-direction (i.e., in the direction perpendicular to the surfaces of the boxes) can be calculated, or this offset can be determined a priori.

[0050] In Fig. 1 shows the different boxes required for the reconstruction: - Front-rear box detector 107 in Figure 103, - Total box detector 106 in Figure 102, - Area_Back_Front_Box AND Area_Total_Box 201, 301, - reconstructed 3-D box 108 in image 104.

[0051] In 105 the finally obtained temporal sequence, e.g. a video sequence, of the obtained reconstructions is shown.

[0052] Fig. 4 shows the individual steps of the method for three-dimensional image reconstruction of a vehicle. First, in 401, an image of at least one vehicle is captured with a camera. Then, in 402, a first rectangular border of the entire vehicle is determined in the image, resulting in a first rectangle. Correspondingly, in 403, a second rectangular border of the front surface of a vehicle, as viewed by the camera, is determined in the image, resulting in a second rectangle. In 404, it is then determined whether the first and second rectangles are borders that relate to the same vehicle. If so, in 405 it is determined whether a side orientation of the vehicle can be assigned from the two rectangles. If so, the side orientation is determined in 406, and finally, in 407, the three-dimensional reconstruction of the vehicle is carried out from the first rectangle, the second rectangle, and the side orientation.

[0053] Fig. 5 shows a system 500 for three-dimensional image reconstruction of a vehicle, which system has a camera 501 for capturing an image with at least one vehicle, and a computing unit 502 for carrying out the method.

[0054] Fig. 6 shows a vehicle 600 having a system 500 as described above.

[0055] A rough orientation estimate can be derived from the 3D reconstruction, e.g.: 0.3 < IoU < 0.4 corresponds to an orientation of 45° when the front is visible.

[0056] If the camera calibration is given and a vehicle model (height, width, length) can be given or assumed, the orientation angle can be estimated based on the ground plane assumption and the ray theorem.

[0057] In Fig. 7-9 shows an exemplary approach of how the orientation can be estimated and how the rear box, or rear rectangle, from the camera's perspective can be calculated.

[0058] Fig. 7 shows a front view of the geometry of the front rectangle 701 and the entire rectangle 702 in the xy-plane with z=0, ie the plane lies in the plane of the front rectangle. In Fig. Variables with lowercase letters represent variables in the image domain. The unit is, for example, pixels. Variables with uppercase letters represent variables in the real world. The unit is, for example, meters.

[0059] The front rectangle 701 has a height h r(0) at z=0 and a width w r(0) . The total rectangle 702 has a width w tot on.

[0060] Fig. 8 shows a top view of the geometry of the rectangles 701, 702, which are shown as a vehicle rotated by an angle β, also depicted as a rectangle. The sizes are shown accordingly. Fig. 7, only in sizes of the real environment: W car Width of the vehicle [m] W r visible width of the front of the vehicle [m] W tot visible width and length of the vehicle [m] L car Length of the vehicle [m] H car Height of the vehicle [m] L1 Distance between front rectangle and back rectangle L2 Distance between front rectangle and back rectangle including offset due to rotation. H (0) , H (L1) , H (L2) represent the height of the vehicle at z=0, z=L1, z=L2. Wcar=2 m Lcar=5 m Hcar=1.8 m Hcar=H(0)=H(L1)=H(L2)

[0061] The relationship between meters and pixels can be expressed, for example, by the ratio of the height h r(0) to H0 can be determined.

[0062] The orientation angle β can now be determined from the relationships sin(β)=Wr / Wcar→β or Wtot−Wr=Lcar cos(β) This allows L1 and L2 to be calculated as L1=Lcar sin(β) L2=L1+Wcar cos(β)

[0063] Fig. 9 shows how, using the ray theorem and the known quantities h r(0) , L1, L2, H (0) , H (L1) , H (L2) and the focal length l foc the camera the distance D and h r(11) , h r(12) and w (11) , w (12) determine: D is determined from D / H(0)=lfoc / hr(0). hr(11) results from D+L1 / H(L1)=lfoc / hr(11). hr(12) is given by D+L2 / H(L2)=lfoc / hr(12). w(11) and w(12) finally result from the ratio r(0) / hr(11) and hr(0) / hr(12) respectively.

[0064] This estimate can be refined using a so-called "expectation / maximization" algorithm, which is familiar to those skilled in the art. This algorithm first iteratively calculates a solution with initial estimates, and then re-initializes the solution as more accurate estimates in a subsequent run. This is therefore more computationally intensive at runtime.

Claims

[1] Method for three-dimensional image reconstruction of a vehicle (600), comprising the steps - capturing (401) an image of at least one vehicle with a camera (501), - detecting (402) a first rectangular border of the entire vehicle on the image to obtain a first rectangle (203), - detecting (403) a second rectangular border of a side of a vehicle (600) on the image to obtain a second rectangle (202), - determining (404) whether the first and second rectangles are boundaries that concern the same vehicle, - determining (405) whether a lateral orientation of the vehicle (600) can be assigned from the two rectangles if the first and second rectangles are borders relating to the same vehicle, - Determination (406) of the side orientation, if a side orientation can be determined, - performing (407) a three-dimensional reconstruction of the vehicle from the first rectangle, the second rectangle and the side orientation. [2] The method of claim 1, wherein determining whether the first (203) and second rectangles (202) are boundaries relating to the same vehicle (600) comprises calculating a first value indicative of the ratio of the common area (201) of the first (203) and second (202) rectangles to the area (204) of the second rectangle, and determining the first and second rectangles as boundaries relating to the same vehicle if the first value exceeds a first threshold. [3] Method according to claim 1 or 2, wherein determining whether a side of the vehicle can be assigned from the two rectangles (202, 203) comprises determining a second value which indicates the ratio of the common area (301) of the first and second rectangles to the sum (304) of the areas of the first (302) and second rectangles (302), and a side orientation is considered determinable if the second value falls below a second threshold value. [4] Method according to one of the preceding claims, wherein performing a three-dimensional reconstruction of the vehicle (600) comprises determining the rear or front surface from the viewpoint of the camera (501) by a projection of the front or rear surface, wherein the direction of the projection takes into account the lateral orientation; and determining the side surfaces by connecting the corners of the front or rear surface to the corners of the rear or front surface. [5] A method according to any one of the preceding claims, wherein determining the back or front surface comprises determining an angle between the lower edge of the front or back rectangle of an adjacent line from a lower corner of the front or back rectangle and the corresponding corner of the back or front rectangle. [6] Method according to one of the preceding claims, wherein determining the rear or front surface comprises calculating the height of the rear or front rectangle, wherein the height is calculated from the same ratio of the focal length of the camera (501) to the length of a lateral rectangle edge of the first or second rectangle on the one hand and the distance of the camera (501) to the vehicle (600) and the estimated real front or rear height of the vehicle (600) on the other hand. [7] System (500) for three-dimensional image reconstruction of a vehicle (600), comprising - a camera (501) for capturing an image with at least one vehicle, and - a computing unit (502) configured to carry out the method according to one of claims 1 to 6. [8] Vehicle (600) comprising a system (500) according to claim 7.

Images