Method for determining a model of a traffic obstacle

Abstract

According to the method for determining a model of a traffic obstacle, a plurality of vehicles are provided, each vehicle having at least one camera and a processor for computer vision processing. A corresponding image of a scene is captured by a corresponding at least one camera of each of the plurality of vehicles. A corresponding processor of each of the vehicles evaluates the corresponding image and generates a corresponding preliminary model of the traffic obstacle. The corresponding preliminary model of the traffic obstacle is transmitted from each of the vehicles to a server. The image processor of the server evaluates the corresponding preliminary model received from each of the vehicles and determines the model of the traffic obstacle.

Description

Technical Field

[0001] The present disclosure relates to a method for determining a model of a traffic obstacle, such as a Jersey Wall, that may be placed on a road to separate various lanes of the road from one another. Background Art

[0002] Detection and modeling of road infrastructure are fundamental requirements for generating an accurate road database that can be used for autonomous or robot-assisted driving. For example, road infrastructure includes traffic signs, poles, guardrails, and traffic barriers that may include the New Jersey Wall. Traffic barriers like the New Jersey Wall can be used at road construction sites to separate the roadway. Road infrastructure can be captured and registered by dedicated vehicles using complex sensor systems, such as stereo cameras that capture images of the road and road infrastructure while driving. Modeling of road infrastructure refers to the recovery and representation of 3D spatial information of road infrastructure. In order to detect and model traffic obstacles, most known methods use laser radar (LIDAR) or stereo cameras that can directly acquire 3D information.

[0003] US20200184233A1 discloses a method for detecting road obstacles like New Jersey walls. Here, vertical and horizontal functions are used, which, when combined, map to multiple image features. These functions are generated by a complex multi-camera system.

[0004] In theory, a simple optical sensor system, such as an optical system including a monocular camera, cannot reconstruct 3D information of traffic obstacles for several reasons. First, a monocular camera cannot directly obtain 3D information of a scene. The only known method for reconstructing 3D information using a monocular camera requires matching relationships between pixels between frames. Second, most traffic obstacles have almost no texture. Therefore, it is almost impossible to obtain matching relationships between pixels on traffic obstacles between frames. Third, the disparity is too small to restore its spatial information. Only when the disparity is sufficient can a monocular camera restore spatial lines through multi-view geometry theory. However, there is almost no difference on the edges of traffic obstacles in consecutive frames. Summary of the invention

[0005] The problem to be solved by the present invention is to provide a method for determining a model of a traffic obstacle by using a cost-effective and simple sensor system, such as a monocular camera. In addition, the method should provide an improved recognition accuracy.

[0006] The solution to this problem is described in the independent claim. The dependent claims relate to further developments of the invention.

[0007] In one embodiment, a method for determining a model of a traffic obstacle by a plurality of vehicles is provided, each vehicle having at least one camera and a processor for computer vision processing. A corresponding image of a scene is captured by a corresponding at least one camera of each of the plurality of vehicles. The corresponding image is evaluated by a corresponding processor of each of the vehicles, and a corresponding preliminary model of the traffic obstacle is generated. The corresponding preliminary model of the traffic obstacle is transmitted from each of the vehicles to a server. The image processor of the server evaluates the corresponding preliminary model received from each of the vehicles, and determines the model of the traffic obstacle. In a very simple embodiment, a single camera may be sufficient. Alternatively, a plurality of cameras may be provided.

[0008] In order to model an object, a series of consecutive images are usually taken with the aid of an optical sensor system, such as a stereo camera system. Then, pixels belonging to the same position on the object are compared in the consecutive images to create a model of the object. The stereo camera system must provide information about the object from different spatial positions. However, a problem with modeling the surfaces of traffic obstacles, such as traffic barriers, for example the New Jersey wall, is that the surfaces of traffic obstacles usually have little or almost no texture. This makes it difficult to identify the same pixels arranged at the same position on the surface of the traffic obstacle to be modeled in consecutive images.

[0009] The method for determining the model of a traffic obstacle allows the use of simple cameras located at different positions to capture images of the traffic obstacle. For example, each camera can be designed as a simple monocular camera, wherein a corresponding monocular camera is located in each of a plurality of vehicles. The plurality of vehicles are located at different positions in the scene so that each camera takes a picture of the traffic obstacle from a separate position. A corresponding processor in each of the vehicles can evaluate the captured image information of the traffic obstacle so that a separate preliminary / virtual model of the traffic obstacle is generated in each vehicle.

[0010] Individual model information for the traffic obstacle, i.e., an individual preliminary / virtual model of the traffic obstacle, is sent to the server by each of the vehicles. The preliminary / virtual models of the traffic obstacle received from the different vehicles are evaluated by the image processor of the server. In summary, according to one embodiment, i.e., a method for determining a model of a traffic obstacle, stereoscopic vision thus occurs on the server, which can generate an accurate model of the traffic obstacle by evaluating and comparing the individual preliminary / virtual models generated and received from each of the various vehicles in the scene.

[0011] In general, these embodiments are applicable to various road borders, curbs, fences, etc. The traffic barrier may include at least one of a Jersey wall, a Jersey barrier, a K-rail, or a median barrier. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Hereinafter, the general inventive concept of examples of embodiments will be described by way of example with reference to the accompanying drawings, but not limiting the embodiments.

[0013] Figure 1 A system for executing a method for determining a model of a traffic obstacle is illustrated;

[0014] Figure 2 A flow chart illustrating method steps of a method for determining a model of a traffic obstacle is shown; and

[0015] Figure 3 A simplified example of a method for determining a model of a traffic obstacle on a server by evaluating respective preliminary / virtual models of the traffic obstacle generated by respective vehicles is illustrated.

[0016] In the following, with the help of Figure 1 and Figure 2 The different steps of an embodiment of a method for determining a model of a traffic obstacle are explained in general terms.

[0017] Figure 1 A system is shown comprising a plurality of vehicles 100, 101 and 102, wherein each of the vehicles comprises at least one camera 10 for capturing images / frames of the respective vehicle's surrounding scene, a processor 20 for computer vision processing, and a storage device 30 for storing a road database. The respective vehicles 100, 101 and 102 may communicate with a server 200 comprising an image processor 201. In a very simple embodiment, a single camera may be sufficient.

[0018] The embodiment of modeling traffic obstacles in a road database system includes method steps V1, V2 and V3 performed by each of a plurality of vehicles in the order of V1, V2, V3, S and method step S performed by a server. These method steps are exemplified in Figure 2 in the flow chart.

[0019] According to one embodiment, in a method for determining a model of a traffic obstacle, a plurality of vehicles 100, 101, and 102 are provided. Each of the vehicles includes a corresponding camera 10 and a corresponding processor 20 for computer vision processing. In step V1 performed by each of the plurality of vehicles, a corresponding image of the environment scene of the vehicle is captured by the corresponding camera 10 of each of the plurality of vehicles 100, 101, and 102. The camera 10 may be embodied as a monocular camera installed in each of the vehicles. The camera may be configured to capture a video of the environment scene in which the corresponding vehicle is located in real time.

[0020] In step V2 performed by each of the plurality of vehicles, the respective images captured in step V1 are evaluated by the respective processor 20 of each of the vehicles 100, 101 and 102, and a respective preliminary / virtual model of the traffic obstacle is generated. For this purpose, a road model may be provided in the respective storage device 30 of each of the vehicles 100, 101 and 102. According to one embodiment, a planar road surface may be assumed for the road model ([0, 0, 1]). Taking into account the road model, the image pixels of the captured image may be projected to 3D points on the road model.

[0021] According to one embodiment of the method, the corresponding captured images are evaluated by computer vision processing by the corresponding processor 20 of each of the vehicles 100, 101 and 102 to extract pixels in the corresponding captured images that represent the edge of the traffic obstacle. For example, the pixels extracted in the corresponding captured images can represent the upper edge of the traffic obstacle. With regard to conventional methods for surface modeling of traffic obstacles, computer vision processing can be used to detect the entire area of ​​the traffic obstacle. However, the boundary between the wall and the ground may not be clear in general. According to one embodiment, i.e., a method for modeling traffic obstacles, by extracting pixels representing the edge of the traffic obstacle from each of the captured images, only the position of the edge of the traffic obstacle, especially the upper wall edge, is considered.

[0022] The corresponding processor 20 of each of the vehicles 100, 101 and 102 generates a corresponding preliminary / virtual model of the traffic obstacle by projecting the corresponding extracted pixels representing the edge of the traffic obstacle in the road model to generate the corresponding 3D points of the edge of the traffic obstacle. According to one embodiment of the method, the corresponding processor 20 of each of the vehicles 100, 101 and 102 generates a corresponding spline curve of the edge of the traffic obstacle. The corresponding spline curve represents the corresponding preliminary / virtual model of the traffic obstacle generated in each of the vehicles. These preliminary / virtual models still do not contain the real traffic obstacle position.

[0023] In step V3 of the method for determining the model of a traffic obstacle, the corresponding preliminary / virtual model of the traffic obstacle is transmitted from each of the vehicles 100, 101 and 102 to the server 200. In step V1, the corresponding images are captured with the individual postures of the corresponding cameras 10 of each of the vehicles 100, 101 and 102. In summary, the different preliminary / virtual models generated from the corresponding processors 20 of different vehicles are generated from different viewpoints. Since the preliminary / virtual models of the traffic obstacle are generated from different viewpoints, the camera position / pose is also important for determining the model of the traffic obstacle by the server. According to one embodiment of the method, the posture of the corresponding camera 10 of each of the vehicles 100, 101 and 102 is transmitted from each of the vehicles to the server 200 together with the corresponding preliminary / virtual model of the traffic obstacle.

[0024] In step S performed by the server, the image processor 201 of the server evaluates the corresponding preliminary / virtual models received from each of the vehicles 100, 101 and 102, and determines the model of the traffic obstacle. After having collected information from each of the vehicles, namely the reported preliminary / virtual models of the respective traffic obstacles and the respective camera poses, the image processor 201 of the server 200 will restore the real spatial information of the traffic obstacle. The model of the traffic obstacle determined by the image processor 201 of the server may contain at least information about the position and height of the traffic obstacle.

[0025] According to an embodiment of a method for determining a model of a traffic obstacle, an image processor 201 of a server 200 evaluates at least two of the respective preliminary / virtual models of traffic obstacles received from at least two of a plurality of vehicles 101 and 102 to determine the model of the traffic obstacle by the server. Figure 3 It illustrates how to recover the model of the traffic obstacle, i.e. the real position of the traffic obstacle, assuming that the server 200 has received two reports of corresponding individual preliminary / virtual models and corresponding individual camera poses of vehicles 101 and 102 passing in different lanes.

[0026] Reference numeral 104 represents the position of the preliminary / virtual model of the traffic obstacle generated from the camera of the vehicle 101. Line 105 passes through the camera position of the vehicle 101 and the preliminary / virtual position of the traffic obstacle 104. The true position of the traffic obstacle edge of the traffic obstacle is on line 105. Reference numeral 103 corresponds to the preliminary / virtual model generated by the camera of the vehicle 102. The true position of the traffic obstacle is located on line 106 connecting the position of the vehicle 102 and the preliminary / virtual model 103. The image processing 200 determines the intersection 107 as the true position of the traffic obstacle. All intersections are fitted by spline curves to model the traffic obstacle.

[0027] It has been shown that it is advantageous if the vehicles for which separate preliminary / virtual models of traffic obstacles are generated pass in different lanes. In another case, if two reports of generated separate preliminary / virtual models are received from vehicles passing in the same lane, they are generated from almost the same field of view. Therefore, the intersection between the two lines has a high variance, making the recovered model of the traffic obstacle unreliable in most cases.

[0028] The method for determining the model of traffic obstacles has several advantages. First, the information reported is of high efficiency. The method only requires the generation of a separate preliminary / virtual wall model and the corresponding camera pose for each of the vehicles. In addition, the method saves communication capacity. Secondly, it is easy to handle the problem of false detections where traffic obstacles are not detected in some frames. If some frames are not detected, it is easy to interpolate or fit the missing parts when generating the preliminary / virtual model.

[0029] The approach to modeling traffic obstacles can be used to generate road databases for self-driving cars, but can also be used in a number of other areas of machine vision and machine orientation, such as outdoor or underwater robot orientation.

Claims

1. A method for modeling a traffic obstacle, comprising: capturing an image of a scene including the traffic obstacle by a monocular camera of the vehicle, evaluating, by a processor of the vehicle, the captured image through computer vision processing to extract pixels in the captured image that represent an upper edge of the traffic obstacle; generating, by the processor of the vehicle, a preliminary model of the traffic obstacle, comprising projecting the extracted pixels representing the upper edge of the traffic obstacle in a road model to generate a 3D point of the upper edge of the traffic obstacle, the road model being stored in a storage device of the vehicle, and The generated preliminary model of the traffic obstacle is transmitted by the vehicle to a server in order to evaluate the transmitted preliminary model and at least one further preliminary model transmitted from a respective further vehicle to the server in order to determine an improved model of the traffic obstacle.

2. The method according to claim 1, in, The processor of the vehicle generates a spline curve of the upper edge of the traffic obstacle, the spline curve representing the preliminary model of the traffic obstacle.

3. The method according to claim 1 or 2, in, capturing the image at a pose of the monocular camera of the vehicle, and Therein, the vehicle transmits the posture of the monocular camera of the vehicle together with the generated preliminary model of the traffic obstacle to the server.

4. The method according to claim 1, in, The improved model of the traffic obstacle comprises at least information about the position and height of the traffic obstacle.

5. The method according to claim 1, in, The traffic barrier includes at least one of a New Jersey wall, a New Jersey guardrail, a K-type guardrail or a roadside guardrail.

6. A vehicle comprising: Monocular camera; as well as Processors for computer vision processing Wherein, the vehicle is adapted to perform the method as claimed in any one of claims 1-5.

Images