Method, traffic monitoring system and computer program product

Abstract

The invention relates to a method (100) for operating a traffic monitoring system (10) for recording a traffic scene (11) on a roadway (12), wherein the roadway (12) can be travelled by at least one vehicle (13), the method (100) comprising the following steps: - reading in (110) the traffic scene (11) on the roadway (12) in the form of image data (24) by means of at least one camera unit (25), in particular a positionally fixed camera unit (25), at least one optical detection unit (16); - recognising (120) at least one object (27) on the vehicle (13) from the image data (24) by means of a processing unit (19), wherein the recognised object (27) serves to identify the vehicle type and / or the vehicle owner, and / or recognising (120) a driver (28) in the vehicle (13) from the image data (24) by means of the processing unit (19) during the reading-in operation (110); - recognising (120) predefined control points (17) from environmental data (18) in the captured image data (24) by means of a processing unit (19); - converting the control points (130) into plane-related data (20) of a plane (21) by means of the processing unit (19); - converting the captured image data (24), the plane-related data (20) and the object data (29) and / or driver data (30) using a transformation algorithm into bird's-eye view data (22) of a bird's eye view for recording the traffic scene (11) on the roadway (12) by means of the processing unit (19); and - evaluating (150) the bird's-eye view data (22) of the recording of the traffic scene (11) on a roadway (12) by means of an evaluation unit (23) to form evaluation data (35) for operating a traffic monitoring system (10).

Description

[0001] Procedure, traffic monitoring system and computer program product

[0002] Description

[0003] The invention relates to a method having the features of independent patent claim 1, a traffic monitoring system having the features of independent patent claim 10 and a computer program product having the features of independent patent claim 14.

[0004] Excessive vehicle speed is a major problem in the enforcement of traffic safety laws. However, another cause of accidents on motorways and expressways is due to vehicles driving too close to the vehicle in front. The road traffic regulations in European countries (Netherlands, Germany, and Austria) and the USA have different laws and derived measurement methods for punishing distance control violations depending on the speed and / or high speeds. The usual measurement methods differ in the sensors used, the angle of the sensors, and the installation and / or application method. As a rule, a single-point measurement is not considered conclusive; therefore, any violation should be measured over a longer distance and documented in a way that can be admissible in court.

[0005] Furthermore, even in countries with driver liability, court and / or expert assessments, automated measurement, documentation, and ticket / penalty filing for following distance violations in traffic require proof of the violation with a driver's photo. There is a legal requirement to prove the following distance violation over a distance of at least 250 meters. Point-based measurement using radar / cameras is not suitable for this purpose, or only with limitations.

[0006] However, it has been found that existing systems and procedures for punishing traffic violations mean that the data or images generated can only depict the speed and, above all, the distance between two vehicles inaccurately due to the perspective, so that the tolerance range and thus the inaccuracy is very high. There are camera arrangements which, from above at an angle of approximately 90° (degrees) above the road, already offer a bird's eye view. This means that there is no need to convert image or sensor data. The disadvantage, however, is that neither a license plate nor a driver can be recognized without having to operate an additional sensor at a suitable angle. Speed ​​measuring sensors, such as radar sensors, cannot be positioned at an angle of 90° to the road either.This would require the costly planning and positioning of two different sensors for different viewing angles.

[0007] It is therefore an object of the present invention to at least partially overcome at least one of the above-described disadvantages of measurement methods for punishing traffic violations. In particular, the object of the invention is to provide a method and a traffic monitoring system for punishing traffic violations that generate perspective images that allow for improved assessment of distances between vehicles and more accurate speed measurement.

[0008] For an economical and efficient arrangement of traffic monitoring sensors, it is advantageous to use as few sensors as necessary. An angular arrangement between 0° and 70° (degrees), as is essential to the invention, and an algorithmic transformation to a bird's-eye view with a theoretical angle of 90° to the road plane are preferred.

[0009] The above object is achieved by a method having the features of independent patent claim 1, by a traffic monitoring system having the features of independent patent claim 10, and by a computer program product having the features of independent patent claim 14. Further features and details of the invention emerge from the subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the traffic monitoring system according to the invention and / or in connection with the computer program product according to the invention, and vice versa, so that reciprocal reference is or can always be made to the individual aspects of the invention with regard to the disclosure.

[0010] According to a first aspect of the invention, a method is provided for operating a traffic monitoring system for recording a traffic scene on a roadway, the roadway being passable by at least one vehicle, the method comprising the following steps:

[0011] - Reading in the traffic scene on the roadway as image data by means of at least one camera unit, in particular a fixed-position camera unit, at least one optical detection unit, wherein the camera unit is aligned at an angle of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°, to the roadway,

[0012] - Detecting at least one object on the vehicle from the image data by means of a processing unit, wherein the detected object serves to identify the vehicle type and / or the vehicle owner, and / or detecting a driver in the vehicle from the image data by means of the processing unit during the reading, wherein the object and / or the driver are read in with the traffic scene on a roadway as object data and / or driver data,

[0013] - Detecting predefined control points from environmental data in the captured image data by means of a processing unit, wherein the control points are positioned at specified locations along and / or on the roadway in the environmental data,

[0014] - Converting the control points to layer data of a layer by means of the processing unit, wherein the layer corresponds to the roadway, - Converting the captured image data, the layer data, and the object data and / or driver data using a transformation algorithm to birds-eye view data of a bird's-eye view for recording the traffic scene on the roadway by means of the processing unit, wherein the object data and / or the driver data are assigned to the vehicle in the birds-eye view data.

[0015] - Evaluation of the Birds-Eye-View data from the recording of the traffic scene on a roadway to generate evaluation data using an evaluation unit for operating the traffic monitoring system.

[0016] A traffic monitoring system is a system for monitoring objects of all kinds and, in particular, traffic in general or a traffic scene.

[0017] In this case, a traffic scene is understood to be a monitored area of ​​road traffic. The traffic scene can be an intersection area that is monitored, for example, to detect a red light violation at a traffic light in the said intersection area, but also to generally monitor the traffic in this intersection area in order to possibly adapt the traffic light cycle to the traffic. However, a traffic scene can also be a multi-lane road, such as a motorway or a country road or similar, where the main focus is on congestion monitoring or speeding violations or distance violations. The traffic scene can also be understood independently of the road traffic regulations in that objects are generally monitored (e.g. pedestrians, passers-by, event participants - even indoors) and these objects in the (traffic) scene are tracked and the resulting data is processed according to the invention.This is particularly true in the context of civil security.

[0018] The roadway is the section of the road on which vehicles travel. This includes motor vehicles, such as passenger cars, trucks, land vehicles, scooters, motorcycles, and bicycles. The image data captured by the camera unit can be 2D and 3D data. This enables improved accuracy and representation of the data for the analysis of the respective traffic scene.

[0019] The checkpoints are positioned at specified locations along and / or on the roadway. Along the roadway is defined as being along the side of the roadway, i.e., parallel to a roadway boundary. A roadway boundary separates the roadway from an area where no vehicles are moving. A roadway boundary can be a curb edge to a footpath, a boundary to a motorway median, a guardrail, or a grass verge. However, the checkpoints can also be positioned on the lane markings, which serve to divide the roadway into multiple lanes.

[0020] In descriptive geometry, a bird's-eye view (also known as a bird's-eye view) refers to perspectives that show an object from above or obliquely. The observer thus sees the object as if it were a bird flying high above them.

[0021] In contrast, standard perspective refers to the view from the normal eye level of a standing observer, and worm's-eye perspective refers to the view from a point below normal eye level. However, all three types do not actually refer to the position and orientation of objects relative to the observer, but rather to the respective position of the horizon line (the observer's eye level) relative to the center of the image (see Wikipedia.org).

[0022] When analyzing the Bird's Eye View data, the vehicles in the traffic scene and their relative characteristics are evaluated. This can include data about the respective vehicle, such as vehicle type, vehicle speed, or the vehicle owner. It is conceivable that the data could be individual images or sequences. Furthermore, the analysis of the Bird's Eye View data could be visualized as an image or video.

[0023] It is conceivable that the evaluation data from individual traffic scenes could be combined into a single overall evaluation. For example, several traffic monitoring systems with at least one optical detection unit could be installed in a complex intersection area. Each individual traffic monitoring system is assigned to a lane and a specific direction of travel. The traffic monitoring systems can be linked to each other so that they execute the process synchronously. All recorded sensor data is time-stamped. The respective evaluations of the individual lanes can be combined into an evaluation of the entire intersection area.

[0024] The method enables a traffic scene, including the vehicles in it, to be displayed from a bird's-eye view. This bird's-eye view allows for better representation and measurement of the distances between vehicles or the distance of a vehicle to a road boundary, such as a no-stopping or no-parking zone. This allows for a more precise assessment of the situation in the traffic scene and leads to more accurate, more plausible, and therefore legally admissible recordings of a traffic scene.

[0025] Regardless of the stretched objects, a bird's-eye view representation can be temporally synchronized and displayed simultaneously with image and / or lidar data without transformation into birds-eye view data. This allows any potential transformation errors to be checked through the simultaneous and synchronous (time-stamped) display in the original data format. This can be achieved, for example, by splitting a monitor to display a traffic scene, with the birds-eye view representation being displayed synchronously as a video on the left half of the monitor and the display without the birds-eye view on the right half of the monitor.Within the scope of the invention, it may be advantageous for the traffic scene on the roadway to be read in as lidar data (laser data) by means of at least one lidar sensor of the at least one optical detection unit. After conversion into the plane data and before conversion into the bird's-eye view data, the image data and the lidar data are superimposed (160) to form overlay data. The lidar unit is advantageously also aligned to the roadway at an angle of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°.

[0026] A supplementary representation of the traffic scene with lidar data increases accuracy and is particularly helpful for the court-proof presentation of a traffic violation, as not only the visual representation but also the lidar data can be provided. The lidar data can also be provided as 2D or 3D data.

[0027] It is conceivable for the image data and the lidar data to represent different perspectives; for example, the image data can be overlaid in a bird's eye view and the lidar data in the recording perspective. This allows, for example, the driver to be displayed and the distance measured in a single analysis, thus enabling improved identification and classification of the vehicle and thus a plausible presentation option as a video display, for example in court. The image data can also be generated, in particular, by a stereo camera or a 360-degree camera. Furthermore, image data can be supplied from different spectral ranges, both in the visible and invisible range. A particularly preferred variant is an infrared (IR) camera, which produces special thermal images.

[0028] Within the scope of the invention, a method can be provided for converting the overlay data into birds-eye view data using the transformation algorithm. Overlaying the image data and the lidar data and converting them together into birds-eye view data enables a more accurate representation of the traffic scene during evaluation.

[0029] The primary purpose of extrinsic calibration is to ensure that reliable and accurate information can be derived from the image data or lidar data. Extrinsic calibration ensures that the internal and external orientations match, meaning that the evaluated image data or lidar data of the traffic scene match the actual surroundings. During calibration, the traffic scene, and especially the plane data of the plane or roadway, are first aligned with the surrounding data.

[0030] For extrinsic calibration, the traffic scene is imported as image data or lidar data, the control points are identified and converted into plane data, i.e., the roadway. The plane data and the image data or lidar data are then converted into birds-eye view data and evaluated. During evaluation, the birds-eye view data is compared with the real environmental data, and care is taken to ensure that the markings, such as lane markings or road edge markings, in the birds-eye view data match the respective markings in the real environmental data. If this is not the case, the alignment of the optical unit is adjusted manually or automatically, and the above steps are repeated until the markings in the birds-eye view data align with the markings in the real environmental data.

[0031] This is advantageous, for example, for gently curved traffic scenes. The position of the optical detection unit can be easily adjusted, as the course of the plane or roadway from the process can be adapted to the course of the roadway in the real environment. Extrinsic calibration can prevent incorrect evaluation of a traffic scene.

[0032] In a method, it can further be provided that the at least one optical detection unit of the traffic monitoring system and / or the method is calibrated and / or aligned by means of a 2D or 3D code, in particular a QR code, at a specific position on a stop line on the roadway in the traffic scene on the roadway.

[0033] The code can be positioned so that it can be easily detected. It can be used as a reference point for calibration and, above all, for recalibration of the optical detection unit. This is particularly advantageous if the position of the optical detection unit has changed due to environmental influences. For example, the original position of the code can be stored for recalibration so that any deviation between the target position of the optical detection unit and the stored actual position can be determined. The code can be read in each time the traffic scene on the road is read in, but cyclical reading and evaluation of the code is conceivable in order to detect any change in position in the evaluated Bird's Eye View data.

[0034] Additionally, within the scope of the invention, it can be provided that, when reading the traffic scene on a roadway, an object on the vehicle is detected in the image and / or lidar data by the processing unit, whereby the detected object serves to identify the vehicle type and / or the vehicle owner. Additionally or alternatively, a driver is detected in the image data and / or lidar data of the vehicle by the processing unit.In this case, the object and / or the driver with the traffic scene on a roadway are read in as object data and / or the driver with the traffic scene on a roadway as driver data, wherein the object data and / or the driver data are converted with the lidar data and the layer data using a transformation algorithm to the birds-eye view data of a bird's-eye view for recording the traffic scene on the roadway by means of the processing unit, and wherein the object data and / or the driver data are assigned to the vehicle in the birds-eye view data.

[0035] In this context, an object on a vehicle is understood to mean license plates for owner identification or specific features for identifying the vehicle type, such as a specific sticker for identifying a truck. This can be useful, for example, if there are different speed limits for a vehicle type depending on the day or time of day, so that a violation of a vehicle-specific limit can be traced and punished.

[0036] By converting the object data and / or driver data into birds-eye view data, the detected object and / or driver can be assigned to the respective vehicle and displayed during the traffic scene analysis. This is particularly advantageous when recording the traffic scene sequentially, since, as mentioned, an assignment of identification data—i.e., the object data and / or driver data—takes place.

[0037] It can be provided in a method that an extrinsic calibration is carried out for aligning the optical detection unit to the roadway, wherein the extrinsic calibration is carried out once or cyclically and / or wherein the extrinsic calibration is carried out automatically, at least partially automatically, or manually, in particular wherein the data from the extrinsic calibration is used for positionally accurate conversion into the plane data of the plane.

[0038] Furthermore, a method can be provided that, during the superimposition, at least one position and / or at least one coordinate of at least one pixel of the image data is compared with a data point of the lidar data corresponding to the pixel, wherein an alarm and / or a rejection of the measured value is triggered if a deviation between the data point and the pixel is detected, wherein in particular an extrinsic calibration is triggered.

[0039] This allows for easy verification of whether the image data and the lidar data belong together. This prevents lidar data and image data from different points in time from overlapping. It is also possible to compare the lidar data and image data of an actual state with the target state from the first extrinsic calibration to determine whether the optical detection unit has changed its position due to environmental influences. If a change is detected, a calibration is triggered.

[0040] Within the scope of the invention, it is possible in a method that the traffic scene on the roadway is read in by means of the optical detection unit from a height of 1.5 m to 10 m, preferably 2 m to 2.8 m, more preferably 2.3 m to 2.5 m.

[0041] Reading from this height simplifies the recording of a traffic scene at a camera unit angle of 0° to 70° and ensures the avoidance of a “blind spot”.

[0042] It can also be provided in a method that, depending on the traffic scene to be monitored on the roadway, at least one measuring line is stored in reference data that is stored in the evaluation unit, wherein the at least one measuring line is or will be aligned orthogonally or substantially orthogonally to a roadway boundary and / or orthogonally or substantially orthogonally to a lane direction of the roadway and / or parallel or substantially parallel to a front and / or rear bumper of a vehicle detected in the image data and / or lidar data, and wherein the measuring line is used in the evaluation of the evaluation data of the traffic scene.

[0043] The reference data is adapted to the respective traffic scene and reflects the respective events to be observed or the violations to be penalized. The measuring line(s) are adapted to the reference data. For example, on a highway, the reference data is adapted to distance violations, traffic observations, and speed violations, so that the measuring line runs across the width of the roadway and is designed to measure a vehicle's speed or acceleration.

[0044] For this purpose, several measurement lines can be provided parallel to each other along the entire traffic scene to capture and evaluate the speed and / or acceleration and / or distance to another vehicle over a specific distance. A parking violation, on the other hand, can be found in the reference data for monitoring an intersection area.

[0045] However, several measuring lines can be present at an intersection; here, too, several measuring lines can be arranged parallel to each other for speed or acceleration measurement. Alternatively or additionally, the shorter measuring lines can demarcate an area of ​​a no-stopping or no-parking zone on the roadway, for example, to detect a parking violation. The measuring lines can also be stop lines, traffic light stop lines, and yellow box lines.

[0046] Alternatively or additionally, it is also conceivable that the measuring line is orthogonal or substantially orthogonal to a lane marking, in particular a lane marking to delimit the part intended for oncoming traffic.

[0047] In this context, road markings are understood to mean markings that delimit the roadway or make the edge of the roadway visible and separate the roadway from shoulders or special paths. A road marking within the meaning of the invention can also be a grass verge or a guardrail.

[0048] Within the scope of the invention, it is further possible for a method to be carried out at different times, in particular cyclically, and / or upon a triggering event, in particular a crossing of the at least one measuring line by a vehicle identified in the image data.

[0049] Cyclically reading the image data, or the image data and the lidar data, allows for more precise monitoring of the traffic scene. However, reading this data during a trigger event can be advantageous, as it requires less storage capacity. A trigger event is understood here as exceeding a specified value. For example, a defined speed can be stored as a specified value, so that the process is triggered when a vehicle crosses the measuring line at a speed higher than the speed specified in the specified value. It is also conceivable that an object is stored to identify a vehicle type in order to detect vehicle-type-specific speed limits and their exceedance.

[0050] Furthermore, a method can provide for several optical detection units spaced apart from one another to detect the traffic scene on the roadway from different angles.

[0051] Multiple optical detection units enable capture from different angles and perspectives, thus avoiding a "blind spot" when observing a traffic scene. This is suitable, for example, for intersections with poor visibility or poor visibility. This results in a more accurate capture of a traffic scene.

[0052] It is conceivable that the image data or the overlay data are recorded and overlaid at different times and / or from different angles.

[0053] This increases the accuracy of the traffic scene analysis and is particularly advantageous for a legally sound representation of the traffic scene on the road. Furthermore, within the scope of the invention, a method can be provided for evaluating the bird's-eye view data at one or more points in time, including a vehicle speed and / or vehicle acceleration of at least one vehicle on the road and / or a stopping violation, in particular a stopping violation at an intersection or a traffic light area of ​​the traffic scene and / or a distance violation of a vehicle relative to a preceding vehicle.

[0054] Using the evaluation data, the traffic scene, and especially the violations, can be easily evaluated and visualized to obtain legally admissible data. The evaluation data can be converted into a single image or, if the traffic scene is read cyclically or continuously, into an image sequence or video.

[0055] Furthermore, within the scope of the invention, it can be provided in a method that the evaluation data are transmitted by means of a data interface or a transmitter of the evaluation unit to an external data storage or external further processing unit.

[0056] The external data storage or the external processing unit can, for example, already be located within a public authority, so that the allocation and final evaluation for penalizing violations can be carried out quickly and easily. This shortens the time and increases the efficiency of traffic monitoring. It would also be conceivable for the external processing unit to be directly available to an enforcement agency for penalizing the respective offense, so that the punishment can be carried out directly. For example, the external processing unit could be installed in a police car that is currently monitoring the traffic scene, so that a warning of a speeding violation or a following distance violation is issued along with the relevant identification data, allowing the police car to directly follow the violator and penalize the violation.

[0057] Within the scope of the invention, a method can further be provided for radar data to be acquired by means of at least one radar sensor of the at least one optical acquisition unit and / or ultrasound data to be acquired by means of at least one ultrasound sensor of the at least one optical acquisition unit, wherein the radar data and / or the ultrasound data are used to convert to the plane data and / or to convert to the birds-eye view data and / or to evaluate the birds-eye view data to the evaluation data and / or to superimpose to the overlay data.

[0058] By adding at least one additional sensor to the optical detection unit, the accuracy of the data read in for traffic scene analysis is increased. This has a positive impact on the legally sound analysis of the respective traffic scene.

[0059] The above object is further achieved according to a second aspect of the invention by a traffic monitoring system for carrying out the method described above, with at least one optical detection unit with at least one camera unit for reading the traffic scene on a roadway as image data, wherein the camera unit is aligned at an angle (α) of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°, to the roadway, and with a processing unit for detecting an object on the vehicle and / or a driver in the vehicle and for detecting predefined control points from environmental data in the image data, for converting the control points to plane data of the plane, wherein the plane corresponds to the roadway, and for converting the detected image data and the plane data using a transformation algorithm to bird's-eye view data of the bird's perspective,and with an evaluation unit for evaluating the traffic scene on a roadway to generate the evaluation data. The predefined control points can be specified manually by the user (e.g., a surveyor) or by an automatic system (computer program product) that detects suitable boundary points, road edges, and / or lane lines and aligns them with the control points, thus (pre)defining them. The control points primarily serve to define a plane, whereby the user should generally set as few control points as necessary in the normal perspective (not a bird's-eye view). Subsequent input by adding or moving control points in the bird's-eye view represents a simpler and thus more error-resistant method for the user (or the computer program product).

[0060] In general, traffic monitoring systems can be used for a variety of tasks in road traffic. For example, they have proven effective in analyzing traffic flows, controlling traffic management systems, and, in particular, in recording and tracking traffic violations.

[0061] Traffic monitoring systems are equipped with an optical detection unit for the prosecution of traffic violations, as this makes subsequent prosecution of the violation, possibly even in court, much easier. Traffic violations include, for example, exceeding the speed limit, ignoring red lights at traffic lights, failing to maintain a minimum following distance, etc.

[0062] Within the scope of the invention, it can be provided in a traffic monitoring system that the at least one optical detection unit has at least one lidar sensor for detecting lidar data.

[0063] The images captured with the lidar sensor increase accuracy and are particularly useful for the court-proof depiction of a traffic violation. Using the lidar sensor, images can be captured regardless of weather conditions. Within the scope of the invention, a traffic monitoring system can further be provided with several optical detection units for reading the traffic scene on the roadway, with the optical detection units arranged at a distance from one another.

[0064] Multiple optical detection units enable detection from different angles and perspectives, thus avoiding a "blind spot" when observing a traffic scene. This is suitable, for example, for intersections with poor visibility or confusing areas.

[0065] Furthermore, within the scope of the invention, in a traffic monitoring system, it can be provided that the evaluation unit has the data interface or the transmitter for transmitting the evaluation data to an external data storage or external further processing unit.

[0066] The external data storage or external processing unit can, for example, already be located within a public authority, allowing the allocation and final analysis for the purpose of penalizing violations to be carried out quickly and easily. This shortens the time and increases the efficiency of traffic monitoring.

[0067] Within the scope of the invention, it is possible in a traffic monitoring system for the at least one optical detection unit to be arranged next to the roadway and / or above the roadway.

[0068] In this case, the arrangement of the optical detection unit next to the roadway means that the optical detection unit is positioned next to the roadway, i.e., behind a road barrier, and records the traffic scene from a lateral perspective. The optical detection unit can be mounted on a mast. If the optical detection unit is positioned above the roadway, it is positioned between a first road barrier and a second road barrier. The optical detection unit can be mounted, for example, on a gate or a tunnel wall or arch.

[0069] Both the next-to-lane and above-lane configurations can be implemented using a one-pole solution. A one-pole solution involves positioning an optical detection unit in such a way that the traffic scene is captured from a fixed point at a single, fixed angle between the optical detection unit and the vehicle, with a view of the driver and license plate. Only a single power connection is required.

[0070] Furthermore, in a traffic monitoring system, it can be provided that the optical detection unit is arranged at a height of 1.5 m to 10 m, preferably 2 m to 2.8 m, more preferably 2.3 m to 2.7 m.

[0071] In a traffic monitoring system, it can also be provided that the at least one optical detection unit has at least one radar unit for detecting radar data and / or at least one ultrasonic sensor for detecting ultrasonic data. Accordingly, lidar data can be combined or fused with image data or any sensor data (e.g., radar data, ultrasonic sensor data).

[0072] By adding at least one additional sensor to the optical detection unit, the accuracy of the data read in for traffic scene analysis is increased. This has a positive impact on the legally sound analysis of the respective traffic scene.

[0073] The above object is further achieved according to a third aspect of the invention by a computer program product according to the invention, comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method described above by means of a traffic monitoring system described above.

[0074] Advantages that are described in detail with regard to the method for operating a traffic monitoring system for recording a traffic scene on a roadway according to the first aspect of the invention apply equally to a traffic monitoring system according to the second aspect of the invention and to a computer program product according to the third aspect of the invention.

[0075] Further advantages, features, and details of the invention will become apparent from the following description, which describes several embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination.

[0076] The invention is explained in more detail below with reference to the accompanying drawings. In the drawings:

[0077] Fig. 1 is a schematic representation of a method according to the invention,

[0078] Fig. 2 is a schematic representation of a visualized evaluation of the method according to the invention with reference to a roadway before an extrinsic calibration,

[0079] Fig. 3 is a schematic representation of a visualized evaluation of the method according to the invention with reference to a roadway after an extrinsic calibration,

[0080] Fig. 4 is a schematic representation of a view of a traffic scene on a roadway before conversion to a bird’s eye view,

[0081] Fig. 5 is a schematic representation of an evaluation of the method according to the invention with reference to an intersection area,

[0082] Fig. 6 is a schematic representation of a first traffic monitoring system according to the invention in a front view, and Fig. 7 is a schematic representation of a second traffic monitoring system according to the invention in a side view.

[0083] Fig. 1 shows a schematic representation of the method 100 for operating a traffic monitoring system 10 for recording a traffic scene 11 on a roadway 12. The roadway 12 is accessible to at least one vehicle 13. The method 100 comprises the following steps:

[0084] - Reading 110 of the traffic scene 11 on the roadway 12 as image data 24 by means of at least one camera unit 25, in particular a positionally fixed camera unit 25, at least one optical detection unit 16, wherein the camera unit 25 is aligned at an angle of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°, to the roadway 12,

[0085] - Recognizing 120 at least one object 27 on the vehicle 13 from the image data 24 by means of a processing unit 19, wherein the recognized object 27 serves to identify the vehicle type and / or the vehicle owner, and / or recognizing 120 a driver 28 in the vehicle 13 from the image data 24 by means of the processing unit 19 during the reading 110, wherein the object 27 and / or the driver 28 are read in with the traffic scene 11 on a roadway 12 as object data 29 and / or driver data 30,

[0086] - Detecting 120 predefined control points 17 from environmental data 18 in the acquired image data 24 by means of a processing unit 19, wherein the control points 17 are positioned at specified locations along and / or on the roadway 12 in the environmental data 18,

[0087] - converting the control points 130 into level data 20 of a level 21 by means of the processing unit 19, wherein the level 21 corresponds to the roadway 12,

[0088] - converting the captured image data 24, the plane data 20, and the object data 29 and / or driver data 30 using a transformation algorithm to Bird's Eye View data 22 of a bird's perspective for recording the traffic scene 11 on the roadway 12 by means of the processing unit 19, wherein the object data 29 and / or the driver data 30 are assigned to the vehicle 13 in the Bird's Eye View data 22,

[0089] - Evaluation 150 of the Birds-Eye-View data 22 of the recording of the traffic scene 11 on a roadway 12 to form evaluation data 35 by means of an evaluation unit 23 for operating a traffic monitoring system 10.

[0090] In Fig. 2 to Fig. 4, 3D image data 24 and in Fig. 5, 2D image data 24 were read in and further processed by the camera unit 25 of the optical detection unit 16.

[0091] In addition, the method 100 may provide for LiDAR data 14 of the traffic scene 11 on the roadway 12 to be read in by means of at least one LiDAR sensor 15 of the at least one optical detection unit 16, as indicated in Fig. 1 and Fig. 4. After the conversion 130 into the plane data 20 and before or after the conversion 140 into the bird's-eye view data 22, an overlay 160 of the image data 24 and the LiDAR data 14 to form overlay data 26 is performed.

[0092] The overlay data 26 are also converted 140 into the birds-eye view data 22 of the bird's-eye view using the transformation algorithm. This creates an accurate representation of the traffic scene 11 in a bird's-eye view.

[0093] In addition, during the reading 110 of the traffic scene 11 on a roadway 12, an object 27 on the vehicle 13 is detected in the image data 24 by means of the processing unit 19. As can be seen 120 in Fig. 5, the detected object 27 is a license plate, i.e., an object 27 for identifying the vehicle owner.

[0094] It would also be conceivable to detect an object 27 to identify the vehicle type. In addition, a driver 28 was detected in the image data 24 by the processing unit 19 for at least one of the vehicles 13 located in the traffic scene 11, since a red light violation occurred.

[0095] In this case, the object 27 or the objects 27 and the driver 28 are read in with the traffic scene 11 on a roadway 12 as object data 29 and the driver 28 with the traffic scene 11 on a roadway 12 as driver data 30. Both the object data 29 and the driver data 30 are converted 140 with the image data 24 and the layer data 20 using a transformation algorithm to the birds-eye view data 22 of a bird's-eye view for recording the traffic scene 11 on the roadway 12 by means of the processing unit 19. In the evaluation and especially in the visualization of the evaluation, the object data 29 and the driver data 30 are assigned to the vehicle 13 in the birds-eye view data 22, as shown in Fig. 5. Optionally, image data can be combined with lidar data or any sensor data (e.g. radar data, ultrasonic sensor data).

[0096] Fig. 2 shows a representation of the roadway 12 before an extrinsic calibration of the optical detection unit 16. It can be seen 120 that the lane markings 44 stored for evaluation (shown in dashed lines) do not match the lane markings 45 of the surrounding lane; in this case, an incorrect evaluation of the traffic scene 11 may occur.

[0097] The calibration and alignment 180 of the optical detection unit 16 to the roadway 12 with an extrinsic calibration 170 is initially performed once and / or optionally automatically. It is conceivable to repeat this calibration if a deviation of the actual position from the target position of the optical detection unit 16 is detected. In this case, the data from the extrinsic calibration is used for precise positional conversion into the plane data 20 of plane 21. The calibration is only considered complete when the lane markings 44 stored for evaluation have been aligned with the lane markings 45 of the surrounding lane, as shown in Fig. 3.For calibration, especially for recalibration of the optical detection unit 16 of the traffic monitoring system 10 or of the method 100 itself, a 2D code 31, in particular a QR code, is provided at a specific position on a stop line 32 on the roadway 12 in the traffic scene 11 on the roadway 12. This step is shown as an optional step in dashed lines in Fig. 1. Such a 2D code can, for example, be attached to a stop line 32, as shown schematically in Fig. 5.

[0098] Depending on the traffic scene 11 to be monitored on the roadway 12, at least one measuring line 33 is stored in reference data 34 which is stored in the evaluation unit 23.

[0099] In this case, the at least one measuring line 33 can be or become aligned orthogonally or substantially orthogonally to a road boundary and / or orthogonally or substantially orthogonally to a lane direction of the roadway 12 and / or parallel or substantially parallel to a front and / or rear bumper of a vehicle 13 detected in the image data 24 and / or lidar data 14, and wherein the measuring line 33 is used in the evaluation of the evaluation data 35 of the traffic scene 11.

[0100] In Fig. 3, two measuring lines 33 are stored in the reference data 34. These run orthogonally or essentially orthogonally to a lane boundary and are used to measure speed or distance in order to detect violations in this regard. In Fig. 5, several measuring lines 33 are stored; these measuring lines 33 each run along the traffic light stop line, and thus partially across lane 12, but orthogonally to the lane boundary. Another measuring line 33 indicates a no-parking zone and runs parallel to a lane boundary.

[0101] The method 100 is performed at different times or at a

[0102] Triggering event, i.e. when a vehicle 13 identified in the image data 24 crosses the at least one measuring line 33, such as a red light violation in Fig. 5.

[0103] For reading 110 the traffic scene 11 on the roadway 12, several spaced-apart optical detection units can be provided so that the traffic scene 11 can be displayed from different angles. This allows the image data 24 or the overlay data 26 to be recorded and overlaid at different times and / or from different angles.

[0104] The vehicle speed and / or vehicle acceleration of at least one vehicle 13 on the roadway 12 can be evaluated from the birds-eye view data 22 at one or more times. Additionally or alternatively, a stopping violation is evaluated. For example, in Fig. 5, a stopping violation in a traffic light zone, i.e., a red light violation, and a parking violation are evaluated using the birds-eye view data 22. The traffic scenes 11 of Figs. 2 to 4 are each evaluated to form evaluation data 35.

[0105] These evaluation data 35 are transmitted, 210, to an external data storage 38 or external further processing unit 39 by means of a data interface 36 or a transmitter 37 of the evaluation unit 23.

[0106] In order to always have precise and, above all, legally sound evaluations and evaluation data 35, at least one position and at least one coordinate of at least one data point of the lidar data 14 are compared 220 with a pixel of the image data 24 corresponding to the data point during the overlay 160. An alarm and / or a rejection 190 of the measured value is triggered if a deviation between the data point and the pixel is detected. Simultaneously, a new extrinsic calibration is triggered upon detection of the deviation.

[0107] In addition to the image data 24 and the lidar data 14, radar data 40 is acquired by means of at least one radar sensor 41 of the at least one optical detection unit 16, and ultrasound data 42 is acquired by means of at least one ultrasound sensor 43 of the at least one optical detection unit 16. The radar data 40 and the ultrasound data 42 are used to convert them into the plane data 20 and to convert them into the birds-eye view data 22, to evaluate 150 the birds-eye view data 22, and / or to overlay 160 them onto the overlay data 26.

[0108] In Fig. 6 and Fig. 7, a traffic monitoring system 10 for carrying out the method 100 according to Figs. 1 to 5 is shown. This has at least one optical detection unit 16 with at least one camera unit 25 for reading 110 the traffic scene 11 on a roadway 12 as image data 24, and a processing unit 19 for detecting 120 an object 27 on the vehicle 13 and / or a driver 28 in the vehicle 13 and for detecting 120 predefined control points 17 from environmental data 18 in the image data 24, for converting the control points 130 to plane data 20 of plane 21, wherein plane 21 corresponds to the roadway 12, and for converting the detected image data 24 and the plane data 20 using a transformation algorithm to bird's-eye view data 22, and an evaluation unit 23 for evaluating 150 the traffic scene 11 on a roadway 12 to the Evaluation data 35.

[0109] The camera unit 25 is aligned at an angle a of 0° to 70°.

[0110] The evaluation unit 23 has the data interface 36 or the transmitter for transmitting 210 the evaluation data 35 to an external data storage 38 or external further processing unit 39.

[0111] In addition, the at least one optical detection unit 16 in Fig. 6 has at least one camera unit 25 for capturing image data 24 from images, at least one radar sensor 41 for capturing radar data 40, and at least one ultrasonic sensor 43 for capturing ultrasonic data 42. Here, it is conceivable that the lidar sensor 15 and / or the radar sensor 42 and / or the ultrasonic sensor 43 are also aligned at an angle a of 0° to 70° to the roadway. The traffic monitoring system according to Fig. 6 is attached to a gate at a height of 2.3 m to 2.7 m or optionally up to 10 m, and can thus be arranged above the roadway 12, and provides a plurality of optical detection units for reading 110 of the traffic scene 11 on the roadway 12. These optical detection units are arranged at a distance from one another in order to be able to record different angles of the traffic scene 11.

[0112] The traffic monitoring system 10 in Fig. 7 is attached to a mast, here a traffic light, next to the roadway 12, and in this case is at least partially integrated into the mast or the traffic light.

[0113] List of reference symbols

[0114] 10 Traffic monitoring system

[0115] 11 Traffic scene

[0116] 12 lane

[0117] 13 vehicles

[0118] 14 Lidar data

[0119] 15 Lidar sensor

[0120] 16 optical detection unit

[0121] 17 checkpoints

[0122] 18 Environmental data

[0123] 19 Processing unit

[0124] 20 layer data

[0125] 21 Level

[0126] 22 Birds-Eye View data

[0127] 23 Evaluation unit

[0128] 24 image data

[0129] 25 Camera unit

[0130] 26 Overlay data

[0131] 27 objects

[0132] 28 drivers

[0133] 29 Object data

[0134] 30 driver data

[0135] 31 2D code

[0136] 32 stop line

[0137] 33 measuring line

[0138] 34 Reference data

[0139] 35 evaluation data

[0140] 36 Data interface

[0141] 37 stations

[0142] 38 external data storage

[0143] 39 external processing unit 0 Radar data 1 Radar sensor 2 Ultrasonic data 3 Ultrasonic sensor

[0144] 100 procedures

[0145] 110 Reading

[0146] 120 Recognize

[0147] 130 Converting the control points

[0148] 140 Convert to Birds Eye View data

[0149] 150 Evaluate

[0150] 160 Overlay

[0151] 170 Calibration

[0152] 180 Align

[0153] 190 Discard

[0154] 200 triggers

[0155] 210 Transfer

[0156] 220 Compare a Angle

Claims

Patent claims 1. A method (100) for operating a traffic monitoring system (10) for recording a traffic scene (11) on a roadway (12), wherein the roadway (12) is passable by at least one vehicle (13), the method (100) comprising the following steps: - reading (110) the traffic scene (11) on the roadway (12) as image data (24) by means of at least one camera unit (25), in particular a position-fixed camera unit (25), at least one optical detection unit (16), wherein the camera unit (25) is aligned at an angle (a) of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°, to the roadway (12), - recognizing (120) at least one object (27) on the vehicle (13) from the image data (24) by means of a processing unit (19), wherein the recognized object (27) serves to identify the vehicle type and / or the vehicle owner, and / or recognizing (120) a driver (28) in the vehicle (13) from the image data (24) by means of the processing unit (19) during the reading (110), wherein the object (27) and / or the driver (28) are read in with the traffic scene (11) on a roadway (12) as object data (29) and / or driver data (30), - detecting (120) predefined control points (17) from environmental data (18) in the acquired image data (24) by means of a processing unit (19), wherein the control points (17) are positioned at specified locations along and / or on the roadway (12) in the environmental data (18), - converting the control points (130) into plane data (20) of a plane (21) by means of the processing unit (19), the plane (21) corresponding to the roadway (12), - Converting the captured image data (24), the plane data (20), and the object data (29) and / or driver data (30) into Bird's Eye View data (22) using a transformation algorithm a bird's-eye view for recording the traffic scene (11) on the roadway (12) by means of the processing unit (19), wherein the object data (29) and / or the driver data (30) are assigned to the vehicle (13) in the birds-eye view data (22), - evaluating (150) the birds-eye-view data (22) of the recording of the traffic scene (11) on a roadway (12) to form evaluation data (35) by means of an evaluation unit (23) for operating the traffic monitoring system (10).

2. Method (100) according to claim 1, characterized in that an extrinsic calibration (170) is carried out for aligning (180) the optical detection unit (16) to the roadway (12), wherein the extrinsic calibration is carried out once or cyclically and / or wherein the extrinsic calibration (170) is carried out automatically, at least partially automatically, or manually, in particular wherein the data from the extrinsic calibration is used for positionally accurate conversion into the plane data (20) of the plane (21).

3. Method (100) according to claim 2, characterized in that the at least one optical detection unit (16) of the traffic monitoring system (10) and / or the method (100) is calibrated and / or aligned by means of a 2D or 3D code (31), in particular a QR code, at a specific position on a stop line (32) on the roadway (12) in the traffic scene (11) on the roadway (12).

4. Method (100) according to one of the preceding claims, characterized in that that the traffic scene (11) on the roadway (12) is read by means of the optical detection unit (16) from a height of 1.5 m to 10 m, preferably 2 m to 2.8 m, more preferably 2.3 m to 2.7 m.

5. Method (100) according to one of the preceding claims, characterized in that, depending on the traffic scene (11) to be monitored, at least one measuring line (33) is stored on the roadway (12) in reference data (34) which is stored in the evaluation unit (23), wherein the at least one measuring line (33) is or will be aligned orthogonally or substantially orthogonally to a roadway boundary and / or orthogonally or substantially orthogonally to a lane direction of the roadway (12) and / or parallel or substantially parallel to a front and / or rear bumper of a vehicle (13) detected in the lidar data (14) and / or image data (24), and wherein the measuring line (33) is used in the evaluation of the evaluation data (35) of the traffic scene (11).

6. Method (100) according to one of the preceding claims, characterized in that the method (100) is carried out at different times, in particular cyclically, and / or upon a triggering event, in particular a crossing of a vehicle (13) identified in the image data (24) over the at least one measuring line (33).

7. Method (100) according to one of the preceding claims, characterized in that for reading (110) the traffic scene (11) on the roadway (12) a plurality of optical detection units (16) spaced apart from one another are provided for detecting the traffic scene (11) from different angles.

8. Method (100) according to one of the preceding claims, characterized in that from the birds-eye view data (22) at one point in time or at several points in time, a vehicle speed and / or vehicle acceleration of at least one vehicle (13) on the roadway (12), and / or a stopping violation, in particular a stopping violation in an intersection area or a traffic light area of the traffic scene (11), and / or a distance violation of a vehicle (13) from a preceding vehicle (13) is evaluated to the evaluation data (35).

9. Method (100) according to one of the preceding claims, characterized in that the evaluation data (35) are transmitted (210) by means of a data interface (36) or a transmitter (37) of the evaluation unit (23) to an external data storage (38) or external further processing unit (39).

10. Traffic monitoring system (10) for carrying out the method (100) according to one of the preceding claims, with at least one optical detection unit (16) with at least one camera unit (25) for reading (110) the traffic scene (11) on a roadway (12) as image data (24), wherein the camera unit (25) is aligned at an angle (α) of 0° to 70°, preferably 10° to 70°, more preferably 15° to 70°, to the roadway (12), and with a processing unit (19) for detecting (120) an object (27) on the vehicle (13) and / or a driver (28) in the vehicle (13) and for detecting (120) predefined control points (17) from environmental data (18) in the image data (24), for converting the control points (130) into plane data (20). the plane (21 ), wherein the plane (21 ) corresponds to the roadway (12), and for converting the acquired image data (24) and the plane data (20) using a Transformation algorithm to birds-eye view data (22) of the bird's perspective, and with an evaluation unit (23) for evaluating (150) the traffic scene (11) on a roadway (12) to the evaluation data (35).

11. Traffic monitoring system (10) according to claim 10, characterized in that a plurality of optical detection units (16) are provided for reading (110) the traffic scene (11) on the roadway (12), wherein the optical detection units are arranged at a distance from one another.

12. Traffic monitoring system (10) according to one of claims 10 or 11, characterized in that the at least one optical detection unit (16) is arranged next to the roadway (12) and / or above the roadway (12).

13. Traffic monitoring system (10) according to one of claims 10 to 12, characterized in that the optical detection unit (16) is arranged at a height of 1.5 m to 10 m, preferably 2 m to 2.8 m, more preferably 2.3 m to 2.7 m.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (100) according to one of claims 1 to 9 by means of a traffic monitoring system (10) according to one of claims 10 to 13.

Images