Image processing method for determining the speed of a vehicle, computer program and image processing device for implementing the image processing method

Abstract

The invention relates to an image processing method for determining the speed of a vehicle 2 in a monitoring area 5, wherein a monitoring scene 6 with at least one vehicle 2 can be arranged in the monitoring area 5, wherein at least one first image of the monitoring scene 6 is captured, wherein a first timestamp is assigned to the first image, and wherein a second image of the monitoring scene 6 is captured, wherein a second timestamp is assigned to the second image 100, wherein a first polyhedron 8 is derived from the first image and a second polyhedron 8 is derived from the second image, wherein the first and the second polyhedron each represent the vehicle 2 200, wherein vehicle dimension information is determined for the first polyhedron and / or for the second polyhedron 300,where, based on the vehicle dimension information and the polyhedron 8, a distance traveled by the vehicle 2 in the monitoring area 5 between the timestamps is determined, where the vehicle speed is determined based on the distance and the timestamps 500.

Description

[0001] The invention relates to an image processing method for determining driving speed. The invention also relates to a corresponding computer program and an image processing device for implementing the image processing method. State of the art

[0002] Monitoring the speeds of different road users on highways, intersections or in parking garages is a fundamental objective of intelligent traffic monitoring systems.

[0003] Camera-based speed estimation typically requires a static and calibrated camera system. Calibration ensures that the camera's position and orientation relative to the road, as well as its intrinsic characteristics, are known, which is used to estimate metric quantities. Disclosure of the invention

[0004] The invention relates to an image processing method for determining a driving speed with the features of claim 1. The invention also relates to a computer program with the features of claim 11 and an image processing device with the features of claim 12. Further features and advantages of the invention will become apparent from the dependent claims, the following description and the accompanying figures.

[0005] The invention relates to an image processing method for determining the speed of a vehicle within a monitoring area. The monitoring area can contain a scene with at least one vehicle. The vehicle is, in particular, a road vehicle, specifically a passenger car, truck, bus, etc. The monitoring scene is specifically designed as a driving area for vehicles. The monitoring scene can be implemented as a road, a highway, an intersection, a parking lot, or the like.

[0006] The driving speed is defined as an absolute speed within the actual monitoring area and is given, for example, in kilometers per hour or mph.

[0007] At least one first image and one second image of the monitored scene are captured. This capture is primarily achieved using a camera. The camera is, for example, a surveillance camera. It can be a black and white or color camera. Technically, it can be a CCD or CMOS camera. The camera includes a lens, and the monitored scene within the monitored area is projected onto a sensor, which generates the first and second images.

[0008] Each of the first and second images is assigned a timestamp. This timestamp can be relative, allowing the time difference between the first and second images to be determined. Alternatively, it can be absolute, representing the current time in the monitored area, optionally supplemented with the date. The timestamp can be assigned to the images directly by the camera, or it can be added in real time during subsequent analysis or image transmission. Specifically, the timestamp represents the point in time when the images were captured.

[0009] A first polyhedron is derived from the first image, and a second polyhedron is derived from the second image, with both representing the same vehicle. Specifically, the same definitions for deriving, and in particular generating, the polyhedra are applied to both the first and second images.

[0010] Based on the polyhedron, a polyhedron position can be defined. This position can be, for example, the centroid, the centroid projected onto the base, or the center point of the polyhedron; preferably, the position is defined as a vertex, center point, or foot of the polyhedron.

[0011] Vehicle dimension information is determined for the first and / or the second polyhedron. This determination can be made by deriving the first and / or the second polyhedron. Alternatively, it can be determined subsequently. The vehicle dimension information comprises at least or exactly one vehicle dimension in one dimension. Alternatively, the vehicle dimension information can comprise exactly two vehicle dimensions in two independent dimensions. Further data processing is simplified if the vehicle dimension information comprises exactly three vehicle dimensions in three independent dimensions. The vehicle dimension information and / or the vehicle dimension is specifically defined as an absolute length measurement within the relevant monitoring area and is expressed, for example, in meters or centimeters.

[0012] The preferred format for vehicle dimensions is vehicle height, vehicle width, and / or vehicle length. If only one dimension is used, one length measurement is used; if exactly two dimensions are used, two length measurements are used; and if three dimensions are used, all three length measurements are used.

[0013] In particular, the vehicle dimension information forms a basis for converting the dimensions of polyhedra or sections thereof from pixels into absolute length measurements. In other words, by considering, and especially calculating, the polyhedra and vehicle dimension information together, the actual size of the polyhedra within the monitored area is known and / or determinable.

[0014] Based on the vehicle dimensions and the polyhedra, the distance of the vehicle within the monitored area between the timestamps and / or between the first and second images is determined. This is possible because, by knowing the length measurements and the geometry of the polyhedra, the distance within the monitored area can be inferred from the actual size. The distance is specifically defined as an absolute length measurement within the actual monitored area and is given, for example, in meters or centimeters.

[0015] Based on the determined distance and the timestamps, the vehicle speed can be determined by dividing the distance (as the route traveled) by the time difference between the timestamps.

[0016] The invention has the advantage that image-based speed measurement can be implemented with an uncalibrated camera system. Calibration is replaced by the inclusion of vehicle dimension information. In particular, some or all of the following advantages can be achieved: Simplified commissioning of the camera and image processing setup, as the complex calibration process is eliminated. Changes in camera pose (e.g., due to environmental factors like wind) do not lead to errors in speed estimation. Systems based on calibration, on the other hand, must be recalibrated to produce valid estimates. Changes in the camera's intrinsic characteristics (e.g., due to temperature fluctuations) are taken into account with every measurement. Systems with static calibration parameters do not consider this and are less robust. By using polyhedral detection, vehicles are captured more precisely in the image compared to conventional 2D detection, enabling more accurate motion estimation.

[0017] It is particularly preferred that the polyhedron completely encloses the vehicle and / or that the vehicle is completely arranged within the polyhedron, including any attachments.

[0018] In particular, coordinates are derived, defining projected vertices of the polyhedron in the image. These coordinates are specifically designed as image coordinates and / or 2D coordinates and represent points in the image on or within the image plane. Eight vertices are determined, defining six faces (front, rear, top, bottom, left, right in the direction of travel). Depending on the vehicle's geometry, right angles are not necessarily required. The polyhedron preferably encloses the vehicle in such a way that neither parts of the vehicle protrude beyond the polyhedron, nor is it too large, creating a gap between the polyhedron and the vehicle. In particular, the polyhedron has six faces, specifically as a cuboid, and most preferably as a rectangular cuboid. The polyhedron represents the vehicle.

[0019] The determined coordinates can encompass eight vertices of the cuboid and thus define it. Alternatively, only four vertices of the cuboid are determined, whereby the four vertices of the cuboid do not lie in a common plane, and the eight vertices of the cuboid are deduced by linear combination of the four vertices.

[0020] In a preferred embodiment of the invention, the coordinates are configured as p3D coordinates. These are, in particular, the projection of the eight (3D) vertices of the vehicle-enclosing polyhedron in the image. The p3D coordinates, as a frame projection, are the projection onto the image of a (conceived in the real world) polyhedron, preferably a cuboid, that directly surrounds the object, forming a 3D frame. The 3D frame and / or the polyhedron, especially the cuboid, is formed, in particular, by straight line segments. "Directly surrounding" is to be understood as meaning that a surface defined by the imagined 3D frame or cuboid (e.g., a polyhedron defined by the frame) surrounds the vehicle as closely as possible, for example, with the smallest possible volume. The respective surfaces and edges of the surface spanned by the 3D frame or the cuboid touch the surface of the object.In other words, a so-called "3D bounding box" is determined, specifically as a 3D frame or cuboid. The 3D frame is, for example, an enclosing body in the form of a rectangular cuboid.

[0021] In a preferred embodiment of the invention, the images are captured and / or selected such that the distance between the vehicle positions of the two images is less than 5 m, preferably less than 2 m. This ensures that a realistic distance is determined even when cornering. Particularly preferably, a plurality of images are captured, with the vehicle being tracked, for example, using a tracking module, so that the vehicle speed can be determined more accurately. Thus, a trajectory of the vehicle is determined, and a local vehicle speed can be calculated between the respective vehicle positions, which can then be combined for the entire trajectory to obtain a common, and in particular averaged, vehicle speed.

[0022] It is preferred that the vehicle dimension information be provided as supplementary information, in particular independent supplementary information. The vehicle dimension information is obtained, in particular, from a reference work, especially from a database.

[0023] In a first embodiment of the invention, the vehicle dimension information is determined assuming a standard vehicle size. In other words, a single size is used for all vehicles.

[0024] In a potential further training program, the vehicle's vehicle class is first determined. This class could include, for example, passenger cars, trucks, trucks with trailers, buses, heavy transport vehicles, etc. The vehicle class can be determined using digital image processing and established pattern recognition. Alternatively, a deep neural network can be used for vehicle class identification. Other machine learning techniques can also be employed. Depending on the vehicle class, a standard size is used as vehicle dimension information. The standard size for trucks is therefore larger than the standard size for passenger cars. Further subdivisions of the vehicle class are possible.

[0025] In the aforementioned explanations, the vehicle dimension information, namely the respective standard size, is provided as independent additional information, for example taken from a database or a set of rules.

[0026] In a possible further development of the invention, the vehicle model, in particular the vehicle variant, is determined, wherein the vehicle dimension information is queried from a database based on the vehicle model. Here, the classification into vehicle classes is further refined, namely to the vehicle models. The determination of the vehicle model can be carried out using the same techniques as the determination of the vehicle class, so reference is made to the preceding description.

[0027] It is also possible that the vehicle dimension information is not provided as independent additional information, but rather estimated from the images or the image itself. This can be implemented in particular using digital image processing or machine learning methods, such as a deep neural network.

[0028] The image processing method makes assumptions that are always met in typical use cases: a. In general, it is assumed that the (unknown) intrinsic calibration of the camera exhibits approximately perspective effects locally (i.e., in the image region where the vehicle is located at the time of recording) (i.e., is distortion-free). b. Furthermore, it is assumed that the vertices of the polyhedron are in the same physical position relative to the vehicle in both images. For example, a point of the polyhedron could be laterally at the level of the side mirror, longitudinally at the level of the license plate, and vertically at the level of the roof, OR offset by any value from these positions (e.g., 10 cm in the direction of the license plate). This offset must be exactly the same for both images.

[0029] In a possible further development of the invention, a pixel offset is calculated between the polyhedra in the images. Based on the polyhedra and the vehicle dimension information, this pixel offset is converted into the distance within the monitored area. In this way, the distance can be determined via the pixel offset and other known parameters. It is assumed that the vehicle moves approximately linearly (i.e., without sharp turns). It is further assumed that the image coordinates of the polyhedron (or p3D box in particular) are available.

[0030] Without pseudo-calibration: 1) A length in the direction of travel (e.g. vehicle length or wheelbase) as vehicle dimension information: If a length in the direction of travel is given, i.e. on the polyhedron, and there is an edge that points in the direction of travel (straight ahead) and whose length is known, then the speed can be determined via a 1D homograph (see, for example, "Multiple View Geometry in Computer Vision, second edition" (page 44)). 2) Two lengths in a plane parallel to the base plane / roadway that are not parallel, and a known angle between the lines measuring these lengths. The polyhedron must also have at least four vertices in the plane (e.g., vehicle width and vehicle diagonal length): If two lengths are known that measure non-parallel line segments, and the corresponding points of the polyhedron lie in a plane parallel to the road surface, then the velocity can be determined via a 2D homographic decomposition. An example of this would be the width and diagonal of a rectangle encompassing the vehicle on the ground plane.

[0031] Alternatively, a local pseudo-calibration can be performed using the known metric quantities, in particular the vehicle dimension information, and the polyhedra.

[0032] For pseudo-calibration, it is assumed that any global distortion (which does not change significantly locally) is isotropic, meaning it does not lead to unequal compression / stretching along the two image axes associated with the first and second images. In pseudo-calibration, a calibration matrix K is estimated from the vertices and corresponding line segments of the polyhedron; this matrix is ​​valid for the local imaging. It includes a focal length and a principal point. Assumption (a.) implies that the focal lengths for both image directions can be assumed to be the same (locally), meaning there is no difference in focal length, for example, in the x and y directions. If the camera's intrinsic calibration is known, any distortions, etc., can of course be present. Pseudo-calibration can, for example,The vanishing points are determined as principal points, which result from the parallel lines of the polyhedra, particularly the p3D box. This method exploits the fact that the axes of the polyhedra, especially the p3D box, are perpendicular to each other and parallel in the world. In this case, it is possible to determine the distance traveled from any length between two polyhedral points in the plane, or in the case of a point in the plane and another point vertically above it. The name "pseudo-calibration" was chosen to clarify that the calibration does not need to be valid for the entire image area of ​​the camera.

[0033] Another subject matter of the invention relates to a computer program having the features of claim 11. Another optional subject matter of the invention relates to a digital storage medium, wherein the computer program is stored on the storage medium.

[0034] Another aspect of the invention relates to an image processing device designed to implement the image processing method as previously described or according to one of the claims.

[0035] The image processing device is specifically designed as a digital data processing device, such as a computer, a cloud instance, etc.

[0036] The image processing device includes an input interface for receiving images from a camera, with each image assigned a timestamp. Optionally, the camera is a component of the image processing device. The timestamp can be applied by the camera and / or by the image processing device.

[0037] The image processing device includes a determination unit for identifying polyhedra and vehicle dimension information. The image processing device can have several modules, for example, a polyhedron module that initially identifies the polyhedra in the images. Subsequently, a dimension determination module can be provided that determines the vehicle information. The polyhedron module can, for example, be based on digital image processing, while the dimension determination module can comprise a database and / or a set of rules. Alternatively, the polyhedron module of the dimension determination module can also be implemented as an artificial intelligence, in particular a neural network, which identifies the polyhedra and the vehicle information.

[0038] Furthermore, the image processing device has an evaluation unit for determining the vehicle speed by determining the distance of the vehicle in the monitoring area and calculating it with the timestamps.

[0039] Optionally, the image processing device includes a tracking module which tracks, in particular follows, the vehicle along a trajectory over a plurality of images, and determines a plurality of local vehicle speeds between the images and summarizes from them a common, in particular averaged, vehicle speed for the trajectory or sections thereof.

[0040] Further advantages and features of the invention will become apparent from the following description of a preferred embodiment and the accompanying figures. These show: Fig. 1. A schematic block diagram of an image processing device and an illustration of the corresponding method; Fig. 2 a schematic representation of a surveillance scene in a surveillance area with a vehicle to which a polyhedron has been assigned.

[0041] The Fig. Figure 1 shows a schematic block diagram of an image processing device 1 for implementing a method for determining the driving speed of a vehicle 2 ( Fig. 2) The image processing device 1 is designed, for example, as a digital data processing device, such as a computer or an instance in the cloud.

[0042] The image processing device 1 has an input interface 3 for receiving images, in particular a first and a second image, from a camera 4. The camera 4 is focused on a monitoring area 5 ( Fig. 2) directed, which is a surveillance scene 6 ( Fig. 2) with the vehicle 2. Optionally, the camera 4 forms a component of the image processing device 1. Preferably, the camera 4 is arranged in a stationary position during the execution of the method and / or has a stationary field of view. In step 100, the images are acquired and / or made available to the image processing device 1.

[0043] The image processing device 1 includes a detection device 7, wherein the images from the camera 4 are directed to the detection device 7. Optionally, a timestamp is assigned to the images by the camera 4, by the detection device 7, or at another location, which allows a time interval between the images to be determined. For example, the timestamp is configured as absolute information, such as Central European Time with a date, or as relative information, so that time information between the images can be determined.

[0044] The determining device 7 is designed to produce a polyhedron 8 from each of the images for the vehicle 2 ( Fig. 2) and to determine vehicle dimension information for the polyhedron 8.

[0045] In principle, the determination device 7 can be designed, for example, as a black box, such as a neural network or an AI, which determines the polyhedra 8 and the vehicle dimension information via the neural network or the other AI. In the illustrated embodiment, the polyhedra 8 are determined via digital image processing in a polyhedron module 9.

[0046] In step 200, the polyhedra 8 are determined from the images using the polyhedral module 9. Specifically, each image shows vehicle 2, but at different positions. A first and a second image are provided, with a first polyhedron 8 being determined in the first image and a second polyhedron 8 in the second image. The first and second polyhedra 8, or more generally, the polyhedra 8, each represent vehicle 2 at its different position.

[0047] The Fig. Figure 2 shows such an image, with polyhedron 8 graphically superimposed. Polyhedron 8 is designed as a rectangular cuboid and closely encloses the vehicle. In the Fig.Figure 2 schematically depicts such an image, showing a surveillance area 5 with the surveillance scene 6 in the form of a road containing vehicle 2. Vehicle 2 is enclosed within polyhedron 8, which can be seen as completely enclosing vehicle 7, so that polyhedron 8 forms a contour representing vehicle 7. Furthermore, polyhedron 8 is positioned as closely as possible around vehicle 2, ensuring that the contour is not too large. It is important to note that polyhedron 8 is based solely on the outer dimensions of vehicle 2 and requires no other keypoints. Polyhedron 8 is designed as a rectangular prism, defined by vertices 11 in image coordinates. Specifically, so-called p3D coordinates are used.

[0048] The determination device 7 has a dimension determination module 12, which is configured to determine vehicle dimension information for the polyhedra 8, in particular for the first and the second polyhedra 8, in a single step 300. There are various possibilities for this: In principle, the dimensioning module 12 accesses a database 13 – also called a rule set – in which the vehicle dimension information is stored. Database 13 can be located locally or in the cloud. The vehicle dimension information is thus provided as supplementary information to the images, in particular as independent supplementary information, or as prior knowledge.

[0049] One possibility is to determine the vehicle dimension information assuming a standard size for vehicle 2. In this case, each vehicle is estimated to be the same. This approach is comparatively inaccurate, but may suffice if the corresponding vehicles 2 are approximately the same size.

[0050] Alternatively, a vehicle class for vehicle 2 is determined, whereby the vehicle dimension information is determined assuming a standard size for the vehicle class. Thus, different vehicle dimension information is taken from database 13, depending on whether vehicle 2 is configured as a passenger car, truck, bus, lorry, truck with trailer, etc.

[0051] In another alternative embodiment, the vehicle model, including the vehicle variant, is determined, and the vehicle dimension information is queried from database 13 based on the vehicle model. This method is particularly accurate because corresponding vehicle dimension information can be used for each vehicle model.

[0052] In other embodiments, the vehicle dimension information is estimated from the images, so that such a database 13 can be dispensed with.

[0053] The polyhedron module 9 can, for example, determine the polyhedron 8 using digital image processing. Determining the vehicle class or model can also be achieved through digital image processing, particularly pattern recognition. Alternatively, a suitable neural network or AI can be used, which has been previously trained with polyhedra, vehicle classes, or vehicle models.

[0054] Training data is therefore provided by images of vehicles with a known polyhedron, a known vehicle class or a known vehicle model, which are labeled, for example, so that the neural network or other AI can be trained in a simple, known way.

[0055] While it is possible to determine the speed of vehicle 2 using only two images, it is preferable to use multiple images, allowing a trajectory of vehicle 2 to be generated. This trajectory is generated in step 400 within a tracking module 14. Here, vehicles 2 are assigned to the same vehicle 2 in successive images. The trajectory then encompasses various vehicle positions, and the local vehicle speed or a common vehicle speed is determined for each pair of consecutive images or for the entire trajectory, as explained below. The tracking module 14, or step 400, can also be located upstream of the measurement module 12 and thus upstream of step 300.For each vehicle 2 in the images of the trajectory, the vehicle information only needs to be determined in principle, since the vehicle information is absolute information about the respective vehicle 2.

[0056] In principle, vehicle dimension information can only exist in one dimension, so that, for example, only one vehicle length, one vehicle width, one vehicle height, or another length measurement is used. More precise methods determine at least or exactly length measurements for two independent dimensions as vehicle dimension information. The method is particularly simple and accurate when length measurements are available for all three independent dimensions, so that, for example, the vehicle dimension information consists of the vehicle length, the vehicle width, and the vehicle height, which can then be processed using the polyhedra 8.

[0057] The vehicle dimension information and the polyhedra are transferred to an evaluation unit 15 of the image processing device 1 for step 500. The evaluation unit 15 is configured to perform step 500, whereby, with knowledge of the vehicle information and the polyhedra 8, a distance of the vehicle 2 in the monitoring area between the timestamps is determined.

[0058] The vehicle speed is then determined based on the calculated distance between two images and the time difference between the images' timestamps. If a trajectory has been determined, local vehicle speeds are measured between the images, and from these, a common vehicle speed or a common speed profile of vehicle 2 along the trajectory is calculated.

[0059] The image processing device 1 has an output interface 16 which is configured to output the vehicle speed, optionally with identification data for identifying the vehicle 2, and optionally the images to a monitoring center 17, another database 18 or a display device 19 for further processing, for example for traffic monitoring, in a step 600. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Multiple View Geometry in Computer Vision, second edition” (page 44

[0030]

Claims

[1] Image processing method for determining the speed of a vehicle (2) in a monitoring area (5), wherein a monitoring scene (6) with at least one vehicle (2) may be arranged in the monitoring area (5), wherein at least one first image of the surveillance scene (6) is captured, wherein the first image is assigned a first timestamp, and wherein a second image of the surveillance scene (6) is captured, wherein the second image is assigned a second timestamp (100), wherein a first polyhedron (8) is derived from the first image and a second polyhedron (8) is derived from the second image, wherein the first and second polyhedra each represent the vehicle (2) (200), where vehicle dimension information is determined for the first polyhedron and / or for the second polyhedron (300), wherein, based on the vehicle dimension information and the polyhedron (8), a distance traveled by the vehicle (2) in the monitoring area (5) between the timestamps is determined, wherein the vehicle speed is determined based on the distance and the timestamps (500). [2] Image processing method according to claim 1, characterized by , that the polyhedra (8) enclose the vehicle (2). [3] Image processing method according to claim 1 or 2, characterized by , that the polyhedra (8) are determined in p3D coordinates. [4] Image processing method according to any one of the preceding claims, characterized by , that the distance of the vehicle (2) between the two images is less than 5 m, preferably less than 2 m. [5] Image processing method according to any one of the preceding claims, characterized by , that the vehicle dimension information is determined assuming a standard size of the vehicle (2). [6] Image processing method according to any one of the preceding claims, characterized by , that a vehicle class of the vehicle (2) is determined and the vehicle dimension information is determined assuming a standard size of the vehicle class. [7] Image processing method according to any one of the preceding claims, characterized by , that the vehicle model of the vehicle (2) is determined, whereby the vehicle dimension information is queried from a database (13) based on the vehicle model. [8] Image processing method according to any one of the preceding claims, characterized by , that the vehicle dimension information is estimated from the images. [9] Image processing method according to any one of the preceding claims, characterized by , that the distance traveled is determined by converting the pixel offset of the polyhedra (8) in the images. [10] Image processing method according to any one of the preceding claims, characterized by, that for each of the polyhedra (8) a local pseudo-calibration is performed using a calibration matrix, whereby the distance traveled is determined on the basis of the calibration matrices and the vehicle dimension information. [11] Computer program, wherein the computer program is configured and / or set up to execute, apply and / or implement the image processing method according to any one of claims 1 to 10 during its execution. [12] Image processing device (1) for implementing the image processing method according to one of claims 1 to 10, with an input interface (3) for receiving images from a camera (4), wherein a timestamp is assigned to the images, with a determining device (7) for determining the polyhedra (8) and the vehicle dimension information with an evaluation device (15) for determining the vehicle speed by determining the distance of the vehicle (2) in the monitoring area (5) between the images and calculating the timestamps of the images.

Images