Image processing method for determining vehicle dimensions of a vehicle in a monitoring area, computer program, storage medium, dimension determination device and traffic monitoring system

Abstract

The invention relates to an image processing method for determining the vehicle dimensions of a vehicle 7 in a monitoring area 8, wherein a monitoring scene 9 with at least one vehicle 7 can be arranged in the monitoring area 8, wherein in a first processing step 200, starting from at least one image 10 of the monitoring area 10, a position range of the vehicle 7 in the image is determined and image position information for the position range is provided, wherein in a second processing step 300, coordinates 11 are derived on the basis of the image position information, wherein the coordinates 11 define the projected vertices of a polyhedron 12 in the image 10, wherein the cuboid 12 represents the vehicle 7, wherein the second processing step 300 is implemented via a neural network, wherein the vehicle dimensions are determined on the basis of the coordinates 11.

Description

State of the art

[0001] In road traffic, there are often sections of road that have strict dimensional restrictions. Examples include tunnel entrances, parking garage entrances, road narrowings at construction sites, and the like.

[0002] In principle, it is known to detect vehicles using digital image processing and, for example, to calculate their trajectory.

[0003] The document US 2019 / 0355150 A1 discloses a method for detecting a vehicle in an image, wherein first a heatmap is generated from the image for each pixel, wherein the heatmap is a probability for the center of a vehicle, and wherein subsequently a 3D bounding box for the vehicle is determined by a neural network on the basis of the heatmap. Disclosure of the invention

[0004] The invention relates to an image processing method for determining vehicle dimensions, a computer program, a storage medium, a dimension determination device, and a traffic monitoring system. Further features and advantages of the invention will become apparent from the dependent claims, the following description, and the accompanying figures.

[0005] The invention relates to an image processing method for determining vehicle dimensions of a vehicle in an image of a surveillance area. A surveillance scene with at least one vehicle can be arranged in the surveillance area. Multiple vehicles can also be arranged in the surveillance scene. The vehicles are, in particular, road vehicles, specifically passenger cars, trucks, buses, etc. The surveillance scene is designed, in particular, as a driving area of ​​vehicles. The surveillance scene can be realized, in particular, as a road, a highway, an intersection, a parking lot, or the like.

[0006] In particular, the vehicle dimensions of several or all vehicles in the surveillance area can also be determined.

[0007] Vehicle dimensions are understood to mean, in particular, the length, width, and / or height of the respective vehicle. In principle, the vehicle dimensions can be output as a relative size, such as pixels or a relative size to objects in the image, etc. However, it is preferred to output the vehicle dimensions in absolute sizes, particularly in length units, such as meters.

[0008] The image can be generated, for example, by a camera, particularly a traffic camera. The image can be a grayscale image or a color image.

[0009] In the method, a first processing step is performed starting from the at least one image. Secondary steps can precede and / or follow the first processing step. In particular, the image can be preprocessed, for example, using digital image processing, to simplify the processing steps.

[0010] In the first method step, a position range of the vehicle in the image is determined. In particular, a flat position range is determined in the image. For example, a window is determined as the position range, with the vehicle being arranged in the window. The position range is in particular designed as a 2D bounding box. It can be provided that the position range closely encloses the vehicle. Alternatively, a position range is selected which is smaller than the image, but larger, in particular several times larger, e.g. at least 2, 3, 4 or 5 times larger than the vehicle in the image. The vehicle is completely recorded in the position range. Image points and / or points in image coordinates can be used, for example, to determine the position range.

[0011] Image position information is provided from the vehicle's position area. The image position information contains information about the position area. However, this information can be implemented in any way, as described below. For example, the position area can be represented in image coordinates or as image sections of the image in the image position information.

[0012] Based on the image position information, coordinates are derived in a second processing step, whereby the coordinates define the projected vertices of a polyhedron in the image. The coordinates are in particular designed as image coordinates and / or as 2D coordinates and represent points in the image on or in the image plane. In particular, there are eight vertices that define six side surfaces (front, rear, top, bottom, left, right in the direction of travel). Depending on the geometry of the vehicle, right angles do not necessarily have to be present. However, the polyhedron preferably encloses the vehicle in such a way that no parts of the vehicle protrude beyond the polyhedron, nor is it too large and creates a "gap" between the polyhedron and the vehicle. In particular, the polyhedron has six side surfaces, specifically as a cuboid and particularly preferably as a rectangular cuboid. The polyhedron represents the vehicle.

[0013] Based on the polyhedron, a polyhedron position can be defined, for example. The polyhedron position can be, for example, a center of gravity or a center of gravity projected onto the base surface or center point of the polyhedron. Preferably, the polyhedron position is configured as a vertex, center point, or base point of the polyhedron. It is envisaged that at least the second processing step is implemented via a neural network.

[0014] Based on the coordinates of the polyhedron in the image, the vehicle's dimensions are determined. For example, using camera calibration, these coordinates are converted into 3D world coordinates, from which the vehicle's dimensions (length, width, height) can be calculated.

[0015] One consideration of the invention is that by dividing the image processing method for determining vehicle dimensions into two basic processing steps, namely the determination of the position range and the subsequent determination of the coordinates of the polyhedron as a prerequisite for determining the vehicle dimensions, the image processing method is particularly efficient and effective. Such determination of vehicle dimensions can be advantageously used in traffic monitoring systems, which allow the driver to be warned, stopped, or subsequently penalized for a violation of vehicle dimension specifications. By implementing at least the second processing step using a neural network, it is achieved that the vehicle dimensions do not have to be deduced from the detected vehicle using complex and time-consuming environmental models.Using a neural network is more robust.

[0016] It is particularly preferred that the polyhedron completely encloses the vehicle and / or that the vehicle is completely arranged within the polyhedron, including any attachments. As a result, the polyhedron defines a collision contour for the vehicle, so that if the polyhedron, as a collision contour, overlaps with another object, a collision with the other object must be assumed. For example, if two polyhedra are arranged so as to overlap, it can be assumed that the vehicles represented by the polyhedra also "overlap" and thus have a collision.

[0017] The determined coordinates can include eight vertices of the cuboid, thus describing it in a defined manner. Alternatively, only four vertices of the cuboid are determined, whereby the four vertices of the cuboid do not lie in a common plane, and the eight vertices of the cuboid are determined by linear combination of the four vertices.

[0018] In a preferred implementation of the invention, the coordinates are embodied as p3D coordinates. This particularly involves the projection of the eight (3D) corner points of the polyhedron enclosing the vehicle into the image. The p3D coordinates, as a frame projection, are the projection into the image of a polyhedron, in particular a cuboid, embodied as a 3D frame (imagined in the real world), preferably directly surrounding the object. The 3D frame and / or the polyhedron, in particular the cuboid, is formed in particular by straight line segments. “Directly surrounding” is to be understood as meaning that a surface defined by the imaginary 3D frame or cuboid (e.g., a polyhedron defined by the frame) surrounds the vehicle as closely / as possible, for example, with the smallest possible volume. The respective surfaces and edges of the surface spanned by the 3D frame or the cuboid touch the surface of the object.In other words, a so-called "3D bounding box" is determined as a 3D frame or cuboid. The 3D frame is, for example, an enclosing body in the shape of a rectangular cuboid.

[0019] In one possible implementation, the image information is configured as image sections of the image with the vehicle. Thus, partial sections of the image are delivered to the second processing step. In a preferred development of the invention, the neural network is configured as a convolutional neural network, in particular as a CNN (convolutional neural network), specifically as a DCNN (deep convolutional neural network). This architecture has the advantage that the convolutional neural network is particularly good at performing local evaluations and can thus process the individual image sections effectively and efficiently.

[0020] In an alternative implementation, the image information includes vehicle area data in the image as well as the image itself. In a preferred development, the neural network is designed as a transformer network, wherein the transformer network can use the localization of the vehicles in the image via the vehicle area data from the first processing step as a priori knowledge and can thus implement the second processing step effectively and efficiently.

[0021] In principle, the first processing step can be implemented using digital image processing, in particular pattern recognition. In a preferred embodiment of the invention, the first processing step is implemented by an upstream neural network, wherein the upstream neural network is configured as another convolutional neural network, in particular a CNN (convolutional neural network), specifically a DCNN (deep convolutional neural network), and / or another transformer network.

[0022] A further subject matter of the invention is a computer program, wherein the computer program is designed and / or configured to carry out, apply and / or implement the method according to the invention and / or the steps of the method when executed.

[0023] A further subject of the invention is a machine-readable storage medium, wherein the computer program is stored on the storage medium.

[0024] A further subject matter of the invention is a dimension determination device with a data processing device, wherein the data processing device is designed to carry out the previously described method and / or implement the computer program. The data processing device is designed, for example, as a computer or as a cloud. The dimension determination device comprises an input interface for receiving the image, wherein the input interface is connected to the data processing device for data transmission and transfers the image to the data processing device. Furthermore, the detection device comprises an output interface for outputting the vehicle dimensions, wherein the output interface is connected to the data processing device for data transmission and outputting the aforementioned data.

[0025] Optionally, the dimension determination device comprises a camera for capturing the image, wherein the camera is connected to the input interface for data transmission. Alternatively or additionally, the dimension determination device comprises an output device, such as a screen, a display, or the like, wherein the output device is connected to the output interface for data transmission and is configured to output the vehicle dimensions. For example, the image can be displayed on the output device with the vehicle dimensions drawn in.

[0026] Preferably, the dimension determination device is designed as a real-time system, wherein the runtime between the acquisition and / or recording of the image and the output of said data is less than 1 s, preferably less than 0.1 s.

[0027] A further subject of the invention is a traffic monitoring system, wherein the traffic monitoring system comprises the dimension determination device and the camera. Furthermore, the traffic monitoring system has a traffic control device, wherein the traffic control device is configured to carry out a traffic warning and / or traffic control based on the vehicle dimensions, e.g., in comparison to predeterminable vehicle dimensions. During traffic control, for example, lanes are closed or traffic lights are activated if vehicle dimensions are exceeded.

[0028] For example, the traffic control system comprises an environmental model of the surveillance scene, whereby areas are entered in the surveillance scene for which the vehicle dimensions are restricted for geometric reasons, such as driving under a bridge with height restrictions, entering a tunnel with height and width restrictions, parking garage entrances, road narrowings at construction sites and similar areas with restrictions on the permissible vehicle dimensions. Alternatively, areas are entered in the surveillance scene for which the vehicle dimensions are restricted for traffic law reasons, such as entering residential areas, safety restrictions, etc. The traffic control system is designed to compare the environmental model with the vehicle dimensions and to initiate countermeasures such as closing lanes, switching traffic lights, lowering barriers, etc. if vehicle dimensions are exceeded.

[0029] The subject of the invention is thus the estimation of vehicle dimensions using the coordinates of a polyhedron, particularly a cuboid, estimated by a neural network. Some particular advantages are: The direct estimation of the vertices of the enveloping polyhedron (as keypoints) is used, as opposed to estimating specific keypoints on vehicles (e.g., tires, license plates, or lights). This eliminates the need for adaptation to specific vehicle types (beyond a truck / car / van class), and easily allows for extrapolation to other vehicle types.

[0030] In contrast to approaches that require fixed vehicle models, the method also works for vehicles that have deviations from the standardized dimensions due to additional load objects (e.g. roof racks).

[0031] Apart from the neural network(s) and camera calibration, no additional model assumptions are required, so the approach also works with multiple ground planes, for example. An extrinsic camera calibration can either be estimated based on the coordinates or determined using another approach.

[0032] Temporal information is not necessarily required, but can be used to filter the detections temporally.

[0033] Further features, advantages, and effects will become apparent from the following description of a preferred embodiment and the accompanying figures. These show: Fig. 1 is a schematic block diagram of a traffic monitoring system; Fig. 2 a flowchart for a method for operating the traffic monitoring system; Fig. 3 a schematic representation of a surveillance scene with a vehicle, wherein the vehicle is represented by a polyhedron or cuboid.

[0034] The Fig. 1 shows a schematic block diagram of a traffic monitoring system 1 for monitoring traffic on roads, parking lots, highways, etc. as an embodiment of the invention.

[0035] The traffic monitoring system 1 comprises a dimension determination device 2 for determining vehicle dimensions of vehicles 7 in a monitoring area 8, wherein a monitoring scene 9 with at least one vehicle 7 can be arranged in the monitoring area. The dimension determination device 2 is designed as a digital data processing device, such as a computer, a server, a client, a cloud, etc. The dimension determination device 2 is designed to implement the method according to the Fig. 2. In particular, a computer program is executed on the dimension determining device 2 which implements the method.

[0036] The dimension determination device 2 has an input interface 3, wherein the input interface is connected for data purposes to a camera 4, wherein the camera 4 forms a component of the traffic monitoring system 1. Multiple cameras 4 may also be present. The camera 4 is directed toward the surveillance area 8 with the surveillance scene 9 and can record at least one image 10 or series of images showing the surveillance area 8, in particular the surveillance scene 9.

[0037] Furthermore, the traffic monitoring system 1, in particular the dimension determination device 2, has an output interface 5, wherein the output interface 5 is connected for data purposes to a traffic control device 6. The traffic control device 6 receives data from the dimension determination device 2 and implements measures to direct traffic. Depending on the detected vehicle dimensions, for example, speed limits can be implemented or changed, lanes can be closed or opened, traffic lights can be activated, or warnings can be issued.

[0038] In particular, the traffic control device 6 becomes active when the vehicle dimensions of the vehicle 7 do not correspond to predeterminable vehicle dimensions. The predeterminable vehicle dimensions can be determined, for example, by the dimensions of tunnel entrances, bridges, or other stationary objects.

[0039] The Fig. 2 shows a schematic flow diagram of an image processing method for determining vehicle dimensions of the vehicle 7 in the monitoring area 8.

[0040] In a step 100, the image 10 of the surveillance area 8 with the surveillance scene 9 is captured.

[0041] In a first processing step 200, one or more positioning areas of vehicles 7 are determined in the image 10. The positioning areas are defined, for example, as 2D bounding boxes. The positioning areas can be defined, for example, by image coordinates.

[0042] The first processing step 200 can be implemented using digital image processing. However, it is also possible for the first processing step 200 to be implemented using a neural network, in particular a CNN or a transformer network. In the first processing step 200, image position information relating to the at least one position range is provided.

[0043] In a second processing step 300, based on the image position information, coordinates 11 of a polyhedron 12, in particular a cuboid, are determined, wherein the polyhedron 12 forms an enveloping body for the respective vehicle 7. Thus, the image position information is used as a priori knowledge to simplify the second processing step 300. The polyhedron 12 is defined in the form of projected vertices in the image, which form the coordinates 11 of the polyhedron 12. The projected vertices of the polyhedron 12 can be configured as p3D coordinates. The coordinates are determined and / or predicted in the second processing step 300 by a DNN (deep neural network). The DNN can be a CNN (convolutional neural network) or a transformer network. The Deep Neural Network (DNN) predicts N keypoints (coordinates) for each of the M objects (vehicles 7) in the image.The input data for the DNN can be either individual images 10 with the image position information or an image series consisting of several related images that are processed simultaneously as a batch. For example, all eight keypoints (coordinates of the vertices of the enveloping cuboid as a polyhedron 12 of the object, projected onto the camera plane) can be used as the prediction result of the DNN. In this case, the DNN would have the full 16 degrees of freedom (each of the 8 projected vertices has an x ​​and y coordinate). Alternatively, a subset of the 8 keypoints can be predicted, e.g., only 4 keypoints, from which the remaining four keypoints result from the linear combination of the predicted four points. In the latter case, the system has eight degrees of freedom.

[0044] In an extension of the system, the method can be combined with a tracker to temporally filter the estimated bounding boxes and / or polyhedra. This filtering / tracking can be performed either on the p3D coordinates or, alternatively, on the 3D real-world coordinates.

[0045] In a step 400, the vehicle dimensions are determined based on the coordinates 11 of the polyhedron 12, in particular based on the projected 3D coordinates of the polyhedron 12 onto the image coordinates. A camera model can be used for this purpose, or other known image processing options can be implemented. In this step, the estimated coordinates 11 of the polyhedron 12, in particular the p3D coordinates, are converted into 3D real-world coordinates using the existing camera calibration parameters, from which the dimensions of the vehicle 7 are then derived. The resulting dimensions are shown in Fig. 3 shown.

[0046] In the Fig. Figure 3 schematically shows an image showing a surveillance area 8 with a surveillance scene 9 in the form of a road with at least one vehicle 7. The vehicle 7 is enclosed in the polyhedron 12, wherein it can be seen that the polyhedron 12 completely encloses the vehicle 7, so that the polyhedron 12 forms an interfering contour representing the vehicle 7. On the other hand, the polyhedron 12 is placed as tightly as possible around the vehicle 7 so that an excessively large interfering contour is not defined. It should be emphasized that the polyhedron 12 is only oriented towards the external dimensions of the vehicle 7, but does not require any other keypoints. In Figure 10, the vehicle dimensions are shown as follows: width 2.7 m, length 11.9 m, height 4.2 m. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2019 / 0355150 A1

[0003]

Claims

[1] Image processing method for determining vehicle dimensions of a vehicle (7) in a surveillance area (8), wherein a surveillance scene (9) with at least one vehicle (7) can be arranged in the surveillance area (8), wherein in a first processing step (200), starting from at least one image (10) of the surveillance area (10), a position area of the vehicle (7) is determined in the image and image position information is provided for the position area, wherein in a second processing step (300) coordinates (11) are derived on the basis of the image position information, wherein the coordinates (11) define the projected vertices of a polyhedron (12) in the image (10), wherein the polyhedron (12) represents the vehicle (7), wherein the second processing step (300) is implemented via a neural network, whereby the vehicle dimensions are determined on the basis of the coordinates (11). [2] Method according to claim 1, characterized by that the polyhedron (9) encloses the vehicle (8). [3] Method according to claim 1 or 2, characterized by that the coordinates (11) comprise eight vertices of the cuboid (12) and / or that the coordinates (11) have four vertices of the polyhedron (12) as a cuboid, wherein the eight vertices of the cuboid are deduced by linear combination of the four vertices. [4] Method according to one of the preceding claims, characterized by that p3D coordinates are determined as coordinates (11), wherein the p3D coordinates define the projected vertices of the polyhedron (12) in the image (10). [5] Method according to one of the preceding claims, characterized by that the image information is formed as image sections of the image (10) with the vehicle (7). [6] Method according to claim 5, characterized by that the neural network is designed as a convolved neural network. [7] Method according to one of the claims, characterized by that the image information includes vehicle area data in the image (10) and the image (10). [8] Method according to claim 7, characterized by that the neural network is designed as a transformer network. [9] Method according to one of the preceding claims, characterized by that the first processing step (200) is implemented by an upstream neural network, wherein the upstream neural network is designed as a further convolved neural network and / or as a further transformer network. [10] Computer program, wherein the computer program is designed and / or configured to execute, apply and / or implement the method according to one of claims 1 to 9 when executed. [11] A machine-readable storage medium, wherein the computer program according to claim 10 is stored on the storage medium. [12] Dimension determination device (2), characterized by that the dimension determination device (2) is designed to carry out the method according to one of the preceding claims 1 to 9, wherein the dimension determination device has an input interface (3) for accepting at least one image (10) and / or an output interface (5) for outputting vehicle dimensions. [13] Traffic monitoring system (1) for monitoring traffic, with the dimension determination device (2) according to claim 12, with a camera (4) for recording at least one image (10), wherein the camera (4) is connected to the input interface (3) for data technology, and with a traffic control device (6) for outputting a traffic warning and / or traffic control information based on the vehicle dimensions, wherein the traffic control device (6) is connected to the output interface (5) for data technology.

Images