Method performed by a sensor system of a traffic infrastructure and sensor system

Abstract

This invention relates to a method performed by a sensor system for traffic infrastructure, in which a transformation rule for coordinate transformation between radar data acquired by radar equipment and video data acquired by a camera is determined based on the correlation between road users detected by a camera and road users detected by radar equipment. The invention also relates to a corresponding sensor system.

Description

Technical Field

[0001] The present invention relates to a method performed by a sensor system for transportation infrastructure according to the preamble of claim 1, and a corresponding sensor system. Background Technology

[0002] In the field of intelligent infrastructure systems for road traffic, the use of high-performance camera and radar systems is becoming increasingly common. These systems enable the automatic detection and location of vehicles and other road users over vast detection areas, facilitating a wide range of applications such as intelligent control of lighting and signaling devices and long-term optimization analysis of traffic flow. Currently, assistance functions for driver assistance systems and autonomous driving are under development, particularly in the use of vehicle-to-X wireless communication, or especially in the context of infrastructure-to-X communication.

[0003] If cameras and radar are used in parallel, it makes sense to combine the data sets of these two subsystems (i.e., fuse them), and depending on the application, it may even be absolutely necessary. To correlate object data acquired in this way, it is usually necessary to know the transformation rules between the individual sensors (“cross-calibration”) or between these sensors and another commonly known coordinate system, especially in order to be able to correlate data of objects (e.g., road users) detected in parallel by cameras and radar.

[0004] Here, these sensors are typically calibrated using a reference object placed at the measurement location within the sensor's field of view, which can be manually or automatically identified in the sensor data. For the static positioning of the reference object, in some cases, it may even be necessary to intervene in the current traffic flow, such as temporarily closing a lane or the entire road.

[0005] As an alternative, relatively easily identifiable static objects (such as the base of a traffic sign) in the overlapping detection areas of the camera and radar can be manually labeled and correlated with each other. However, this requires a sufficient number of these objects to be present in the overlapping field of view of the sensors and to be clearly identifiable in the data from both sensor types. Road surfaces, in particular, typically do not provide any static objects that can be identified in radar data.

[0006] Therefore, the described methods typically require relatively extensive manual configuration support, such as for manually locating reference objects or for marking locations in sensor data.

[0007] Sometimes a high-quality, and therefore high-cost, system is needed, for example, to determine the position of a reference object in a global coordinate system via differential GPS.

[0008] Therefore, a solution is needed to overcome the aforementioned drawbacks.

[0009] According to one embodiment of the method performed by a sensor system for traffic infrastructure, road users are detected by at least one camera of the sensor system, the at least one camera having a first detection area of ​​the traffic infrastructure, and road users are detected by at least one radar device of the sensor system, the at least one radar device having a second detection area of ​​the traffic infrastructure, wherein the first detection area and the second detection area at least partially overlap and detect at least one road of the traffic infrastructure having multiple lanes. Here, a transformation rule for coordinate transformation of the data acquired by the radar device and the data acquired by the camera is determined based on the association between road users detected by the camera and road users detected by the radar device. The association between road users detected by the camera and road users detected by the radar device, and the determination of the transformation rule, are performed automatically here.

[0010] For example, radar detection occurs in the coordinates of the radar coordinate system, such as xy coordinates, while camera detection advantageously occurs in the camera's pixel grid or pixel coordinate system. According to at least one embodiment, transformation rules define coordinate transformations between the radar coordinate system for acquiring radar data and the camera coordinate system for acquiring video data, and / or coordinate transformations from the radar and camera coordinate systems to a third coordinate system. In the case of transformations to a third coordinate system, transformation rules from the respective coordinate systems to the third coordinate system are specified. With the defined transformation rules, objects or road users detected in one coordinate system can thus be associated with potentially identical objects or road users detected in another coordinate system. Detected road users can, in principle, be displayed in a third coordinate system different from the radar and camera coordinate systems for further processing.

[0011] According to at least one embodiment, location information of road users detected by a camera is accumulated over time, and location information of road users detected by a radar device is also accumulated over time. In this case, detection is particularly based on video data provided by the camera or radar data provided by the radar device. The result of accumulating location information over time specifically represents, for both radar data and video data, the composite movement direction of the road user within the relevant coordinate system during the observation period. In other words, lane direction is identified by the cumulative detection of road user positions or the detection of their movement directions, wherein the acquired location information primarily relates to the corresponding coordinate system, i.e., the camera coordinate system and / or the radar coordinate system.

[0012] According to at least one implementation, individual lanes of a road are identified based on location information accumulated over time by a camera, and in parallel, individual lanes are identified based on location information accumulated over time by a radar device. Thus, individual lanes are identified, particularly independently of each other, in camera images or based on video data and radar data. The corresponding accumulation of location information means the formation of a clear detection accumulation, especially along the centerline of the road lanes. These maximum values ​​can be used accordingly to identify the corresponding lanes.

[0013] According to at least one embodiment, the determined maximum value of position information accumulated over time by a camera and / or the determined maximum value of position information accumulated over time by a radar device is approximately represented by a polyline, especially a spline. As previously mentioned, significant detection accumulation typically forms along the centerline of the lane. According to this embodiment, the determined maximum value of these accumulations is approximately represented by a polyline, which thus mathematically represents the lane direction in the corresponding sensor coordinate system.

[0014] According to the improved scheme, location information of road users detected by cameras and / or by radar devices is accumulated over time within a preset and / or adjustable time period. In this sense, "adjustable" specifically means a preset time interval that can be manually changed, and / or an automatic adjustment of the time period based on specified conditions, or an automatic adjustment of the time period until specified conditions are met. Conversion rules or calibrations can be determined, in particular, during current road traffic. No additional reference objects are required, and high-precision measured reference locations are not necessary. It is also possible to specify that the conversion rules are automatically determined after a defined time period (e.g., within a range of minutes to hours) and then used for detection.

[0015] On the other hand, calibration can also be performed permanently or repeatedly during the continuous operation of the transportation infrastructure. This allows for compensation for any changes that occur over time or for continuous optimization, and in particular, it can be specified that such recalibration be compared with the results of the original calibration or the previous calibration. This allows for the automatic identification of sensor misalignment.

[0016] According to at least one embodiment, location information accumulated over time by a camera and location information accumulated over time by a radar device are used to determine the parking position of a road user in each lane. In particular, the parking position ahead of the road user is detected. This is the case, for example, when the road user is parked at the stop line at an intersection.

[0017] According to at least one embodiment, the forward parking positions of these lanes are determined, wherein a maximum value of the accumulated position information of the relevant lane over time is determined. Here, the maximum value of the detection result comes particularly from a parking time at a certain location that is longer in time than the detection of a moving road user, and therefore more frequently detected at that location within the relevant time period. According to an improvement, if the speed of the object can be determined, then for this purpose, a substantially stationary object in the relevant lane is directly determined. Alternatively or supplementarily, the local maximum value closest to the camera and / or closest to the radar device can be specified as the parking position of each lane. However, a prerequisite for doing so is the corresponding arrangement of the sensors and the corresponding detection areas in the direction of the detected road with the nearest parking line. This process can also be used as a standard, or to support finding the corresponding maximum value in combination with at least one of the processes described above (e.g., as the starting point of a corresponding search).

[0018] According to at least one implementation, a correlation is established between a parking location determined by a camera and a parking location determined by a radar device.

[0019] According to the improved scheme, the time occupancy of parking locations identified based on video data is specifically combined with that identified based on radar data. This results in a number of possible associations corresponding to the product of the number of parking locations identified in the video data and the number of parking locations identified in the radar data.

[0020] For such combinations, for example, binary occupancy states—whether a vehicle is in a parking position, i.e., yes or no—can be combined over certain time intervals, such as a few minutes, using an XOR NOT operation. The XOR NOT operation produces 1 in the case of the same state and 0 in the case of different states. Parking positions that do not reach a preset minimum number of occupancy state changes (0→1, 1→0) during a preset detection time will be particularly ignored, or the detection time will be extended accordingly to ensure a sufficient basis for statistical evaluation. Possible combinations can be categorized, in particular, according to the number of time shares or corresponding initial values, and at least one association table containing the most likely associations of parking positions from radar data and video data can be created, for example.

[0021] One approach that can be used complementaryly or alternatively, and is particularly suitable for sensors capable of providing non-binary or continuous data (such as the probability of parking space occupancy), is to consider cross-covariance. This can be determined as the lateral correlation between different sensor outputs in order to establish a correlation between parking locations based on video and radar data.

[0022] According to at least one implementation, an association is established between lanes identified by cameras and lanes identified by radar devices based on the associated parking locations of road users.

[0023] According to the improved plan, in this case, associated lanes are considered to establish a connection between road users detected by cameras and road users detected by radar equipment.

[0024] According to at least one embodiment, in order to identify lane direction by accumulating the detection of road user locations, road users detected by cameras and radar equipment are selected based on road users who are moving or have previously moved and / or road users who have been classified as vehicles. For this purpose, using the classification of road users detected in camera data and / or radar data, or receiving object data of the corresponding classification via a processing computing device, may prove advantageous.

[0025] According to at least one implementation, a correlation is established between parking locations determined by a camera and parking locations determined by a radar device by comparing the detected time points when a road user is in a parking location and / or moves to a parking location and / or leaves a parking location. A clearer situation exists when, for example, only one road user is detected on the road by the radar device and camera. If the road user moves to the identified parking location at a specific time point, that parking location can already be substantially considered to be the same as the one detected by the radar device and camera. Since road users detected by the radar device and camera are not necessarily correlated in principle, and the situation may rarely prove so clear, a statistical evaluation of the time points is specifically prescribed. It can be assumed here that, in reality, the probability of a road user moving to a parking location at the same time point within the considered time period is relatively small. Within the considered time period, since the time difference regarding multiple road users driving through parking locations is also relatively small, a statistically possible correlation arises between parking locations detected by the radar device and camera. This also applies to the cases of staying in a parking location and leaving a parking location, or considering both events simultaneously.

[0026] According to at least one implementation, a correlation is established between lanes identified by cameras and lanes identified by radar devices based on associated parking locations. This is therefore possible because parking locations form the maximum value of a lane and are thus directly associated with it, and the association of parking locations in different sensor coordinate systems allows for lane association.

[0027] According to at least one embodiment, an association is established between road users detected by a camera and road users detected by a radar device, taking into account associated lanes, in such a way that a road user detected by the radar device who is in a specific lane at the time closest to the parking position corresponds to a road user detected by the camera who is in the lane associated with that lane and is closest to the parking position associated with that parking position.

[0028] According to the improved plan, road users who are positioned as second, third, or other proximity to the parking location can also be associated. However, this could potentially lead to a higher error rate because the (partial) occlusion of road users located closer to the associated parking location may result in less accurate detection of the associated road users.

[0029] According to at least one implementation, classification information provided by radar equipment and / or cameras is used to detect and / or associate and / or verify the association of road users.

[0030] According to at least one embodiment, at least one pair of associated points are stored in radar coordinates and camera coordinates at at least one point in time to determine a transformation rule for at least one associated road user. In this case, a point represents a detected element in the coordinate system, such as a camera pixel and a measurement point in the case of the radar device. Thus, two sets of points are generated within the considered time period, each having a one-to-one association (corresponding) point in the other set. According to an improved embodiment, a homography matrix between the radar detection plane and the camera image plane is determined from the set of points generated in this way. Homography is a projection transformation between the camera image coordinates and the radar device detection plane, or a projection transformation between the image coordinates and the ground plane in front of the radar device. The second embodiment is particularly advantageous when the installation location (e.g., the height and tilt angle of the radar device) is known.

[0031] According to at least one implementation, at least one optimization method (e.g., RANSAC) is used to avoid detection errors and association errors of the detected points. Since the number of point pairs generated is typically significantly greater than the number of point pairs required for homography calculation (e.g., four corresponding point pairs), there is no deterioration in the accuracy of the calculated transformation rules or homography matrix.

[0032] If the possible distortion caused by the camera optics can be assessed as negligible for a particular application and / or can be corrected in advance by internal calibration and / or can be determined directly from the generated point pairs, for example by the Bouguet method, then such distortion can be considered negligible.

[0033] According to at least one embodiment, the external calibration of the camera is determined relative to radar, where corresponding point pairs are considered as a perspective n-point (PnP) problem. This problem can be solved, for example, by RANSAC, or by the Bouguet method. The external calibration of the camera specifically describes the precise position and orientation of the camera in space. Inherent camera parameters can be advantageously used for this purpose.

[0034] According to the improved plan, it can be stipulated that other information sources, especially information received through vehicle-to-X communication, can be used to determine the conversion rules.

[0035] The present invention also relates to a sensor system for traffic infrastructure, the sensor system comprising at least one camera having a first detection area of ​​the traffic infrastructure and at least one radar device having a second detection area of ​​the traffic infrastructure, wherein the first detection area and the second detection area at least partially overlap and detect at least one road of the traffic infrastructure having multiple lanes, wherein the sensor system is configured to perform a method according to at least one of the described embodiments or improvements of the described method.

[0036] According to at least one embodiment, the described sensor system includes one or more computing devices for performing the method.

[0037] The proposed method and sensor system overcome the shortcomings of existing solutions. Specifically, the transformation rules between the camera coordinate system of traffic infrastructure cameras and the radar coordinate system of radar equipment can be automatically determined, thereby enabling, for example, the association of individual pixels in a video image with their corresponding objects in radar data, and vice versa. As a result, video and radar data can advantageously be obtained simultaneously within the system. This significantly reduces or completely eliminates the need for manual assistance, as, for example, manual labeling (so-called annotation) of the data is no longer required.

[0038] Sensor systems, particularly fixed sensor systems for traffic infrastructure, are understood in particular as those configured to be stationary for the specific use purposes of the corresponding traffic infrastructure. This differs from sensor systems intended for mobile use (e.g., in or by vehicle).

[0039] Transportation infrastructure is understood, for example, to mean land, water, or air transportation routes, such as roads, railway tracks, waterways, air routes, or intersections of said transportation routes, or any other transportation infrastructure suitable for transporting people or payloads. The use of sensor systems at road intersections, especially those with multiple entry lanes and the stopping positions ahead of the entry lanes within the sensor's detection area, has proven particularly advantageous.

[0040] Road users can be vehicles, cyclists, or pedestrians. Vehicles can be motor vehicles, especially passenger vehicles, trucks, motorcycles, electric or hybrid vehicles, boats, or aircraft.

[0041] In the improved version of the illustrated sensor system, the system has a memory. In this case, the illustrated method is stored in the memory as a computer program, and the computing device is configured to execute the method when the computer program is loaded from the memory into the computing device.

[0042] According to another aspect of the invention, a computer program includes program code means for performing all the steps of one of the described methods when the computer program is executed by a computing device of a system.

[0043] According to another aspect of the invention, a computer program product includes program code stored on a computer-readable data carrier, and when the program code is executed on a data processing device, the program code performs one of the described methods. Attached Figure Description

[0044] Some particularly advantageous designs of the invention are described in the dependent claims. Further preferred embodiments also emerge from the subsequent description of the embodiments with reference to the accompanying drawings.

[0045] In the diagram:

[0046] Figure 1 One implementation of the method is shown.

[0047] Figure 2 Another implementation of the method is shown.

[0048] Figure 3a This shows a composite image accumulated over time from radar data from the detection area of ​​a radar device, where the radar device (not shown) is positioned on the left side of the image using a right-facing viewing direction.

[0049] Figure 3b This shows a composite image of a traffic intersection formed from radar data from multiple radar devices calibrated to each other in the same coordinate system.

[0050] Figure 4 This shows the correlation between the parking positions ahead of road users.

[0051] Figure 5 It shows the association of lanes for road users.

[0052] Figure 6 It shows the connection between the road users themselves, and

[0053] Figure 7 An exemplary implementation of the sensor system is shown. Detailed Implementation

[0054] To allow for a brief description of the embodiments, elements that are substantially the same in function are provided with the same reference numerals.

[0055] Figure 1 Figure 3 to 3, shown as examples for use at road traffic intersections, illustrate this. Figure 6 Transportation infrastructure for 300, 400, 500 and 600 as referenced Figure 7 This is one embodiment of the method 100 performed by the sensor system 700 described in the embodiments. In step 102a, by according to Figure 7 The sensor system 700's radar device 770 detects road users 620, 640, and 660. This radar device has the features shown in Figure 3. Figure 4 , Figure 5 and Figure 6 The first detection areas of traffic infrastructure 300, 400, 500, and 600, and separately therefrom, in step 102b, through Figure 7 The sensor system 700 uses camera 760 to detect road users 620, 640, and 660. This camera has the features shown in Figure 3. Figure 4 , Figure 5 and Figure 6 The second detection area of ​​traffic infrastructures 300, 400, 500, and 600, wherein the first and second detection areas at least partially overlap and detect at least one road with multiple lanes of traffic infrastructures 300, 400, 500, and 600. Here, the coordinate transformation rules for the radar data acquired by radar device 770 and the video data acquired by camera 760 are determined based on the association between road users 620, 640, and 660 detected by camera 760 and road users 620, 640, and 660 detected by radar device 770.

[0056] According to this embodiment, radar detection occurs in the xy coordinates of the radar coordinate system, such as... Figure 3a )and Figure 3b As shown in the example, camera detection occurs at the camera's pixel coordinates (video image), as shown from... Figure 4 , Figure 5 and Figure 6As can be seen in the schematic diagram. According to at least one embodiment, the coordinate transformation between the radar coordinate system and the camera coordinate system for acquiring video data is defined by the transformation rules determined by methods 100 and 200. Alternatively or supplementarily, a coordinate transformation occurs from the radar coordinate system and the camera coordinate system to a third coordinate system in which the data is aggregated. For this purpose, corresponding transformation rules from the respective coordinate systems to the third coordinate system are specifically specified. With the aid of the determined transformation rules, objects or road users detected in one coordinate system can be associated with potentially identical objects or road users detected in another coordinate system.

[0057] In step 104, road users detected by the camera are correlated with road users detected by the radar device. This is understood to refer to the correlation of the same road users in the video data and radar data, regardless of which process is chosen for this purpose or what the starting point of the correlation is.

[0058] In step 106, at least one point pair in radar coordinates and camera coordinates is detected at at least one time point to determine a transformation rule for at least one associated road user. This results in two sets of points generated within the considered time period, each having a one-to-one association (corresponding) point in the other set. Based on this example, a homography matrix is ​​thus determined as the transformation rule between the radar detection plane and the camera image plane. Furthermore, optimization methods (e.g., RANSAC) can be used to avoid detection errors and association errors of the detected points. Since the number of point pairs generated is typically significantly greater than the number of point pairs required for homography calculation, this usually does not lead to any degradation in the accuracy of the calculated transformation rule or homography matrix.

[0059] Figure 2 Another embodiment of the method is shown. In step 202a, road users 620, 640, and 660 are detected cumulatively over time by radar device 770 having a first detection area of ​​traffic infrastructure 300, 400, 500, and 600, and separately, in step 202b, road users 620, 640, and 660 are detected cumulatively over time by camera 760 having a second detection area of ​​traffic infrastructure 300, 400, 500, and 600, wherein the first and second detection areas at least partially overlap and detect at least one road 320, 420, 520, or 620 with multiple lanes of traffic infrastructure 300, 400, 500, and 600.

[0060] In this context, detection is particularly based on video data 762 provided by camera 760 or radar data 772 provided by radar device 770. The results of acquiring location information cumulatively over time specifically represent the composite movement direction of the road user within the relevant coordinate system during the observation period, for both radar data 772 and video data 762. In other words, lane direction is identified through the cumulative detection of road user positions or the detection of their movement directions, wherein the acquired location information primarily relates to the corresponding coordinate system, i.e., the camera coordinate system and / or the radar coordinate system. For radar detection, Figure 3a )and Figure 3b An example of cumulative detection is shown here. Figure 3a The results of detecting intersection side streets using a single radar device 770 are shown, and Figure 3b The image shows the results of fusion detection of the entire intersection by multiple radar devices.

[0061] According to the improved scheme, location information of road users detected by camera 760 and / or detected by radar device 770 is accumulated over time within a preset and / or adjustable time period. In this sense, "adjustable" is particularly understood to mean a preset time interval that can be manually changed, and / or the time period is automatically adjusted according to specified conditions (e.g., the quality level of detection) or automatically adjusted until the specified conditions are met.

[0062] In step 204a, each lane of the road is identified based on the location information accumulated over time by camera 760, and in parallel, in step 204b, each lane of the road is identified based on the location information accumulated over time by radar device 770. Thus, each lane is identified, particularly independently, from the image of camera 760 or based on video data 762 and radar data 772 based on radar device 770. The corresponding accumulation of location information means that a clear detection accumulation is formed, especially along the centerline of the road lanes. These maximum values ​​can be used accordingly to identify the corresponding lanes.

[0063] According to at least one embodiment, in order to identify lane direction by accumulating the detection of road user locations, road users detected by cameras and radar equipment are selected based on road users who are moving or have previously moved and / or road users who have been classified as vehicles. For this purpose, using the classification of road users detected in camera data and / or radar data, or receiving object data of the corresponding classification via a processing computing device, may prove advantageous.

[0064] According to at least one embodiment, the determined maximum values ​​of position information accumulated over time by a camera and / or by a radar device are approximated by polylines, particularly splines. As previously mentioned, significant detection accumulation typically forms along the centerline of the lane. According to this embodiment, these determined maximum values ​​are approximated by polylines, which thus mathematically represent the lane direction in the corresponding sensor coordinate system. An example of the polyline approximation result in this respect can be found in... Figure 5 As seen in the left part of the image, this image was generated based on video data.

[0065] In step 206a, the location information accumulated over time by camera 760 and the location information accumulated over time by radar device 770 are used to individually determine the parking position of the road user with respect to each identified lane. In particular, the parking position ahead of the road user is detected. This is, for example, when the road user is parked at the stop line at an intersection.

[0066] According to at least one embodiment, in order to determine the parking position ahead of a lane, a maximum value of the accumulated position information of the relevant lane over time is determined. Here, the maximum value of the detection result comes particularly from a parking time at a certain location that is longer in time than the detection of a moving road user, and therefore more frequently detected at that location within the relevant time period. According to an improvement, if the speed of the object can be determined, then for this purpose, a substantially stationary object in the relevant lane is directly determined. Alternatively or supplementarily, the local maximum value closest to camera 760 and / or closest to radar device 770 can be specified as the parking position of each lane. However, a prerequisite for doing so may be the corresponding arrangement of the sensors and corresponding detection areas in the direction of the detected road with the nearest parking line. This process can also be used as a standard, or to support finding the corresponding maximum value in combination with at least one of the processes described above (e.g., as the starting point of a corresponding search).

[0067] In step 208, at parking positions 401a, 402a, and 403a (e.g., determined by camera 760 or video data 762) Figure 4 (as shown) and parking positions 401b, 402b, 403b determined by radar device 770 or radar data 772 (similarly as shown) Figure 4 Establish a relationship between (as shown) as indicated by the corresponding arrows. This is in Figure 4 The illustration uses four cameras (Cam1 to Cam4) and four radar devices to detect an intersection as an example. For clarity, reference numerals are not used in the accompanying drawings.

[0068] Other parking locations are indicated by the sign. According to the improvement plan, this is based on video data 762.

[0069] The time occupancy of the identified parking locations is specifically combined with the parking locations identified based on radar data 772. This produces a number of possible associations corresponding to the product of the number of parking locations identified from video data 762 and the number of parking locations identified from radar data 772.

[0070] According to at least one embodiment, a correlation is established between a parking location determined by video data 762 and a parking location determined by radar data 772 by comparing the detected time points when a road user is in the parking location and / or moves to the parking location and / or leaves the parking location.

[0071] For such combinations, for example, binary occupancy states—whether a vehicle is in a parking position, i.e., yes or no—can be combined over certain time intervals, such as a few minutes, using an XOR NOT operation. The XOR NOT operation produces 1 in the case of the same state and 0 in the case of different states. Parking positions that do not reach a preset minimum number of occupancy state changes (0→1, 1→0) during a preset detection time will be particularly ignored, or the detection time will be extended accordingly to ensure a sufficient basis for statistical evaluation. Possible combinations can be categorized, in particular, according to the number of time shares or corresponding initial values, and at least one association table containing the most likely associations of parking positions from radar data and video data can be created, for example.

[0072] One approach that can be used complementaryly or alternatively, and is particularly suitable for sensors capable of providing non-binary or continuous data (such as the probability of parking space occupancy), is to consider cross-covariance. This can be determined as the lateral correlation between different sensor outputs in order to establish a correlation between parking locations based on video and radar data.

[0073] In step 210, as Figure 5 As shown, based on the associated parking locations of road users, a correlation is established between lanes 501a, 502a, and 503a identified by video data 762 and lanes 501b, 502b, and 503b identified by radar data 772, as indicated by the corresponding arrows. Figure 4In contrast, the detection association is only shown through camera 760 and radar device 770. According to the improved solution, in this case, associated lanes are considered to establish an association between road users detected by camera 760 and road users detected by radar device 770. Therefore, this is feasible because parking positions form the maximum value of the already determined lane and are thus directly associated with it, and the association of parking positions in different sensor coordinate systems thus enables lane association.

[0074] In step 212, associated lanes are considered to establish a connection between road users 620a, 640a, and 660a detected by camera 760 and road users 620b, 640b, and 660b detected by radar device 770. This is done so that a road user detected by radar device 770 who is in a specific lane at the time closest to the parking position corresponds to a road user detected by camera 760 who is in a lane associated with that lane and is closest to the parking position associated with that parking position. According to an improved embodiment, the association of road users who are close to the parking position, such as second and third, can also be specified.

[0075] In step 214, based on the association between road users 620a, 640a, and 660a detected by camera 760 and road users 620b, 640b, and 660b detected by radar device 770, a coordinate transformation rule is determined for the radar data 772 acquired by radar device 770 and the video data 762 acquired by camera 760. For example, a reference transformation rule has already been established. Figure 1 The embodiments are described. Using automatically determined transformation rules, road users detected individually by camera 660 and radar device 670 can then be correlated with each other, and their location information can be transformed, for example, to a matching coordinate system.

[0076] Figure 7 An embodiment of a sensor system for traffic infrastructure is shown, the sensor system including at least one camera 760 having a first detection area of ​​the traffic infrastructure and at least one radar device 770 having a second detection area of ​​the traffic infrastructure, wherein the first and second detection areas at least partially overlap and detect at least one road with multiple lanes of the traffic infrastructure, wherein the sensor system is configured to perform a method according to at least one of the described embodiments or improvements of the described method, such as as shown in reference. Figure 1 and Figure 2 To describe.

[0077] According to at least one embodiment, the described sensor system 700 includes one or more computing devices, such as a controller 720, for performing the method. According to an example, the controller 720 includes a processor 722 and a data memory 724. Furthermore, an embodiment of the sensor system 700 according to the example includes an association device for associating road users detected by the camera 760 with road users detected by the radar device 770. The sensor system also includes a determination device 728 for determining a transformation rule for the coordinate transformation of radar data 772 acquired by the radar device 770 and video data 762 acquired by the camera 760. The controller 720 is capable of outputting processed data to a signal interface 730 for transmission to an evaluation device 800, or receiving data from the evaluation device.

[0078] The proposed method and sensor system can, in particular, automatically determine the transformation rules between the camera coordinate system of the traffic infrastructure camera 760 and the radar coordinate system of the radar device 770. This allows, for example, the association of individual pixels in a video image with their corresponding objects in radar data, and vice versa. Consequently, video data 762 and radar data 772 can advantageously be obtained simultaneously within the system. This improves the automatic detection and location of vehicles and other road users, especially through intelligent control of lighting and signaling devices and long-term optimization analysis of traffic flow via smart infrastructure.

[0079] If, during the execution of the method, it is found that a feature or set of features is not absolutely necessary, the applicant immediately desires that the formulation of at least one independent claim no longer includes that feature or set of features. This could, for example, be a sub-combination of claims existing on the filing date or a sub-combination of claims existing on the filing date that is limited by other features. Such a restated claim or feature combination is understood to also be covered by the disclosure of this application.

[0080] It should also be noted that the designs, features, and variations of the invention described and / or shown in the accompanying drawings in various embodiments or examples can be arbitrarily combined with each other. Individual or multiple features can be arbitrarily interchanged with each other. The resulting combinations of features are to be understood as also covered by the disclosure of this application.

[0081] A reference in a dependent claim is not construed as a waiver of independent substantive protection for the features of the referenced dependent claim. These features may also be combined with other features as desired.

[0082] Features disclosed solely in the specification, or features disclosed in the specification or claims in combination with other features, can in principle have independent significance for the invention. Therefore, they can also be included individually in the claims to distinguish them from the prior art.

[0083] Generally, it should be noted that vehicle-to-X communication is understood in particular as direct communication between vehicles and / or between a vehicle and infrastructure. For example, this communication can therefore be vehicle-to-vehicle communication or vehicle-to-infrastructure communication. If reference is made to vehicle-to-vehicle communication within the scope of this application, then such communication can, in principle, occur as part of vehicle-to-vehicle communication, which is typically implemented without handover via a mobile radio network or similar external infrastructure, and is therefore distinguishable from other solutions, such as those based on mobile radio networks. For example, vehicle-to-X communication can be implemented using the IEEE 802.11p or IEEE 1609.4 standards. Vehicle-to-X communication can also be referred to as C2X communication or V2X communication. These subdomains can be referred to as C2C (vehicle-to-vehicle), V2V (vehicle-to-vehicle), or C2I (vehicle-to-infrastructure), V2I (vehicle-to-infrastructure). However, this invention explicitly does not exclude vehicle-to-X communication with handover, for example, via a mobile radio network.

[0084] Different implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs capable of executing and / or interpreting on a programmable system including at least one programmable processor, which may have special or general purpose applications, and is additionally configured to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system.

[0085] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in high-level and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program (e.g., disk, optical data carrier, memory, SPS) used to provide machine instructions and / or data to a programmable processor, including machine-readable media containing machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0086] The subject matter and the functional flow described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures described herein and their structural counterparts), or in one or more combinations thereof. Furthermore, the content described herein can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable data carrier for execution by a data processing device or for controlling the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material structure that initiates machine-readable propagation signals, or one or more combinations thereof. The terms "data processing device," "computing device," and "computing processor" include all data processing devices, apparatuses, and machines, such as programmable processors, computers, or multiple processors or computers. In addition to hardware, the device may also contain code that creates an execution environment for the associated computer program, such as code representing processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. The propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device.

[0087] In this context, the computing device can be any device designed to process at least one of the signals. Specifically, the computing device can be a processor, such as an ASIC, FPGA, digital signal processor, central processing unit (CPU), multi-purpose processor (MPP), etc.

[0088] Although these processes are shown in a specific order in the accompanying drawings, this should not be construed as meaning that these processes must be performed in the stated order or sequence, or that all shown processes must be executed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of different system components in the above embodiments should not be construed as such separation existing in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0089] Various implementations have been described. However, it is assumed that various modifications can be made without departing from the spirit and scope of this disclosure. Other implementations fall accordingly within the scope of the following statements.

[0090] Various implementations have been described. However, it is assumed that various modifications can be made without departing from the spirit and scope of this disclosure. Other implementations fall accordingly within the scope of the following statements.

Claims

1. A method (100) performed by a sensor system (700) for traffic infrastructure, wherein, The sensor system (700) detects (102a) road users through at least one camera (760) having a first detection area of ​​traffic infrastructure, and detects (102b) road users through at least one radar device (770) having a second detection area of ​​traffic infrastructure, wherein the first detection area and the second detection area at least partially overlap and detect at least one road with multiple lanes of traffic infrastructure, characterized in that a transformation rule for coordinate transformation of radar data (772) acquired by radar device (770) and video data (762) acquired by camera (760) is determined (106) based on the association (104) between road users detected by camera (760) and road users detected by radar device (770); Specifically, location information is accumulated over time from road users detected by cameras and from road users detected by radar devices, and each lane is identified based on the location information accumulated over time from cameras and from the location information accumulated over time from radar devices. Specifically, the parking positions of these road users in each lane are determined using location information accumulated over time by cameras, and the parking positions of these road users in each lane are determined using location information accumulated over time by radar devices, and a correlation is established between the parking positions determined by cameras and the parking positions determined by radar devices.

2. The method according to claim 1, characterized in that, Based on the associated parking locations, a correlation is established between lanes identified by cameras and lanes identified by radar devices, and the associated lanes are considered to establish a correlation between road users detected by cameras and road users detected by radar devices.

3. The method according to claim 1, characterized in that, Road users detected by cameras and radar equipment are selected based on whether they are currently moving or have previously moved and / or have been classified as vehicles.

4. The method according to claim 1, characterized in that, Location information is acquired over time from road users detected by cameras and / or from road users detected by radar devices within a preset and / or adjustable time period.

5. The method according to claim 1, characterized in that, The maximum value of location information accumulated over time by cameras and / or by radar equipment is approximated by a broken line.

6. The method according to claim 1, characterized in that, Determine the parking positions ahead of these lanes, and determine the maximum value of the accumulated position information of the relevant lanes over time.

7. The method according to claim 1, characterized in that, A correlation is established between parking locations determined by cameras and parking locations determined by radar devices by comparing the detected time points when road users are in parking locations and / or moving to parking locations and / or leaving parking locations.

8. The method according to claim 1, characterized in that, At at least one point in time, at least one pair of associated points in the radar coordinates and camera coordinates are detected in order to determine a transformation rule for at least one associated road user, and from multiple point pairs in this respect, a homography matrix between the radar detection plane and the camera image plane is determined.

9. A sensor system (700) for traffic infrastructure, the sensor system comprising at least one camera (760) having a first detection area of ​​the traffic infrastructure and at least one radar device (770) having a second detection area of ​​the traffic infrastructure, wherein, The first detection area and the second detection area at least partially overlap and detect at least one road with multiple lanes of traffic infrastructure, wherein the sensor system (700) is configured to perform the method according to any one of the preceding claims.

Images