Method for detecting infrastructure facilities in the environment of a vehicle

Abstract

In order to provide a method for detecting infrastructure facilities in the environment of a vehicle, which enables precise detection and positioning of an infrastructure facility in the environment of a vehicle, a method for detecting infrastructure facilities (10) in the environment of a vehicle (11), in particular an electric vehicle, is proposed, comprising the following steps: e) recording sensor data (12a, 12b), in particular image data, of the surroundings of the vehicle (11) by means of at least one first sensor device (13a) and one second sensor device (13b) of the vehicle (11), f) detecting at least one infrastructure facility (10) from the sensor data (12a) of the first sensor device (13a) and determining a position of the infrastructure facility (10) relative to the vehicle (11) by means of the first sensor device (13a), g) detecting at least one infrastructure facility (10) from the sensor data (12b) of the second sensor device (13b) and determining a position of the infrastructure facility (10) relative to the vehicle (11) by means of the second sensor device (13b), h) generating fused position data from the position determined by means of the first sensor device (13a) and the position of the infrastructure facility (10) determined by means of the second sensor device (13b).

Description

The invention relates to a method for detecting infrastructure devices in the surroundings of a vehicle, to a device designed to carry out this method, and to a vehicle comprising such a device.Systems and methods are known from the prior art for recognizing various infrastructure devices, such as charging stations, parking spaces, etc., in the surroundings of a vehicle. For this purpose, sensor devices, such as, for example, environmental cameras of the vehicle, are used, inter alia, in order to detect a charging station in a vehicle environment. The information about a charging station present in the vehicle environment can be used for navigation of the vehicle, for navigating the vehicle to the charging station, or for moving a (partially) automatically operated vehicle to the charging station. Such a system is disclosed, for example, in U.S. Pat. No. 2021 / 0095978 A1.Furthermore, KR 2022 0047196 A discloses a method for creating a map of a parking garage using a vehicle sensor system.For some applications of such systems, precise detection of the objects to be detected is necessary to enable reliable control of the vehicle based on the detected position of the object.The invention is based on the object of providing a system for recognizing an infrastructure device in a vehicle environment, which enables precise recognition and position determination of an infrastructure device in an environment of a vehicle.This object is achieved by the features specified in claim 1. Further advantageous embodiments of the invention are described in the dependent claims.The method according to the invention provides that at least two sensor devices of the vehicle are used to detect an infrastructure device in an environment in order to obtain fused position data of the infrastructure device. By fusing the data of a plurality of sensor devices to form a data record, the accuracy of the position detection of the infrastructure device can advantageously be increased.The first and / or second sensor device is in particular a surroundings camera of the vehicle for recording image data. The sensor data or the image data of the first or the second sensor device are preferably transmitted to a control device of the vehicle, where the sensor data are analyzed. This analysis comprises in particular the step of recognizing that it is an infrastructure device which is of interest and whose position is to be determined or tracked. After the infrastructure device has been recognized, the position is then determined in particular from the sensor data of each sensor device.Preferably, each infrastructure device recognized for the first time is assigned an identifier or an identification number (ID).This identification number is used in particular for following the infrastructure device. The infrastructure device is tracked both in the sensor data of the individual sensor devices and via a chronological series of recordings of the same infrastructure device. This ensures that the same infrastructure device is recognized in different sensor data, in particular camera recordings, as one and not more different infrastructure devices. Furthermore, this tracking (also called tracking) serves to improve the position determination of the detected infrastructure device.For determining the position from the sensor data of the individual sensor devices, a common coordinate system is preferably used. The coordinate system is in particular a coordinate system of the vehicle or a coordinate system which has the vehicle as a fixed reference point.The fused position data preferably comprise information about the detected infrastructure device, in particular its type, and its position.During the method, a plurality of infrastructure devices are preferably detected in the vehicle environment, for which infrastructure devices the detection and position determination thereof takes place on the basis of the sensor data of the first and second sensor devices in a manner analogous to the infrastructure device described above. The fused position data then comprise all infrastructure devices identified in the vehicle environment and their respectively ascertained position.Preferably, the sensor data of the first or of the second sensor device are image data. The detection and position determination of the at least one infrastructure device is then carried out by a corresponding image analysis method. In this case, a plurality of infrastructure devices can also be detected from a recording or an image, provided these are in the field of view of the respective sensor device.The sensor data of the individual sensor devices, which were recorded at the same time, are combined to form a data record, and the fused position data are generated from this data record. While the infrastructure device was identified and the position was determined in the image data beforehand, a tabular data record is preferably generated for the fused position data, so that the fused position data are present in a table or database form.The infrastructure device is preferably identified by means of the first and / or the second sensor device with the aid of a segmentation method, in particular an instance segmentation method, in particular using data of a neural network. By means of the segmentation method it can be detected whether it is an infrastructure device which is basically to be detected and is of interest.The method preferably comprises the determination of a characteristic of the infrastructure device from the sensor data and the assignment of the infrastructure device to an infrastructure device class. In addition to detecting whether it is an infrastructure device, it is thus advantageously also possible, on the basis of a detected characteristic, to differentiate which type or class of the infrastructure device it is. The characteristics can be, in particular, optical features. The differentiation between the types or classes can be effected in particular by means of a correspondingly trained neural network.The infrastructure device may be a charging station and / or a parking area. By means of the method, it is thus possible, on the one hand, to identify a charging station and / or a parking area as such on the basis of the sensor data. Furthermore, the type or class of the charging post or of the parking area can also be recognized. For this purpose, characteristics, for example a shape, a coloring and / or a size of the charging post or of the parking area, are preferably recognized, and the charging post or the parking area is assigned to a charging post class or a parking area class on the basis of this information. With the knowledge about the charging column type, further technical details can be used in database or backendbasing fashion, for example the length of the charging cable, the installed plug types or the possible charging power of the charging column.If an infrastructure device cannot be assigned to a class, a class "undefined infrastructure device" can be defined to which unknown infrastructure devices are assigned.Preferably, steps a) to d) of a movement of the vehicle are continuously repeated to track the infrastructure device along a vehicle trajectory. This advantageously enables the infrastructure device to be tracked, and it is always known where the one or more infrastructure devices are located relative to the vehicle.A vehicle odometry is preferably used for determining the vehicle trajectory during the tracking of the infrastructure device.A timer is preferably used for the time-synchronous recording of the sensor data by means of the first and the second sensor device. The fused position data are generated from the data of the individual sensor devices, which were recorded at the same time. Only if a time relationship can be established between the images or data of the various sensor devices or, optimally, the timer initiates the synchronous image or data recording can the following of the infrastructure devices be carried out reliably.Preferably, at least one characteristic parameter of the first and / or second sensor device is used to determine the position of the infrastructure device.The characteristic parameter is preferably a position of the sensor device on the vehicle, an opening angle of the sensor device and / or an orientation of a sensor device.The position of the infrastructure device is preferably determined by detecting a base point at which the infrastructure device is in contact with a ground surface. The base point is in particular that contact point between the ground surface and the infrastructure device which has the smallest distance from the vehicle. Based on the base point, the position of the infrastructure facility can be estimated. The base point can be determined using a distance determination method using the first or second sensor device or by means of image analysis. A three-dimensional position of the infrastructure installation can be estimated from the base point.The position of a parking area is preferably estimated by means of image analysis. Preferably, curbstones or curbstone edges can also be determined by means of detection of the base point.Preferably, an environment map of the vehicle comprising all detected infrastructure devices is created from the fused position data. The environment map can in this case comprise information about the position of the detected infrastructure devices and further specific information such as the respectively assigned class. The environment map can be used for navigation and / or for (partially) automated movement of the vehicle.A position of the infrastructure device currently ascertained by means of the first and / or the second sensor device is preferably compared with an expected value, the expected value being determined from position values ascertained temporally behind.In this case, the following possibilities for determining the expected value and the comparison with current data can be differentiated:• An infrastructure equipment not yet recognized is included in the tracking after a sufficient number of positive detections (e.g. after 5 images) and gets a new ID.• An infrastructure device that is no longer recognized is deleted after a sufficient number (e.g. after 5 images).• A known and re-detected infrastructure facility is acknowledged in its new position.The method preferably comprises the recording of further sensor data by means of at least one third and / or fourth sensor device, wherein the fused position data are generated from the position data of the first to fourth sensor devices. In particular, more than four sensor devices can also be provided. Each of the further sensor devices is used in the course of the method analogously to the first or second sensor device, and is correspondingly designed, so that in particular the fused position data are generated from the sensor data of all sensor devices.The sensor devices are arranged in particular on the vehicle or around the vehicle on the vehicle. The sensor devices are preferably arranged on the vehicle in such a way that the complete vehicle environment can be detected. Furthermore, the detection areas or fields of view of the individual sensor devices can overlap in regions, so that reliable detection of all infrastructure devices in the environment of the vehicle is ensured.The object according to the invention is furthermore achieved by a device designed for carrying out the method described above. The device can in particular comprise a control and / or data processing device or be a control and / or data processing device.Furthermore, the object according to the invention is achieved by a vehicle having the device described above.Exemplary embodiments of the invention are explained in more detail below with reference to the drawings. The following are shown: FIG. 1 shows a flow diagram of a method for recognizing an infrastructure facility, FIG. 2 shows a schematic illustration of a method for detecting a charging station by means of a sensor device of a vehicle, FIG. 3 shows a flow diagram of a method for generating fused position data, and FIG. 4 shows an exemplary embodiment of the generation of fused position data for a plurality of charging stations.In the figures, the same structural elements have the same reference numerals.FIG. 1 shows a flow chart for recognizing and tracking at least one infrastructure device 10 by means of a plurality of sensor devices 13 a, 13 b, 13 c, 13 dof a vehicle 11. sensor data 12 a, 12 b, 12 c, 12 dare recorded by means of the sensor devices 13 a, 13 b, 13 c, 13 d, wherein the sensor devices 13 a, 13 b, 13 c, 13 dare a camera and the sensor data 12 a, 12 b, 12 c, 12 dare in particular camera images. A segmentation A is carried out from the sensor data 12 a, 12 b, 12 c, 12 dusing addition of data 14 of a neural network in order to identify the infrastructure device 10. Furthermore, in the step of segmentation A, a type of infrastructure device 10 is recognized. The infrastructure device A is in particular a charging column.In a next step, the position determination B of the infrastructure device is carried out by the individual sensor devices 13 a, 13 b, 13 c, 13 dusing addition of characteristic parameters 16 of the sensor devices. The position determination B is carried out here, for example, by determining a base point 11 of the charging column. The position determination B is continuously repeated while the vehicle 11 is moving. As a result, with addition of the vehicle odometry data 17, a tracking C of the charging station along a vehicle trajectory is achieved. The position data, which are respectively determined by the individual sensor devices 13 a, 13 b, 13 c, 13 dfrom the sensor data 12 a, 12 b, 12 c, 12 d, are combined in a subsequent step by generating D fused position data.FIG. 2 shows a schematic illustration of a vehicle 11 and an infrastructure device 10. the infrastructure device 10 is a charging station which is detected by means of a sensor device 13 a, 13 b, 13 c, 13 dof the vehicle 11 and the position of which is determined by means of the sensor device 13 a, 13 b, 13 c, 13 d. For this purpose, image data are recorded by means of the sensor device 13 a, 13 b, 13 c, 13 dand a segmentation is first carried out on the basis of the image data in order to detect the charging station and determine the type thereof. For the segmentation, only the expected range 12 bis considered, which is initially determined and indicates the range in which the charging column is expected. The detection and type determination of the charging column then takes place within the expectation range 12 band the segmentation boundary 12 ais determined, which represents the boundaries or the contour of the charging column.Further, the position of the charging post is estimated by determining a base point 19 at which the charging post meets the ground. This point is selected from a series of measurement points 18, 19, 20 which correspond to individual distance measurements or projections of a measurement beam of the sensor device 13 a, 13 b, 13 c, 13 d. Individual measured values 18 are discarded because they either correspond to an excessive distance compared to the remaining measured values, or because the measured value corresponds to a position behind the sensor device 13, 13 b, 13 c, 13 d.FIG. 3 shows a flow diagram of the method for generating the fused position data. The individual sensor data 12 a, 12 b, 12 c, 12 dare shown, from which the position determination B 1, B 2, B 3, B 4 and the tracking C 1, C 2, C 3, C 4 of a charging column are respectively carried out separately during the travel of the vehicle 11. The separately detected charging stations and their position data are then merged by generating D a merged data set. The generation D of the fused data set takes place continuously during the travel of the vehicle 11 as well as the detection and position determination of the charging stations, so that current position data of the charging stations are always present for each vehicle position and at each point in time.The detected charging stations and their positions are entered into an environment map or these are generated on the basis of the information data comprising the IDs of the charging stations and the position data of the charging station. This map of the surroundings can be used as the basis for navigation of the vehicle. Furthermore, the environment map can be used for the (partially) automated movement of the vehicle.The environment map thus created is advantageously characterized in that it comprises particularly precisely and robustly detected charging stations, since the detection and the position determination for each of the charging stations takes place on the basis of sensor data 12 a, 12 b, 12 c, 12 dof a plurality of sensor devices 13 a, 13 b, 13 c, 13 d, as a result of which a greater accuracy in the detection and position determination can be achieved.FIG. 4 shows an exemplary embodiment in which the position data which were generated on the basis of the sensor data 12 a, 12 b, 12 c, 12 dof the individual sensor devices 13 a, 13 b, 13 c, 13 dare fused in order to insert these into a common environment map. In this case, new, unique IDs must also be assigned to some extent for the charging stations. By means of various influencing variables (camera- or objective-specific uncertainties, uncertainty in the segmentation, uncertainty in the position or distance determination), the detected position of the charging stations can deviate between the individual sensor devices 13 a, 13 b, 13 c, 13 d. FIG. 4 shows how the incoming IDs of the charging stations of different sensor devices 13 a, 13 b, 13 c, 13 dare fused to form new, unique IDs.Here, circles filled with f n( ) represent the fused charging columns from a generation of the fused position data in FIG. 4 a. The filled circles or the fused charging stations thus each represent a detected charging station at a determined position, wherein the data assigned to the charging station, which are generally the position data and an ID, were generated from the sensor data 12 a, 12 b, 12 c, 12 dof a plurality of sensor devices 13 a, 13 b, 13 c, 13 d. The n n( blank diamonds) symbolise new IDs of different sensor devices 13 a, 13 b, 13 c, 13 d. At the beginning in FIG. 4 a, there are already three fused charging stations with the IDs 1 to 3. The fused ID contains f 1 the charging station 0 from a first sensor device 12 a, in this case a camera directed forward (front), and the charging station 0 from a second sensor device 13 b, in this case the camera directed rightward (right). The fused ID f 2 contains the charging post 2 from the camera forward and the charging post 1 from the camera rightward. ID f 3 contains the charging post 3 only from the camera forward.In FIG. 4 b, after a time t, new data of the camera arrive to the right. This is the new data n n. The object now consists in ascertaining for these new data or candidates for recognized infrastructure devices whether these are already known or not yet recognized infrastructure devices.For this purpose, in FIG. 4 c, the fused charging stations f 1 and f 2 are matched to the new data n n. It is determined here that n 0( right 0) and n 1( right 1) are found as tracked objects of the camera to the right at the expected location of f 1 and f 2 and are thus directly assigned.In FIG. 4 d, the position of the charging column n 3( right 3) is not in sufficient proximity (expected location) to an ID already existing in the fusion. A distance-based optimum is thus sought via a suitable algorithm, for example a Kuhn-Munkres algorithm or a Greedy algorithm. As a result, this charging post can be assigned to the existing ID f 3. The charging column n 2( right 2) is not yet known in the fusion or no fused charging column exists in sufficient proximity, thus a new fused ID f 4 is created for this charging column.Thus, after generating fused data sets and continuously recording new sensor data, the result is a data set of an optimized position estimate comprising unique IDs of the charging stations around the vehicle 11.List of reference characters10 Infrastructure device 11 Vehicle 12 aSensor data of the first sensor device 12 bSensor data of the second sensor device 12 cSensor data of the third sensor device 12 dSensor data of the fourth sensor device 13 aFirst sensor device 13 bSecond sensor device 13 cThird sensor device 13 d Vierte sensor device 14 Data of the neural network A Segmentation step 12 a Segmentierung limit 12 b Erwartung range of segmentation 16 Characteristic parameters of a sensor device B, B 1-B 4 Position determination 17 Vehicle odometry data C, C1-C4 Tracking of the infrastructure facility D Generating fused position data E Creating an environment map 18 discarded measurement points 19 base point 20 measurement points f 1 to f 3 identifiers n 1 to n 3 newly recognized charging stationsReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedUS 2021 / 0095978 A1

[0002] KR 2022 0047196 A

[0003]

Claims

Method for recognizing infrastructure devices (10) in a vicinity of a vehicle (11), in particular of an electric vehicle, comprising the following steps: a) recording sensor data (12a, 12b), in particular image data, of the vicinity of the vehicle (11) by means of at least one first sensor device (13a) and one second sensor device (13b) of the vehicle (11), b) recognizing at least one infrastructure device (10) from the sensor data (12a) of the first sensor device (13a) and determining a position of the infrastructure device (10) relative to the vehicle (11) by means of the first sensor device (13a), c) recognizing at least one infrastructure device (10) from the sensor data (12b) of the second sensor device (13b) and determining a position of the infrastructure device (10) relative to the vehicle (11) by means of the second sensor device (13b), d) Generating fused position data from the position determined by means of the first sensor device ( 13 a) and the position of the infrastructure device ( 10) determined by means of the second sensor device ( 13 b).Method according to Claim 1, wherein the infrastructure device (10) is identified by means of the first and / or the second sensor device (13a, 13b) with the aid of a segmentation method, in particular an instance segmentation method, in particular using data (14) of a neural network.Method according to one of the preceding claims, comprising the determination of a characteristic of the infrastructure device (10) from the sensor data and the assignment of the infrastructure device (10) to an infrastructure device class.Method according to one of the preceding claims, wherein the infrastructure device (10) is a charging column and / or a parking area.Method according to one of the preceding claims, wherein steps a) to d) are repeated continuously during a movement of the vehicle (11) in order to track the infrastructure device (10) along a vehicle trajectory.Method according to Claim 5, wherein a vehicle odometry (17) is used for determining the vehicle trajectory during the tracking of the infrastructure device (10).Method according to one of the preceding claims, wherein a timer is used for the time-synchronous recording of the sensor data by means of the first and the second sensor device (13a, 13b, 13c, 13d).Method according to one of the preceding claims, wherein at least one characteristic parameter (16) of the first and / or second sensor device (13a, 13b) is used to determine the position of the infrastructure device (10).Method according to Claim 8, wherein the characteristic parameter (16) is a position of the sensor device (10) on the vehicle (11), an opening angle of the sensor device and / or an orientation of a sensor device.Method according to one of the preceding claims, wherein the position of the infrastructure device (10) is determined by detecting a base point (19) at which the infrastructure device (10) is in contact with a ground surface.Method according to one of the preceding claims, wherein a map (E) of the environment of the vehicle comprising all detected infrastructure devices is created from the fused position data.Method according to one of the preceding claims, wherein a position of the infrastructure device (10) currently determined by means of the first and / or the second sensor device (13a, 13b) is compared with an expected value, wherein the expected value is determined from position values determined in a past manner.Method according to one of the preceding claims, comprising the recording of further sensor data (12c, 12d) by means of at least one third and / or fourth sensor device (13c, 13d), wherein the fused position data are generated from the position data (12a, 12b, 12c, 12d) of the first to fourth sensor devices (13a, 13b, 13c, 13d).Device designed to carry out a method according to one of the preceding claims.Vehicle (10) comprising a device according to claim 14.

Images