Method, operating method, device, computer program and storage medium for providing reference data

Abstract

A method for providing reference data of at least one reference vehicle to a first providing system is proposed, wherein position data of a plurality of reference vehicles is transmitted to a second providing system and position data of at least one test vehicle is transmitted to the first providing system, the method having the following steps: providing the position data of the reference vehicles by the second providing system to the first providing system; comparing, by the first providing system, the current position data of the at least one test vehicle with the position data of the plurality of reference vehicles at the same time to determine at least one reference vehicle at the current position of the at least one test vehicle; transmitting the reference data from the second providing system to the first providing system if at least one reference vehicle has been determined at the current position of the at least one test vehicle.

Description

Technical Field

[0001] The present invention relates to a method for providing reference data for at least one reference vehicle, a method for operating a vehicle, an apparatus, a computer program, and a machine-readable storage medium. Background Technology

[0002] The development, assurance, evaluation, and validation of algorithms and systems for partially and fully automated driving, as well as the training of machine learning methods (such as deep learning) for partially and fully automated driving, require different types of reference data. This reference data particularly includes labeled samples of sensor data measured in (partially) autonomous vehicles and records of “Ground Truth” data, which represent the most accurate possible description of the actual vehicle environment. This reference data is typically acquired specifically during test drives and is often manually annotated, which now constitutes a large portion of the cost in the development of driver assistance features.

[0003] The creation of "ground-based real-time" data is typically accomplished using specialized test vehicles equipped with sensors of superior quality to the system under test. For example, a combination of lidar, radar, and video can be used as a reference for developing purely video-based AEB (Autonomous Emergency Braking) functionality. A drawback of this approach is that typically only small test fleets are equipped with these sensors.

[0004] Another known possibility is the use of a "rabbit vehicle" equipped with communication devices and accurate measuring devices, which transmits this data to the test vehicle, for example, through a fixed communication channel.

[0005] The drawback of this method is that it can only calibrate a single, predefined scenario, but cannot discover new and interesting scenarios in the field. Summary of the Invention

[0006] One possibility for reducing costs is to use an existing fleet of reference vehicles to generate reference data. This means that if one of the reference vehicles in the fleet happens to be near the test vehicle, that reference vehicle can be used as a reference for the test vehicle without manual annotation. The larger the number of such reference vehicles, the more reference data can be generated.

[0007] Many companies aiming to develop autonomous driving capabilities have only small fleets of a few hundred vehicles. For these companies, accessing the fleet's reference data offers a cost advantage compared to manual annotation. The reference data supplier can then be paid by these companies for this advantage. On the other hand, there are numerous OEMs and large fleet operators, some of whom have no interest in developing autonomous driving capabilities. Some of these companies (or possibly private individuals) can provide their location data as reference data for a fee.

[0008] Beyond cost, simply collecting data from a small test fleet is often insufficient for developing automated driving features: obtaining 99.99% of all driving situations and developing a function capable of handling those situations is straightforward. Acquiring the final 0.01% of exceptional situations requires several times more road kilometers, and the less likely a situation is to occur, the more road kilometers are needed to "experience" it at least once. For this reason, a method for data collection is needed that utilizes data from a larger fleet, such as a complete fleet from an OEM.

[0009] Using an exchange platform, reference data from one or more fleets of reference vehicles can be used by companies developing test vehicles. This reference data can be used to develop, safeguard, evaluate, and validate algorithms and systems using location data and / or sensor data from other vehicles. Here, data transfer is not direct from vehicle to vehicle (V2X), but rather through backend servers (the cloud).

[0010] According to various aspects of the invention, a method for providing reference data for at least one reference vehicle, a method for operating a vehicle, an apparatus, a computer program, and a machine-readable storage medium are provided. Advantageous configurations are also given below.

[0011] Throughout the description of this invention, the order of the method steps is shown to facilitate understanding of the method. However, those skilled in the art will recognize that many of the method steps can be performed in a different order and lead to the same or corresponding results. In this sense, the order of the method steps can be changed accordingly. Some features are accompanied by counters to improve readability or to make the assignment more explicit, but this does not imply the presence of a defined feature.

[0012] According to one aspect, a method is proposed for providing reference data of at least one reference vehicle to a first providing system, wherein position data of a plurality of reference vehicles are transmitted to a second providing system, and position data of at least one test vehicle is transmitted to the first providing system. The method includes the following steps:

[0013] In one step, the location data of the reference vehicle is provided to the first providing system via the second providing system.

[0014] In another step, the current position data of at least one test vehicle is compared with the position data of a plurality of reference vehicles at the same time by a first providing system to determine at least one reference vehicle at the current position of at least one test vehicle.

[0015] In another step, if at least one reference vehicle has been identified at the current location of at least one test vehicle, reference data is transmitted from the second providing system to the first providing system.

[0016] Here, the reference data includes reference location data (which has higher accuracy compared to location data) and / or other data of the reference vehicle.

[0017] Here, the current location is determined using the current location data.

[0018] To generate reference data, the reference vehicles are those vehicles configured to determine an accurate location and transmit that location wirelessly to a second providing system.

[0019] The vehicle whose reference data is used in conjunction with the reference vehicle is referred to hereinafter as a test vehicle. This includes, for example, test vehicles on the road for functional verification within the scope of testing activities. However, it can also refer to a series of vehicles that have encountered abnormal conditions during routine operation, which should now be stored for functional review and further development.

[0020] Compared to manually creating reference data, this method offers a significant cost advantage. Furthermore, it is not limited to small fleets using dedicated test vehicles. This method can be applied to any series of vehicles, allowing for the acquisition of a very large number of scenarios in a very short time. This enables the acquisition of rare and interesting scenarios for the development of automated driving functions in a relatively short period.

[0021] Compared to known methods for creating reference data, such as through V2X automation, this method has the following advantages: it requires very little hardware in third-party vehicles. This provides a further cost advantage, and the fleet that can be used for reference data generation can thus be significantly larger again.

[0022] The following are sufficient for reference data generation: the vehicle is capable of determining its precise location, for example, using a radar system, and / or a location determination using a global navigation satellite system and inertial sensors; and the vehicle has a communication connection, for example, via mobile radio. Radar sensors and mobile radio connectivity are likely to be standard equipment in future new vehicles. The solution can even be easily retrofitted into existing vehicles via an interface connector and the driver's mobile phone.

[0023] Furthermore, compared to other methods, the bandwidth required to transmit reference data, especially vehicle location, is very low, and can be further reduced by leveraging other aspects of this method.

[0024] The method used to provide reference data can, for example, be used to verify driver assistance functions. In other words, the method coordinates the provision of reference data. For example, the method continuously checks a database for, for example, fleets of delivery services and mobile providers. This involves determining the location data of reference vehicles for the entire fleet of OEMs. This location data can be read directly from the vehicles, or it can be received wirelessly first via, for example, the OEM's backend server system (cloud) and then further transmitted. Data from multiple such reference vehicle fleets can also be merged.

[0025] The vehicles for which reference data should be used are referred to hereinafter as test vehicles. This includes, for example, experimental vehicles driven around the area for functional verification within the scope of testing activities. However, it could also refer to a series of vehicles that encountered anomalies during routine operation, which should now be stored for functional review and further development.

[0026] If this type of vehicle requires reference data, the test vehicle also sends its location to the backend server system (cloud). This searches the database for reference vehicles in the test vehicle's environment. If enough reference vehicles exist to perform the test, the server can query additional data from the reference vehicles, such as more precise location, vehicle identification number (VIN) (and therefore vehicle model, geometry, color, etc.).

[0027] This data requires only low communication bandwidth. Additional environmental sensor data from the reference vehicle can also be transmitted if desired, and if the communication channel is sufficiently high-performance. Furthermore, the reference data in the backend server cloud can be enriched with other data, such as lane information presented from digital maps.

[0028] According to one aspect, in order to determine at least one reference vehicle at the current position of at least one test vehicle, the position data of at least one reference vehicle is examined to determine whether at least one reference vehicle is located within a defined radius of the current position of at least one test vehicle. That is, if the position data of the reference vehicle matches the position of the test vehicle accurately enough to detect the reference vehicle from the test vehicle, for example by means of a sensor system, then the reference vehicle or reference data of the reference vehicle at the current position of the test vehicle is available. Here, the typical distance between the test vehicle and at least one reference vehicle can be 400m in all directions.

[0029] According to one aspect, if at least one reference vehicle has been identified at the current location of at least one test vehicle, vehicle data is transmitted from the at least one test vehicle to a first providing system. To this end, the first providing system may send a trigger signal to the test vehicle to enable the test vehicle to provide vehicle data.

[0030] Here, vehicle data may contain important and relevant detailed data about a specific test performed with the aid of the corresponding test vehicle. In particular, vehicle data may contain reference position data, at least one position of at least one reference vehicle detected by the test vehicle or trigger determination, and / or other data of the test vehicle, which have higher accuracy than position data about the test vehicle's own position.

[0031] According to one aspect, if at least one reference vehicle has been identified at the current location of at least one test vehicle, reference data is transmitted from at least one reference vehicle to a second providing system.

[0032] According to one aspect, if the first providing system identifies an available reference vehicle at the current location of the test vehicle, the first providing system sends a trigger signal to the second providing system for providing reference data.

[0033] According to one aspect, if at least one reference vehicle has been identified at the current location of at least one test vehicle, reference position data of the reference vehicle is provided to the first providing system by means of a second providing system.

[0034] In this context, reference location data can achieve higher location accuracy than location data.

[0035] According to one aspect, the reference data includes reference location data and / or vehicle data of the reference vehicle and / or identification number of the reference vehicle (e.g., chassis number (VIN and therefore vehicle model, geometry, color, etc.)) and / or lane information from a digital map.

[0036] According to one aspect, the vehicle data includes reference location data and / or other detailed measurement data and / or sensor data of the test vehicle.

[0037] According to one aspect, the reference vehicle determines its position using a radar system and / or a global navigation satellite system and / or an acceleration sensor, and transmits that position wirelessly, for example, via a mobile radio, to a second providing system.

[0038] According to one perspective, the first and second providing systems are parts of the overall providing system.

[0039] According to one aspect, reference data is provided from at least one reference vehicle to the test vehicle.

[0040] According to one aspect, reference data of a reference vehicle is provided to a third providing system via a first providing system, wherein a test vehicle is coupled to the third providing system corresponding to the previously described first providing system. Here, the position data of the test vehicle can be provided to the first providing system via the third providing system.

[0041] According to one perspective, providing reference data for the test vehicles of the third providing system is an economic compensation or reward from the operator of the third providing system to the operator of the first providing system, and / or the first providing system calculates the value of the compensation.

[0042] According to one aspect, a first providing system and a second providing system are provided by means of a back-end server system, and the back-end server system is configured to wirelessly exchange data with multiple reference vehicles and / or wirelessly exchange data with at least one test vehicle.

[0043] According to one aspect, the first providing system requests reference data in the reference vehicle by means of the second providing system. Here, the request can be made in the form of a trigger signal.

[0044] According to one aspect, the transmission of vehicle data from at least one test vehicle to the first providing system is carried out in a time-staggered manner. Specifically, for example, the test vehicle may transmit vehicle data after arriving at a fixed location such as a garage.

[0045] According to one approach, reference data will be read directly from the test vehicle and / or received via the OEM's cloud and then further transmitted.

[0046] According to one aspect, in addition to the second providing system, such as the second providing system of an OEM, a plurality of other providing systems, such as a plurality of other providing systems of other OEMs, are coupled to the first providing system in terms of signaling to provide additional reference data of other reference vehicles for at least one test vehicle, and these other providing systems corresponding to the second providing system each have other reference vehicles.

[0047] Therefore, this method is not fixed to a single supplier or OEM, but can access reference data from multiple OEMs in parallel. Thus, in the case of different OEMs / fleet suppliers, a "second" supply system can be provided n times, and the first supply system queries the appropriate reference vehicle or reference data from all other supply systems.

[0048] Therefore, location data can be read directly from the vehicle, or it can be received first via the cloud, such as the OEM's cloud, and then further transmitted. Data from multiple fleets can also be merged.

[0049] According to one aspect, the second providing system calculates the amount of economic compensation used to provide reference data to the first providing system.

[0050] According to one perspective, the second providing system negotiates with the first providing system the amount of economic compensation for providing reference data.

[0051] This compensation can be in cash. In other words, the fleet operator of the reference vehicles can use this method to initially provide, free of charge, somewhat inaccurate and, in particular, anonymized location data of their reference vehicles, enabling a comparison with the test vehicle's location. Once sufficient reference vehicles are available in the test vehicle's current environment, complete reference data can be retrieved, and the fleet operator receives payment for it. This reference data can then be further passed to users, who pay the operator of the initial system provider for this data (potentially along with an exchange fee).

[0052] An application of reference data of at least one of a plurality of reference vehicles is proposed for testing and / or validating a system of the test vehicle for object recognition, and / or for validating a system of the test vehicle for object localization, wherein the reference data is provided by a second providing system to a first providing system in accordance with one of the methods described above.

[0053] According to one perspective, the application and / or further processing of the reference data are performed directly in the test vehicle. Therefore, the reference data must be transmitted to the test vehicle.

[0054] According to one approach, the comparison between test vehicle data and reference data is performed in a first providing system. For this, the comparison data of the test vehicle must be transmitted to the first providing system. While this requires a larger data volume, it is only applicable to short time periods. Specifically, this can also be performed at later times, for example, via WLAN in a garage.

[0055] According to one perspective, the comparison between the test vehicle data and the reference data is conducted in a third-party system.

[0056] One approach proposes merging reference data from multiple systems to allow for the consideration of a larger number of reference vehicles.

[0057] According to one aspect, a first providing system provides a test vehicle with information about at least one reference vehicle at the current location of the test vehicle. Here, information about a sufficient number of reference vehicles can be provided to the test vehicle, enabling the test vehicle to determine the required comparison data.

[0058] A method is proposed in which control signals for operating a vehicle that is at least partially automated are provided based on reference data provided according to one of the above methods; and / or warning signals for warning vehicle occupants are provided based on the provided reference data.

[0059] The term "based on" should be interpreted broadly with respect to the following characteristic: control signals are provided based on provided reference data, corresponding to one of the methods described above. This term should be understood as meaning that, based on the provided reference data, corresponding to one of the methods described above, any determination or calculation of the control signal can be considered, where this does not exclude the use of other input parameters for determining the control signal. This applies accordingly to the provision of warning signals.

[0060] An apparatus is proposed, which is configured to perform one of the methods described above.

[0061] A computer program is proposed that includes instructions, which, when executed by a computer, cause the computer to perform one of the methods described above. This computer program enables the described methods to be used in different systems.

[0062] A machine-readable storage medium is proposed on which the aforementioned computer program is stored. This machine-readable storage medium makes the aforementioned computer program transportable. Attached Figure Description

[0063] refer to Figures 1 to 3 Embodiments of the present invention are shown and further described below. The accompanying drawings illustrate:

[0064] Figure 1This illustrates the information flow of the method used to provide reference data;

[0065] Figure 2 This shows the information flow used for exchanging reference data;

[0066] Figure 3 The information flow used to verify object identification with reference data from a reference vehicle is shown. Detailed Implementation

[0067] Figure 1 The information flow of a method for providing reference data of at least one reference vehicle 120 to a first providing system 220 is schematically outlined, wherein the position data of a plurality of reference vehicles 120 are transmitted to a second providing system 240S30, and the position data of at least one test vehicle 100 are transmitted to the first providing system 220S20.

[0068] In step S10, the position data of the reference vehicle 120 is provided to the first providing system 220 by the second providing system 240. In another step S20, the first providing system 220 compares the current position data of at least one test vehicle 100 with the position data of the plurality of reference vehicles 120 at the same time to determine at least one reference vehicle 120 at the current position of at least one test vehicle 100.

[0069] In another step S34, if at least one reference vehicle 120 has been identified at the current position of at least one test vehicle 100, reference data is transmitted from the second providing system 240 to the first providing system 220. To provide position data, if at least one reference vehicle 120 has been identified at the current position of at least one test vehicle 100, the first providing system 220 may send a trigger signal to the second providing system 240 S32. To provide position data of the reference vehicle 120, the second providing system 240 may send a trigger signal to at least one reference vehicle 120 so that the reference vehicle 120 transmits the position data of the reference data to the second providing system 240 S10.

[0070] If at least one reference vehicle 120 has been identified at the current location of at least one test vehicle 100, the first providing system 220 may trigger the test vehicle 100S24 to provide reference position data and other detailed measurement data and / or sensor data.

[0071] The first providing system 220 and the second providing system 240 may be parts of the overall providing system 200.

[0072] In other words, for example, all reference vehicles 120 in the OEM fleet continuously transmit their location and / or identification data (e.g., chassis number) to the second providing system 240 (OEM cloud) S10. The second providing system, particularly anonymously and possibly with low accuracy, further transmits the location data to the first providing system 220 S30. The test vehicle (self-vehicle) 100 also continuously transmits its location data to the first providing system 220 S20. The first providing system 220 now searches for reference vehicles 120 in the environment of the test vehicle 100. If a reference vehicle is found, a detailed measurement S24 is requested (triggered) in the test vehicle 100 for verification S22, and the accurate reference location and identification of the reference vehicle 120 is requested in the second providing system 240 S32, which then provides the accurate reference location and identification to the first providing system 220 S34.

[0073] Figure 2 The information flow for the exchange of reference data is schematically outlined. Here, reference data is collected by the first providing system 220 from the second providing system 240 and the fourth providing system 260. The reference data thus collected is provided by the first providing system 220 to the third providing system 280. Here, at least one test vehicle is coupled to the third providing system in terms of signaling, as described above with respect to the first providing system 220 and the test vehicle. Thus, possible tests can be performed with the aid of the third providing system 280. For this example, neither the third providing system 280 nor the first providing system 220 requires a reference vehicle 120 or a fleet of reference vehicles coupled to these providing systems. Here, the second providing system 240 and the fourth providing system 260 are able to receive payment for the provision of reference data, and the first providing system is able to receive payment from the operator of the third providing system 280 for the exchange and further transmission of reference data.

[0074] Figure 3The information flow of an example application of this method in the verification of object recognition is schematically outlined. The test vehicle (self-vehicle) 100, through environmental perception, identifies a reference vehicle 120 traveling ahead. The accuracy of this identification should be checked using reference data provided by the method. To this end, the test vehicle 100 continuously transmits its position (x, y) to a first providing system 220 (S20). The reference vehicle 120 transmits its position data to a second providing system 240 (S10). The particularly anonymous and potentially inaccurate position of the reference vehicle 120 is further transmitted from the second providing system 240 to the first providing system 220 (S30). The first providing system 220, using this position data of the reference vehicle 120, identifies that the reference vehicle 120 exists in the environment of the test vehicle 100, and triggers S32 for the environment of the test vehicle 100, whereby the second providing system 240 provides reference data (S34). The second providing system 240 provides the accurate reference position of the reference vehicle 21 traveling ahead, along with (possibly) vehicle data (S34). Now, the first providing system 220 triggers S22 to provide complete and detailed data S24 of the environmental model in the test vehicle 100. With the help of the first providing system 220, the observation position 300 of the test vehicle 100 can be compared with the reference position of the reference vehicle 120 traveling ahead.

[0075] This method enables, for example, field validation or verification of new algorithms and / or systems. For instance, if a new method or system has been developed for object recognition from environmental sensors, the position of a reference vehicle in the environment can be transferred from a second providing system to a first providing system, where the reference position can be compared with the newly identified environmental position.

Claims

1. A method for providing reference data of at least one reference vehicle (120) to a first providing system (220), wherein, The method involves transmitting the location data of multiple reference vehicles (120) to a second providing system (240) and transmitting the location data of at least one test vehicle (100) to the first providing system, the method comprising the following steps: The location data of the reference vehicle (120) is provided to the first providing system (220) through the second providing system (240) (S10). The first providing system (220) compares the current position data of the at least one test vehicle (100) with the position data of the plurality of reference vehicles (120) at the same time to determine at least one reference vehicle (120) at the current position of the at least one test vehicle (100). If at least one reference vehicle (120) has been identified at the current location of the at least one test vehicle (100), the reference data is transmitted from the second providing system (240) to the first providing system (220) (S30).

2. The method according to claim 1, wherein, If at least one reference vehicle (120) has been identified at the current location of the at least one test vehicle (100), vehicle data is transmitted from the at least one test vehicle (100) to the first providing system (220) (S20).

3. The method according to any one of the preceding claims, wherein, If at least one reference vehicle (120) has been identified at the current location of the at least one test vehicle (100), the reference data is transmitted from the at least one reference vehicle (120) to the second providing system (240) (S10).

4. The method according to claim 1 or 2, wherein, If at least one reference vehicle (120) has been identified at the current location of the at least one test vehicle (100), the reference position data of the reference vehicle (120) is provided to the first providing system (220) by the second providing system (240) (S34).

5. The method according to claim 1 or 2, wherein, The first providing system (220) and the second providing system (240) are part of the overall providing system (200).

6. The method according to claim 1 or 2, wherein, The reference data is provided from the at least one reference vehicle (120) to the test vehicle (100).

7. The method according to claim 1 or 2, wherein, The reference data of the reference vehicle (120) is provided to the third providing system (280) by the first providing system (220), wherein the test vehicle (100) is coupled to the third providing system (280) corresponding to the first providing system (220), and the position data of the test vehicle (100) is provided to the first providing system (220) by means of the third providing system (280).

8. The method according to claim 1 or 2, wherein, The first providing system (220) requests reference data in the reference vehicle (120) by means of the second providing system (240) (S22).

9. The method according to claim 1 or 2, wherein, The second providing system (240) calculates the amount of economic compensation for providing the reference data to the first providing system (220).

10. The method according to claim 1 or 2, wherein, Based on the provided reference data, control signals are provided for operating a vehicle that is at least partially automated; and / or warning signals are provided based on the provided reference data for warning vehicle occupants.

11. A method for applying reference data of at least one of a plurality of reference vehicles (120), the reference data being used to test and / or verify a system of a test vehicle (100) for object recognition, and / or to verify a system of the test vehicle (100) for object localization, the reference data being provided by a second providing system (240) to a first providing system (220) in accordance with the method of any one of claims 1 to 10.

12. A computer program product comprising instructions that, when implemented by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

13. A machine-readable storage medium on which a computer program is stored, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images