Method for processing sensor data of a motor vehicle

Abstract

The invention relates to a method for processing sensor data of a motor vehicle. The invention relates to a method for processing sensor data in a motor vehicle (1), comprising at least the following steps: a) detecting first data using a first sensor (2) of a first sensor type; b) detecting second data using a second sensor (3) of a second sensor type; c) extrapolating from the first data detected in step a and / or from the second data detected in step b in order to create extrapolated data which enables a comparability of the first data to the second data; and d) performing common data processing on the first data and / or the second data by using the extrapolated data.

Description

Technical Field

[0001] A method for processing sensor data in a motor vehicle is described. The method can be used, in particular, to improve the position determination of a global navigation satellite system sensor using environmental data obtained by sensors, such as lidar or radar sensors, or cameras. The method can also be used to verify correlations between sensor data from different sensor types. Background Art

[0002] In previous applications, GNSS sensors, lidar sensors, radar sensors and / or cameras were usually used independently of one another, and the quality of the respective sensor data was independently verified for each sensor system setup. Summary of the Invention

[0003] Starting from here, a new approach will be described for processing sensor data from different sensors, where the different sensors are of different sensor types.

[0004] This object is achieved by the present invention according to the features of the independent patent claims. Further advantageous embodiments are given in the independent patent claims, in the specification, and in particular in the description of the drawings. It should be noted that a person skilled in the art can combine the individual features in a technically meaningful manner to produce further embodiments of the invention.

[0005] Described here is a method for processing sensor data in a motor vehicle, comprising at least the following steps:

[0006] a) detecting first data using a first sensor of a first sensor type;

[0007] b) detecting second data using a second sensor of a second sensor type;

[0008] c) extrapolating the first data detected according to step a) and / or the second data detected according to step b) to create extrapolated data, which enables comparability of the first data and the second data; and

[0009] d) performing common data processing on the first data and / or the second data using the extrapolated data.

[0010] The first sensor and the first sensor type, or the second sensor and the second sensor type, are preferably environmental sensors that can be used to monitor the vehicle environment. The first sensor and the second sensor are of different types and, particularly preferably, even employ different physical measurement principles. In alternative embodiments, it is also possible for the first sensor type and the second sensor type to utilize the same physical measurement principle but be configured for different sensor characteristics. For example, the different sensor configurations can be adapted to specific monitoring areas in the vehicle environment. In particular, this can involve focusing the corresponding sensor on a specific monitoring area at a specific distance in front of and / or behind the vehicle.

[0011] Acquiring the first data in step a) and the second data in step b) here essentially means that the data are acquired by the sensor module. This also means that the data generated by the sensor are acquired by the controller that executes the method.

[0012] The extrapolation step in step c) describes a modification of the first data and / or the second data, by means of which the comparability of the data is achieved. Thus, the basis for the joint data processing in step d) is achieved. The possible joint data processing of the first data and / or the second data with the extrapolated data can be used for different purposes. In a first embodiment variant of the method, the joint data processing according to step d) includes in particular checking or verifying different data from different data sources (first sensor and second sensor). In a second embodiment variant of the method, the joint data processing according to step d) also includes obtaining additional information, which in particular includes fusing and / or complementing the first data or the second data with the extrapolated data.

[0013] The method described here is described in particular for two different sensors (a first and a second sensor). Preferably, however, the method can also be applied to more than two sensors, in particular also two or more sensors of two or more different sensor types (e.g., at least three sensors of at least three different sensor types).

[0014] It is particularly advantageous if the first sensor used in step a) and the second sensor used in step b) are each one of the sensor types included in the following list:

[0015] -LiDAR

[0016] -radar

[0017] -Front camera

[0018] -Infrared camera

[0019] - Ultrasonic sensor; or

[0020] – Interior space camera.

[0021] The aforementioned sensor types are environmental sensors that operate according to different physical measurement principles. By means of the extrapolation in step d), the data from sensors of different sensor types can be compared with one another and processed together.

[0022] Furthermore, it is advantageous if the extrapolation in step c) comprises at least one of the following sub-steps:

[0023] i. Obtain at least one scaling factor for scaling the second data and the first data relative to each other;

[0024] ii. Obtaining a shift coefficient for describing the shift of the second data and the first data relative to each other;

[0025] iii. Applying at least one scaling factor and / or at least one shifting factor to the first data and / or the second data to create extrapolated data.

[0026] The scaling factor and / or the shift factor can be determined, for example, by assigning specific structures identified in the first and second data to one another and then determining the scaling factor and the shift factor based on this assignment. For example, if structure A and structure B have a first distance in the first data from the first sensor and a second distance in the second data from the second sensor, the scaling factor can be determined, for example, using the quotient of these two distances. Shift factors can also be determined based on these principles, where a larger number of reference points (structures) may be required in order to determine the shift and scaling factors in different spatial directions.

[0027] This method is particularly preferably applicable when the vehicle is traveling in an urban environment with buildings on the roadside. In such an environment, there are multiple structural features on both sides of the roadway, which can be identified and allocated to each other in the first data and the second data.

[0028] It should be considered that the areas in the vehicle environment monitored by different sensors of different sensor types do not necessarily have to be identical at every point in time. For example, it is possible to use a first sensor of a first sensor type to monitor an area at a first distance in front of the vehicle. Conversely, a second sensor of a second sensor type can monitor an area at a second distance in front of the vehicle, where the second distance is less than the first distance. In one example, the second distance is, for example, 10 meters less than the first distance. The second sensor can then be used to identify a structure that was identified by the first sensor at an earlier point in time. For example, the earlier point in time at which the first sensor identified a certain structure occurs before the later point in time at which the second sensor identified the same structure. In this case, this time interval corresponds to the time required for the vehicle to travel the distance between the first and second distances. In this case, this is 10 meters. For example, if the vehicle is traveling at 5 meters per second, the time interval is 2 seconds. These dependencies must be taken into account, in particular, by using shift and scaling factors to extrapolate the data into extrapolated data according to step c).

[0029] Furthermore, it is advantageous if, in step c), an extrapolation is performed with the aid of map data.

[0030] The use of map data often additionally helps to associate specific structures with one another both in the sensor data and in the map data.

[0031] The map data used within the scope of this method is, in particular, high-resolution map data, which can very accurately depict structural features (such as buildings) in the road surroundings. Using the high-resolution map, the method described in step d) preferably determines which of the two sensors (the first sensor or the second sensor) is still functioning properly in the event of conflicting position information. Other systems can also be used to determine which system is functioning properly. However, this typically requires that the evaluation ranges of the first and second sensors overlap, while simultaneously monitoring a specific area in the vehicle surroundings. The method described herein, and in particular the extrapolation according to step c), allows the sensor data to be assigned to one another independently of the map data, allowing the map data to be used like a third system. According to step c), the first data from the first sensor and the second data from the second sensor are preferably checked separately against the map data. This allows each to determine whether the first data from the first sensor and the second data from the second sensor match the map data. The extrapolation step according to step c) also allows the first and second data acquired according to steps a) and b) to be compared and checked for plausibility, independent of the map data.

[0032] As described above, the first and second sensors are preferably so-called environmental sensors capable of monitoring the vehicle's environment. The data acquired by such sensors is typically environmental data containing information about the vehicle's environment. In a vehicle, the environmental data is typically processed in conjunction with map data in order to locate the vehicle within the map data. Structures identified by the environmental sensors in the environmental data are assigned to structures contained in the map data. The environmental data also includes the distances and directions of the identified structures. Based on these distances and directions, the vehicle can be located within the map data. Therefore, a general purpose of processing the data from the environmental sensors is to regularly determine the vehicle's own position (own position) and, if necessary, its speed (own speed).

[0033] Furthermore, it is particularly advantageous if the extrapolation in step c) comprises a time extrapolation, wherein a time shift in the occurrence of a certain structure in the first data and the second data is taken into account.

[0034] Furthermore, it is particularly advantageous if the extrapolation in step c) comprises a spatial extrapolation, wherein a certain spatial shift of structures occurring in the first data and the second data is taken into account.

[0035] The temporal and spatial extrapolations are regularly coupled to one another by the movement of the motor vehicle, and this behavior is particularly preferably described using equations of motion (also called dynamic equations).

[0036] Processing environmental data often involves identifying so-called fixed points. These are distinctive, identifiable points within the map data. Due to the diversity of sensor types, these fixed points are regularly identified at different points in time using different sensor types. The example above describes a situation where a first sensor of a first sensor type detects a specific structural feature or structure at a first distance in front of the vehicle, while a second sensor of a second sensor type detects the same structural feature or structure at a second distance. This results in identification or detection at different points in time, given a specific vehicle speed.

[0037] However, if the dynamic equations that describe the movement of the vehicle are taken into account, it is possible to calculate the different times at which a structure is recognized or detected using different sensors of different sensor types. In the simplest case (when the motor vehicle is moving in a straight line at a constant speed), the dynamic equations only describe the vehicle's speed. However, in real driving situations, where the motor vehicle frequently brakes, accelerates, and / or turns, such dynamic equations can also include movement and rotational speeds in all spatial directions, including the corresponding accelerations. With the help of such dynamic equations, sensor data from a first sensor of a first sensor type and sensor data from a second sensor of a second sensor type can be extrapolated to each other. This is particularly feasible if the first measuring range of the first sensor and the second measuring range of the second sensor do not actually overlap, so that the first sensor and the second sensor do not detect the structure at the same time.

[0038] Furthermore, it is advantageous if structures identified along the route traveled by the motor vehicle in the first and / or second data are assigned to waypoints on the route and in step c) structures identified both in the first and second data are assigned to one another.

[0039] Furthermore, it is advantageous if the data processing in step d) includes checking the first data and / or the second data for errors.

[0040] Particularly preferably, the method for detecting errors in the first data and / or the second data is used if further data (for example third data from a third sensor of a third sensor type) are also processed in addition.

[0041] Furthermore, it is advantageous if the data processing in step d) includes the identification of properties of structures in the surroundings of the motor vehicle, which properties are identified by means of different sensor properties of the first sensor type and the second sensor type.

[0042] Furthermore, it is advantageous if the data processing in step d) includes determining the visible range of the GNSS satellite signals.

[0043] In particular, the method can also be used to determine the visibility range within which GNSS satellite signals are visible using different sensors of different sensor types.

[0044] The visual range here refers in particular to the area where there is an unobstructed line of sight from the motor vehicle to a global navigation satellite system satellite. The unobstructed visual range from the motor vehicle is described by the horizon. Above the horizon is the unobstructed visual range. GNSS satellites that have an unobstructed visual range from the GNSS satellite toward the motor vehicle are located within the unobstructed visual range. GNSS satellites that are located behind the horizon from the perspective of the motor vehicle are located outside the unobstructed visual range. In urban environments, in particular, the unobstructed visual range is often limited by structures on the side of the road (such as houses). The shape and design of these structures cannot be clearly identified using conventional environmental sensors, especially when the environmental sensors are oriented forward or backward. Using an environmental sensor oriented forward, it is usually only possible to detect the front side of the structure. And using an environmental sensor oriented backward, it is always only possible to detect the rear side of the structure. The method can be used to fuse data from sensors with different orientations. Therefore, the design of the structure in the vehicle environment can be fully identified.

[0045] Preferably, the method is also used to check which satellites are actually visible based on the visibility range detected by the environment sensors, so that there is an unobstructed line of sight to these satellites. If a satellite is visible, but it is clear from the visibility range detected by the environment sensors that it is not visible, it is possible that the environment detection using the environment sensors is faulty or that the GNSS satellite signal is faulty, because, for example, the GNSS signal may still be detected due to signal reflections but is therefore unsuitable for evaluation and position determination.

[0046] In this case, to identify whether the error lies in the environmental data or in the GNSS signal, it is preferred to additionally consider map data. If the structure identified in the map data can be tracked, the error lies in the GNSS signal. If the structure cannot be tracked based on the map data, the error lies in the environmental sensor or its data.

[0047] A device for data processing in a motor vehicle will also be described herein, comprising a processor configured to execute the method.

[0048] Furthermore, a computer program product will be described, comprising instructions which, when executed by a computer, cause the computer to perform the method.

[0049] Further described is a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The invention and its technical background will be explained in more detail below with reference to the accompanying drawings. The drawings show preferred embodiments of the invention to which the invention is not limited. It should be noted in particular that the drawings and, in particular, the proportions shown in the drawings are only schematic. The drawings show:

[0051] Figure 1 : Vehicles with different sensor types;

[0052] Figure 2 : comparison of the situation of the motor vehicle's route as seen from above with the situation of the motor vehicle further along the route;

[0053] Figure 3a : A three-dimensional diagram of the status of the motor vehicle route;

[0054] Figure 3b :according to Figure 3a a cross-sectional view of the condition;

[0055] Figure 4a :according to Figure 3a The visible range of GNSS satellite signals in different conditions;

[0056] Figure 4b :according to Figure 4a A cross-sectional view of the visible range; and

[0057] Figure 5 : Flowchart of the method. DETAILED DESCRIPTION

[0058] Figure 1 A motor vehicle 1 is shown with a plurality of sensors. The first sensor 2 and the second sensor 3 can be different types of sensors. The different types of sensors have detection ranges that deviate significantly from one another. Figure 1 In FIG, each sensor type is shown according to its detection range. For example, radar sensor 9, side camera 10, long-range radar sensor for the front area 11, lidar sensor 12, front camera 13, ultrasonic sensor 14 and infrared camera 15. Figure 1 The sensor types shown are not exhaustive. The method can be applied to further sensor types.

[0059] Figure 2Comparison of the situation at the vehicle's route 5 from above and the situation further along this route 5; in both situations, a structure 4 can be seen on one side of route 5. Vehicle 1 has a first sensor 2 with a first monitoring range and a second sensor 3 with a second monitoring range. For example, first sensor 2 is configured to detect a structure at a distance of 15 meters in front of vehicle 1. For example, second sensor 3 is configured to detect a structure at a distance of 5 meters in front of vehicle 1. In the scenario shown on the left, structure 4, shown here as an example, is detected by first sensor 2. In the scenario shown on the right, structure 4 is detected by second sensor 3. The scenarios shown on the left and right are correlated by the dynamics of the vehicle's movement. According to the extrapolation in step c), the first data from first sensor 2 and the second data from second sensor 3 are correlated and extrapolated accordingly, resulting in an assignment / comparison of these data. Based on the dynamics of vehicle 1's movement, the structure can be assigned to a waypoint on vehicle 1's route.

[0060] Figure 3a and Figure 3b In three-dimensional perspective ( Figure 3a and Figure 3b A cross-sectional view shows a vehicle with structure 4 along route 5. Sensors oriented toward the front of the vehicle can only detect one side of structure 4 in the surroundings of route 5. Environmental sensors oriented toward the front (or rear) cannot fully detect the three-dimensionality of the vehicle's surroundings. As proposed herein, multiple sensors can better detect this three-dimensionality of the surroundings by extrapolating the data from different sensors and performing data processing.

[0061] Figure 4a and 4b Shown from above ( Figure 4a ) and cross-sectional view ( Figure 4b ) an unobstructed visible range 7 of GNSS satellite signals 8 in the surroundings of the motor vehicle 1 . Figure 4a and Figure 4b The situation shown in Figure 3a and Figure 3b The situation shown in is a model. Figure 4a As can be seen in the figure, structures 4 (houses) on the left and right sides of the roadway narrow the visible range 7. Along the route, visible range 7 is either unrestricted or less restricted. The method described allows data from different sensors to be processed together to determine such visible range 7.

[0062] Figure 5The flowchart of the method is shown. It can be seen that according to steps a) and b) the sensor data are acquired and then the method step c) is carried out, which preferably includes at least one of the three sub-steps i), ii) and iii). In step d), joint data processing is carried out.

Claims

1. A method for processing sensor data in a motor vehicle (1), comprising at least the following steps: a) detecting first data using a first sensor (2) of a first sensor type; b) detecting second data using a second sensor (3) of a second sensor type; c) extrapolating the first data detected according to step a) and / or the second data detected according to step b) to create extrapolated data, the extrapolated data enabling comparability of the first data and the second data; as well as d) performing common data processing on the first data and / or the second data using the extrapolated data.

2. The method according to claim 1, wherein the first sensor (2) used in step a) and the second sensor (3) used in step b) are each one of the sensor types included in the following list: - LiDAR; -radar; -Front camera; - Infrared camera; - Ultrasonic sensor; or -Interior space camera.

3. The method according to any one of the preceding claims, wherein the extrapolation in step c) comprises at least one of the following sub-steps: i. Obtain at least one scaling factor for scaling the second data and the first data relative to each other; ii. obtaining a shift coefficient, the shift coefficient describing the shift of the second data and the first data relative to each other; iii. Applying at least one scaling factor and / or at least one shifting factor to said first data and / or said second data to create extrapolated data. 4 . The method according to claim 1 , wherein the extrapolation in step c) is performed with the aid of map data.

5. The method according to claim 1, wherein the extrapolation in step c) comprises a time extrapolation in which a time shift of the occurrence of a certain structure (4) in the first data and the second data is taken into account.

6. The method according to any of the preceding claims, wherein the extrapolation in step c) comprises a spatial extrapolation in which a spatial shift of a determined structure (4) occurring in the first data and the second data is taken into account.

7. A method according to any of the preceding claims, wherein structures (4) identified in the first data and / or second data along a route (5) travelled by the motor vehicle (1) are assigned to waypoints (6) on the route, and structures (4) identified both in the first data and in the second data are assigned to one another in step c).

8. The method according to any of the preceding claims, wherein the data processing in step d) comprises checking the first data and / or the second data for errors.

9. A method according to any of the preceding claims, wherein the data processing in step d) includes identifying characteristics of structures (4) in the surroundings of the motor vehicle (1), said characteristics being identified by means of different sensor characteristics of the first sensor type and the second sensor type.

10. The method according to any of the preceding claims, wherein the data processing in step d) comprises determining a visible range (7) for a GNSS satellite signal (8).

11. A device for data processing of a motor vehicle (1), comprising a processor configured such that the processor executes the method according to any one of claims 1 to 10.

12. A computer program product comprising instructions, which, when the computer program product is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images