Method, environmental sensing system and computer program for providing environmental data

Abstract

Method for providing environmental data describing the environment of a vehicle (1), wherein sensor data describing the environment are acquired by at least one sensor (12) of the vehicle (1), and wherein the at least one sensor (12) provides the sensor data with a defined resolution, a defined rate, and in a defined coordinate system, and environmental data based on the sensor data are provided by transmission to other road users (6) and / or to an infrastructure facility (100) as receivers, wherein the environmental data are obtained by converting the sensor data, in which at least one or more conversions are performed, which are selected from i) increasing or decreasing the resolution, ii) increasing or decreasing the rate, and iii) performing a deconstruction of the sensor data, which abstracts and thereby anonymizes the sensor data, characterized in that the provided environment data is cryptographically signed so that its integrity and origin can be traced and that a target format is individually specified by the receiver, whereby the receiver requests environment data from the vehicle (1) in the specified target format.

Description

[0001] The invention relates to a method for providing environmental data describing the surroundings of a vehicle, wherein sensor data describing the surroundings are acquired by at least one sensor of the vehicle, and wherein the sensor delivers the sensor data with a defined resolution and a defined rate, and environmental data based on the sensor data are made available by transmission to other road users and / or to an infrastructure facility as a receiver. Further aspects of the invention relate to an environmental sensing system for a vehicle and a computer program configured to execute the method. State of the art

[0002] Modern vehicles increasingly feature driver assistance systems that support the driver in performing various driving maneuvers. Vehicles with automated driving functions are also known, allowing autonomous driving without driver intervention, at least under certain conditions.

[0003] These driver assistance systems and autonomous driving functions rely on precise data about the vehicle's surroundings. To this end, the vehicles are equipped with sensors that collect this data. These sensors are used, for example, to locate objects around the autonomous vehicle. Based on this information, an autonomous driving function can plan a trajectory and control vehicle actuators within the vehicle. These actuators then influence the lateral control (steering) and / or longitudinal control (acceleration and braking).

[0004] The processing of sensor data describing the environment is carried out, for example, using intelligent object recognition algorithms, which can be implemented as neural networks or artificial intelligence. These object recognition algorithms typically use sensor data with a defined resolution and data rate. The processing algorithms within an autonomous vehicle are designed to handle the resolutions of the various sensors arranged on the vehicle, or the evaluation algorithms are parameterized or, in the case of neural networks, trained on the corresponding resolutions and data rates of the sensor data from the different sensors of the respective vehicle.

[0005] Furthermore, it is known that vehicles can exchange data with each other (car-to-car communication) or with an infrastructure facility (car-to-infrastructure communication). This exchanged data can include, in particular, environmental data describing the vehicle's surroundings.

[0006] DE 10 2017 200 654 A1 describes a method for providing sensor-based vehicle functions. In this method, the sensors are connected via an integration component and are designed independently of other vehicle functions. The integration component reads sensor data and converts it into environmental data or, more generally, into status data. Additionally, external devices can be connected, for example, via Car2X communication or vehicle-to-vehicle communication. For this purpose, the integration component can transmit at least some of its status data to at least one externally executed, outsourced vehicle function. Vehicle functions can retrieve status data and sensor data from the integration component in the same way, so that it is irrelevant to the vehicle functions whether the data originates from a vehicle-integrated sensor or not.

[0007] DE 10 2017 223 634 A1 describes a method for determining road conditions by evaluating sensor data from a road vehicle. In one embodiment, a sensor unit of the vehicle sends sensor data to a central database. Standardized raw data is transmitted.

[0008] DE 10 2016 225 437 A1 describes a device, a method, and a computer program for a vehicle for transmitting an accident report to an emergency call center. The device includes at least one sensor interface configured to receive sensor data from the vehicle's internal and external sensors. The device also includes at least one communication interface configured to communicate with the emergency call center. This allows for the transmission of more detailed data based on the sensor data to the emergency call center, enabling more precise planning of rescue operations by the emergency services.

[0009] DE 10 2017 209 195 A1 describes a method for documenting the driving operation of a motor vehicle, comprising the steps of: determining a movement profile that describes the locations of the motor vehicle during a recording interval, and determining recording information that describes for several external cameras whether each of these is a vehicle-capturing camera in whose detection range the motor vehicle is located in at least one sub-interval of the detection interval, depending on the movement profile and the respective detection range.

[0010] US 2016 / 0162743A1 describes a vehicle environment sensing system comprising a camera and a non-imaging sensor. The camera and the non-imaging sensor are positioned on the vehicle such that the camera's field of view overlaps at least partially with the non-imaging sensor's detection field. The image data captured by the camera and the sensor data captured by the non-imaging sensor are processed to determine the vehicle's driving situation. This involves determining Kalman filter parameters associated with the determined driving situation. Using these Kalman filter parameters, a Kalman filter fusion can be determined. The determined Kalman filter fusion can then be applied to the captured image and sensor data to identify an object present in the overlapping area.

[0011] US 2020 / 0130570A1 describes techniques for implementing a so-called Self-Adaptive Multi-Resolution Digital Signage (SAMDP) system, which enables the shared use of vehicle condition information via a dynamically generated binary two-dimensional multispectral pattern. The binary multispectral 2D pattern adjusts its resolution to adapt to the visual perception process while communicating vehicle condition information without relying on wireless communication (i.e., V2V, V2X, etc.).

[0012] US 2020 / 0226790A1 discloses a sensor calibrator comprising one or more processors configured to receive sensor data representing a calibration pattern acquired by a sensor during a period of relative motion between the sensor and the calibration pattern. A calibration setting is determined from the set of images, and a calibration instruction is sent to calibrate the sensor according to the determined calibration setting.

[0013] When exchanging sensor data from different sensors between vehicles or with infrastructure, a problem arises: the evaluation algorithms within a vehicle are often limited to specific resolutions and / or data rates. This makes using the exchanged sensor data difficult or even impossible. Furthermore, data privacy concerns can arise, particularly when exchanging image data, especially if people or license plates are identifiable in the images. Such privacy concerns can prevent this sensor data from being exchanged and thus render it unavailable for evaluation by other vehicles or infrastructure. Disclosure of the invention

[0014] A method for providing environmental data describing a vehicle's surroundings is proposed. This method involves acquiring sensor data describing the environment using at least one sensor on the vehicle. This sensor delivers the sensor data with a defined resolution, rate, and coordinate system. The resulting environmental data, based on this sensor data, is then transmitted to other road users and / or an infrastructure receiver. Furthermore, the environmental data is obtained through a conversion of the sensor data, involving at least one or more transformations selected from [various parameters]. i) increasing or decreasing the resolution, ii) increasing or decreasing the rate, and iii) performing a deconstruction of the sensor data, which abstracts and thereby anonymizes the sensor data.

[0015] Preferably, the sensor data is also converted into the vehicle coordinate system of the receiver or into a global coordinate system (world coordinate system).

[0016] The at least one sensor could be, for example, a video camera, a microphone, a camera, an ultrasonic sensor, a LiDAR sensor, or a radar sensor. The vehicle may, in particular, have more than one sensor, and may have several identical sensors and / or sensors of different types.

[0017] Depending on the type of sensor (at least one), the sensor data can be of various types, such as image data representing pictures, videos (in the form of a sequence of images or image data), waveforms representing sound recordings, three-dimensional point clouds, and the like. Each of these data types has a resolution, which indicates the precision of the data. Furthermore, the data types have a rate, which can correspond, in particular, to the repetition rate of the measurement performed by the respective sensor.

[0018] For example, resolution refers to the image resolution or color depth of image data, the bit depth of samples of a waveform, the numerical precision of a discrete sensor value, the numerical precision of a 3D model or a point cloud.

[0019] The rate can refer, for example, to the frame rate of a stream of image data, the sampling rate of a waveform, the repetition rate of discrete sensor values, the update rate of a 3D model or a point cloud.

[0020] For the evaluation of sensor data, algorithms and / or AI methods are preferably used that, for example, extract specific features from the sensor data in order to identify objects in the vehicle's vicinity. These algorithms and / or AI methods typically expect the sensor data to be in a specific input format, particularly a specific resolution and / or rate. To make the sensor data processable for other road users and / or infrastructure, the invention provides for converting the sensor data into a predetermined target format and making the resulting environmental data available for use by others.

[0021] The other road users are primarily other vehicles located nearby. By transmitting environmental data based on their own sensor data, these other vehicles can gain a different perspective on the current traffic situation. This allows them, for example, to obtain information about areas of the environment that cannot be seen by their own sensors, such as those obscured by objects. Each road user can process the received environmental data using their own evaluation algorithms and / or AI methods, ensuring that the functionality of all vehicle-specific assistance and autonomous driving functions is fully available, regardless of whether their own sensor data or received environmental data is used.

[0022] The infrastructure could be, for example, a central server, particularly a cloud server. This infrastructure could make the received environmental data available to other road users and / or perform its own analyses of the environmental data. For instance, the infrastructure could use an algorithm or AI method to determine traffic density or identify available parking spaces based on the received environmental data.

[0023] This derived information can then be made available to other road users.

[0024] The target format for the conversion is individually specified by the recipient. The recipient is expected to request environmental data from the vehicle in the specified target format, thus transmitting information about the desired target format with the request.

[0025] Wireless communication methods are preferred for transmitting environmental data. This can involve, for example, establishing a direct connection between two road users or connecting to a communication network such as the internet. Suitable communication methods include, in particular, mobile communication technologies such as GSM, UMTS, LTE, and 5G, as well as wireless networking technologies such as Bluetooth or Wi-Fi.

[0026] Preferably, one or more algorithms, machine learning methods or artificial intelligence methods, as well as combinations thereof, are used to perform the conversion, wherein different algorithms, machine learning methods or artificial intelligence methods are used for different types of sensor data and / or for different conversions, or one algorithm, machine learning method and / or artificial intelligence method performs several conversions for one or more types of sensor data.

[0027] When using different algorithms for converting the sensor data, for example, one algorithm can convert sensor data to a uniform resolution, while a downstream algorithm adjusts the rate, and yet another downstream algorithm calculates a deconstruction of the sensor data.

[0028] However, it may also be intended that only certain parameters of the sensor data are changed during the conversion. For example, it may be intended that only the rate or only the resolution is adjusted during the conversion.

[0029] Preferably, the rate is increased using extrapolation or interpolation algorithms from two adjacent data points of the sensor data, whereby intermediate points are calculated. To decrease the rate, preferably the data points of the sensor data to be transmitted are selected according to the target rate and / or intermediate points are calculated from two adjacent data points of the sensor data using extrapolation or interpolation algorithms.

[0030] If the sensor data is image data, a neural network is preferably used to change the image resolution of the image data and / or to deconstruct the image data.

[0031] A neural network can be used to calculate changes in the resolution or image resolution of image data. For example, if the resolution of the image data is reduced, certain pixels can be omitted, and the resulting image can be smoothed. Conversely, if the resolution is increased, the neural network is preferably used to extrapolate missing pixels from the existing pixels in the image data.

[0032] The use of neural networks to change the resolution of image data is well known to those skilled in the art. An example of this is described in the publication by Mehdi SM Sajjadi, Bernhard Schölkopf, and Michael Hirsch, "EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis", arXiv:1612.07919v2.

[0033] A change in the rate, or in the case of video, the number of frames over time, is preferably also calculated using a neural network. For example, sensor data is only transmitted to other road users or infrastructure after a delay of one cycle. One cycle corresponds to the rate at which the environmental sensor delivers data. In the case of image data as sensor data, increasing the rate (frame rate) by a factor of two, for example, uses a neural network to extrapolate an image from two consecutive images. The neural network identifies the differences between the pixels in the two images and, based on this, uses an extrapolation algorithm to calculate the missing image between the two. All three images are then transmitted.In this way, image data in this example can be generated from the existing image data at twice the rate and then transmitted to other road users or an infrastructure facility. The factor of two is merely an example. The more intelligent the algorithm, the more image data can be generated stepwise from the two adjacent images, depending on the required frame rate at the receiver. Reducing the frame rate using the neural network is also conceivable. In this case, for example, only every second image is transmitted to a receiver. If the frame rate is to be reduced by a third, for example, two or three adjacent images are used, and based on this, the pixel content of each pixel of the image is calculated for one-third of the frame rate using an extrapolation algorithm within the neural network.

[0034] In sensor data deconstruction, input sensor data is processed by a neural network. The neural network converts, for example, image data into a machine-readable format that only a machine can understand. From this machine-readable format, a subsequent neural network can then extract specific features. These extracted features are suitable, for example, for describing the positions of objects and / or classifying the type of object, allowing for differentiation between vehicles, trees, and pedestrians. However, the machine-readable format obtained through deconstruction, or the extracted features, do not allow for the identification of individuals or the reading of license plates. Furthermore, it is not possible to reconstruct the original image data using the deconstructed data or the extracted features.In this way, for example, it is possible to anonymize image data and transmit it as environment data without data protection concerns.

[0035] The vehicle's sensor data is typically first acquired with reference to a local vehicle coordinate system, which is defined as a coordinate system based, for example, on a sensor position or a reference point of the vehicle. Before transmission, the vehicle's sensor data is preferably transformed into a vehicle coordinate system of the receiver or into a global coordinate system. This transformation can be performed, for example, using the received position and orientation of another road user and a known position of the vehicle. Furthermore, it may be possible to convert the defined coordinate system of the sensor data into a global coordinate system. Such a global coordinate system is preferably globally unique and can be processed directly by an infrastructure device, for example.Other road users may, if necessary, transform the received environmental data into their own local coordinate system via a further transformation.

[0036] Alternatively, it may be possible to transmit the position and orientation of the vehicle and, if applicable, the position and orientation of the sensor together with the environmental data to the receiver, so that it can perform a corresponding transformation of the coordinate system.

[0037] The vehicle's position and, if applicable, its orientation can be determined, for example, via receivers for signals from navigation satellites. If necessary, signals from the vehicle's sensors, such as an odometry sensor or a compass, can be taken into account to assist with localization.

[0038] The provided environment data is cryptographically signed, so that its integrity and origin can be traced.

[0039] For this purpose, the environmental data can be assigned a uniform and unique hash value before being transmitted to a recipient. This allows the environmental data to be assigned to a specific road user at any time when stored on another road user or infrastructure device. The hash value of the environmental data is calculated using a unique key that corresponds to exactly one vehicle (private key). The hash value of the environmental data is then verified outside the vehicle using a public key.

[0040] Another aspect of the invention is the provision of an environment sensing system for a vehicle. The environment sensing system comprises at least one sensor with which sensor data describing the environment can be acquired, as well as a communication device for transmitting environment data to other road users or an infrastructure facility, and a control unit.

[0041] The environmental sensing system is configured to execute one of the procedures described herein. Therefore, characteristics described within one of the procedures apply accordingly to the environmental sensing system, and conversely, characteristics described within one of the procedures apply accordingly to the environmental sensing system.

[0042] Preferably, the at least one sensor is a video camera, a microphone, a stereo camera, an ultrasonic sensor, a LiDAR sensor, a radar sensor, or a combination of several of these sensors. In particular, several identical sensors and / or combinations of several different types of sensors can be used.

[0043] The communication device is preferably set up for wireless communication, implementing, for example, mobile communication technologies such as GSM, UMTS, LTE, 5G and / or wireless networking methods such as Bluetooth or WiFi.

[0044] The control unit may in particular include a microcontroller or a processor that is programmed accordingly for the execution of the procedure and is specifically designed for performing the conversion of the sensor data.

[0045] The invention further proposes a computer program according to which the steps of the methods described herein, to be performed by the vehicle, are carried out when the computer program is executed on a programmable computer device. The computer program may, for example, be a module for implementing a driver assistance system or a subsystem thereof in a vehicle, or an application for driver assistance functions, which can be executed, for example, on a smartphone or tablet. The computer program can be stored on a machine-readable storage medium, such as a permanent or rewritable storage medium, or associated with a computer device, or on a removable CD-ROM, DVD, Blu-ray Disc, or USB flash drive. Additionally or alternatively, the computer program can be made available for download on a computer device, such as a server.via a data network such as the Internet or a communication connection such as a telephone line or a wireless connection. Advantages of the invention

[0046] In the method according to the invention, sensor data from a sensor used to detect the environment of a vehicle are converted into a predefined format for provision as environmental data before being transmitted to other road users or an infrastructure facility. This allows the environmental data to be evaluated or further processed directly on the receiver side without requiring any further conversion. This accelerates the evaluation of the environmental data on the receiver side. The receiver can specify the target format for the conversion in order to obtain the environmental data in precisely the format required for further processing.

[0047] During conversion, it is advantageous to deconstruct sensor data, particularly image data. This ensures that the privacy of pedestrians, for example, is protected during subsequent processing by the recipient. Even under strict data protection regulations, this allows image data to be transmitted as environmental data to other road users or infrastructure without data privacy concerns, thus providing recipients with more comprehensive information about the environment. This additional environmental data significantly enhances road user safety, as it provides a more precise and complete representation of the surroundings.

[0048] For recipients of environmental data, the algorithms or AI methods used for processing can be designed particularly simply, since they always receive the environmental data in the expected format. Brief description of the drawings

[0049] Embodiments of the invention are explained in more detail with reference to the drawings and the following description.

[0050] The single figure schematically shows a road intersection where three vehicles meet, two of which have an environment detection system according to the invention.

[0051] The figure only schematically represents the subject of the invention. Embodiments of the invention

[0052] The Fig. Figure 1 schematically shows a road intersection 2. A vehicle 1 and another road user 6 are each equipped with an environmental sensing system 10. In the simplified example of the Fig. Each sensor 10 comprises a sensor 12, a control unit 14, and a communication device 16. Naturally, the environmental sensing system 10 can also have multiple sensors 12. It is conceivable to design the control unit 14 and, if applicable, the communication device 16 together with the sensor 12 as an integrated unit.

[0053] The sensors 12 each have a field of view 13, 13' within which they can detect objects. The sensor 12 of vehicle 1 can detect an oncoming vehicle 8 because it is within the field of view 13 of sensor 12 of vehicle 1. The sensor 12 of the other road user 6 cannot detect the oncoming vehicle 8 because it is within the field of view 13 of sensor 12 of vehicle 1. Fig. In the situation shown, a house 4 limits the field of view 13' of the sensor 12 of the other road user 6.

[0054] Thus, the assistance systems or autonomous driving functions of the other road user 6 initially have no knowledge that the oncoming vehicle 8 is also approaching the intersection.

[0055] However, vehicle 1 can provide environmental data, based on sensor data from sensor 12 of vehicle 1, to the other road user 6 via the environmental sensing system 10. For this purpose, sensor data from sensor 12 of vehicle 1 is converted into standardized environmental data, whereby, for example, the rate and resolution are changed during the conversion. This standardized environmental data is then transmitted to the environmental sensing system 10 of the other road user 6 via the communication device 16.

[0056] The sensor data from vehicle 1 are preferably transformed into the vehicle coordinate system of the other road user 6 before transmission. For this purpose, the control unit 14 can perform a corresponding transformation using a known position of vehicle 1 and a received position and orientation of the other road user 6. Alternatively, the position and orientation of vehicle 1, and optionally the position and orientation of sensor 12, along with the environmental data, can be transmitted to the other road user 6, so that the control unit 14 of the other road user 6 can perform a corresponding transformation. Vehicle 1 and the other road user 6 can determine their respective positions relative to a global coordinate system, for example, via signals from navigation satellites.

[0057] Assistance systems or an autonomous driving function of the other road user 6 can now process the received environmental data as if it had been detected by a sensor 12 of the other road user 6 itself. Thus, the other road user 6 can take the oncoming vehicle 8 into account when planning a trajectory before it enters the field of view 13' of the sensor 12 of the other road user 6. Driving safety is therefore increased.

[0058] Furthermore, the environmental sensing systems 10 of the vehicle 1 and / or the other road user 6 can provide environmental data of an infrastructure facility 100.

[0059] Infrastructure facility 100 could, for example, be a central server located in the Fig.The example outlined in Figure 1 includes a communication unit 102 and a computing unit 104. The infrastructure unit 100 can perform its own analyses of the received environmental data. For example, the infrastructure unit 100 can use an algorithm or an AI method to determine traffic density or identify available parking spaces based on the received environmental data. This derived information can then be made available to vehicle 1 and other road users 6.

[0060] The invention is not limited to the embodiments described here and the aspects highlighted therein. Rather, within the scope specified by the claims, a multitude of modifications are possible that fall within the bounds of what is considered skilled in the art.

Claims

[1] Method for providing environmental data describing the environment of a vehicle (1), wherein sensor data describing the environment are acquired by at least one sensor (12) of the vehicle (1), and wherein the at least one sensor (12) provides the sensor data with a defined resolution, a defined rate and in a defined coordinate system, and environmental data based on the sensor data are provided by transmission to other road users (6) and / or to an infrastructure facility (100) as receivers, wherein the environmental data are obtained by converting the sensor data, in which at least one or more conversions are carried out, which are selected from i) increasing or decreasing the resolution, ii) increasing or decreasing the rate, and iii) performing a deconstruction of the sensor data, which abstracts and thereby anonymizes the sensor data,characterized by that the provided environment data is cryptographically signed so that its integrity and origin can be traced and that a target format is individually specified by the receiver, whereby the receiver requests environment data from the vehicle (1) in the specified target format. [2] Method according to claim 1, characterized by , that additionally, the sensor data is converted into the vehicle coordinate system of the receiver or into a global coordinate system. [3] Method according to one of claims 1 or 2, characterized by , that the resolution is an image resolution or color depth of image data, the bit depth of a waveform, the numerical precision of a discrete sensor value, the numerical precision of a 3D model and / or that the rate is a frame rate of a stream of image data, a sampling rate of a waveform, a repetition rate of discrete sensor values, the update rate of a 3D model. [4] Method according to any one of claims 1 to 3, characterized by , that one or more algorithms, machine learning methods or artificial intelligences, as well as combinations thereof, are used to perform the conversion, wherein different algorithms, machine learning methods or artificial intelligences are used for different types of sensor data and / or for different conversions, or one algorithm, machine learning method and / or artificial intelligence performs several conversions for one or more types of sensor data. [5] Method according to claim 4, wherein to increase the rate intermediate points are calculated from two adjacent data points of the sensor data using extrapolation algorithms or interpolation algorithms, and to decrease the rate a selection of the data points of the sensor data to be transmitted is made according to the target rate specified and / or intermediate points are calculated from two adjacent data points of the sensor data using extrapolation algorithms or interpolation algorithms. [6] Method according to claim 4 or 5, wherein a neural network is used to change the resolution of image data and / or to deconstruct image data. [7] Environment sensing system (10) for a vehicle (1), wherein the environment sensing system (10) comprises at least one sensor (12) capable of acquiring sensor data describing the environment, as well as a communication device (16) for transmitting environment data to other road users (6) or an infrastructure device (100) and a control unit (14), characterized by , that the environmental sensing system (10) is configured to perform one of the methods according to one of claims 1 to 6. [8] Environment detection system (10) according to claim 7, characterized by , that the at least one sensor (12) is a video camera, a microphone, a stereo camera, an ultrasonic sensor, a LiDAR sensor, a radar sensor or a combination of several of these sensors. [9] Computer program that performs the steps of the method according to any one of claims 1 to 6 to be carried out by the vehicle (1) when running on a computer.

Images