Checking sensor data from a driver assistance system

Abstract

The invention relates to a method for checking sensor data (13.1, 13.2) for implementing a driver assistance system (16) of a vehicle, wherein sensor data (13.1, 13.2) originating from at least one sensor (15, 20), which contain information about an external environment (14) of the vehicle (1), are checked by means of a decentralized computer network (24). The sensor data (13.1, 13.2) are intended to be used in a driver assistance system (16) of a vehicle (1) to generate a representation of the external environment (14) of the vehicle (1).

Description

[0001] The invention relates to the verification of sensor data of a driver assistance system. In this context, a method for verifying sensor data for the implementation of a driver assistance system of a vehicle is claimed.

[0002] Modern vehicle driver assistance systems rely on multiple sensors such as cameras, Li-DAR, ultrasonic sensors, and radar to obtain data about the vehicle's surroundings. This data is then processed and fused to create a comprehensive picture of the environment. This aids in various driver assistance functions, such as lane keeping, adaptive cruise control, and collision avoidance. The integrity, authenticity, and availability of sensor data are critical to the performance of these systems. Conventional data management techniques can be vulnerable to tampering or falsification, which can lead to inaccurate or incomplete data fusion and potentially compromise the safety of the driver assistance system. This issue is even more critical when considering external sensor data sources such as C2X or traffic infrastructure.The reliability of these external data sources is difficult to verify.

[0003] One object of the present invention can be seen in improving the integrity, authenticity, and availability of sensor data for a driver assistance system of a vehicle. This object is achieved by the subject matter of the independent patent claims. Advantageous embodiments are the subject matter of the subclaims, the following description, and the figures.

[0004] According to the present invention, the above-mentioned object is proposed to be achieved by using a decentralized computer network. For example, distributed ledger technology can be used, which utilizes a blockchain to verify, manage, and merge sensor data in driver assistance systems. In particular, a consensus algorithm can be used, which is explained in more detail below in connection with an embodiment of the method according to the invention. The invention can use a decentralized, secure, and tamper-proof data management approach that ensures the integrity, authenticity, and availability of sensor data and ultimately improves the performance and security of driver assistance systems.

[0005] In this sense, according to one aspect of the invention, a method for checking sensor data for the implementation of a driver assistance system of a vehicle is provided. The vehicle is in particular a motor vehicle that is powered by an engine, for example an automobile (e.g. a passenger car weighing less than 3.5 t), bus or truck (e.g. weighing more than 3.5 t). The vehicle can, for example, belong to a vehicle fleet. The vehicle can be controlled by a driver, possibly assisted by a driver assistance system. However, the vehicle can also, for example, be remotely controlled and / or (partially) autonomously.

[0006] Sensor data originating from at least one sensor, which contains information about the vehicle's external environment, can be examined using a decentralized computer network. In a computer network, several computers can communicate with each other. The computer network can be implemented on the Internet. A decentralized or peer-to-peer computer network can be understood as a computer-to-computer connection in which all computers are interconnected and, in particular, have equal rights. The contrast to a peer-to-peer connection is the client-server model, in which a central server offers a service and a client uses this service. The sensor data is intended to be used in a vehicle's driver assistance system to generate a representation of the vehicle's external environment.The external representation may in particular be of a visual nature and be shown on a display inside the motor vehicle.

[0007] In particular, the sensor data can be collected from multiple sensors. The sensors can be, without limitation, cameras, LiDAR, ultrasonic sensors, or radar. For example, a camera can generate camera data and a radar can generate radar data from the vehicle's external environment. The camera data and the radar data can be collected to provide different and complementary data for a representation of the vehicle's external environment.

[0008] The sensor data can be encrypted to ensure its confidentiality and integrity. In this sense, according to a further embodiment, the sensor data is first encrypted into encrypted sensor data and stored in a data block. The encrypted sensor data is then verified using the decentralized computer network.

[0009] As mentioned above, the decentralized computer network can comprise multiple computers, whereby checking the sensor data involves the execution of a consensus algorithm by the computers of the decentralized computer network. In particular, virtual nodes can be used as validators that validate the input variables according to the defined consensus algorithm. A consensus algorithm can be understood as a computer-implemented mechanism used in distributed systems, including distributed ledger technology (DLT) or blockchain systems, to reach agreement on which transactions are included in the shared, decentralized database and in what order. The purpose of the consensus algorithm is to ensure that all participants in the network reliably and uniformly agree on which state of the ledger or blockchain is considered valid.

[0010] By executing the consensus algorithm, one of the computers in the decentralized computer network can first be selected to verify the sensor data. The selected computer can then verify the sensor data. The verifying computer in the decentralized computer network can be selected using one of the methods described below, for example. Proof of Work (PoW) can be used, in which participants use computing power to solve mathematical problems and add new blocks to the blockchain. Furthermore, Proof of Stake (PoS) can be used. With this approach, the selection of the block producer depends on the number of cryptocurrency units that the participant owns and is willing to stake.One option is Delegated Proof of Stake (DPoS), which works similarly to PoS but requires a limited number of selected block producers to perform transaction validation. Byzantine Fault Tolerance (BFT) is also possible. BFT consensus algorithms aim to ensure the integrity of the system even in the face of faulty or malicious participants by requiring a certain number of consents.

[0011] The consensus algorithm can encompass various aspects and lead to a validation result value, on the basis of which a decision is made as to whether the sensor data is validated or not. One of these aspects can be the reputation of a data source. The reputation can be based on the history of the at least one sensor, e.g., how often the sensor(s) have provided sensor data in the past. The quality of the data provided (quality, timeliness, etc.) can also be included as a reputation criterion in the calculation of the validation result value. Furthermore, it can be considered how compatible or up-to-date the data from the at least one sensor is (e.g., the software version of the data source). It can also be taken into account whether and, if so, how many manipulation or hacker attacks the at least one sensor has experienced and the extent of the damage.The integrity of the data provided by the at least one sensor can also represent a meaningful reputation criterion. In this sense, according to a further embodiment, checking the sensor data includes determining a reputation of the at least one sensor, wherein a check result value is determined based on the reputation of the at least one sensor.

[0012] Additionally or alternatively, a physical property or type of the at least one sensor (data source) can be detected by an aspect of the consensus algorithm, e.g., if the data source is a camera or a C2X device. In this sense, according to a further embodiment, checking the sensor data includes determining a type of the at least one sensor, wherein the check result value is determined based on the type of sensor.

[0013] Furthermore, one aspect of the consensus algorithm may include a data tamper check, e.g., by verifying a hash of the data received from the data source. In this sense, according to a further embodiment, verifying the sensor data includes checking whether the sensor data has been tampered with in a tamper check, and determining the check result value based on the result of the tamper check. Additionally or alternatively, a data validation duration, a data timestamp, or the like may be included in the consensus algorithm to determine the check result value.

[0014] A previously validated and verified block of data can be stored in a blockchain, which serves as an immutable, secure, and transparent ledger for the sensor data. A blockchain can be understood as a decentralized, distributed database that stores transaction data in the form of blocks and links them together using cryptography. Each block contains, for example, a list of data (e.g., transaction data or, in this case, encrypted sensor data). The block can also contain a timestamp and / or a cryptographic hash of the previous block. This chaining mechanism ensures the immutability and integrity of the data stored in the blockchain. Blockchain technology enables multiple parties in a network to track and verify transactions securely and transparently, without the need for a central authority.The decentralized nature of the blockchain makes it resistant to fraud and tampering. Every transaction added to the blockchain requires the approval of the network according to the established consensus algorithm, which ensures data security. In particular, the consensus algorithm can contain a predefined range of values. If the verification result value lies within this predefined range of values, it leads to a positive verification result, with the consequence that the sensor data is stored in a data block of the blockchain. If the verification result value lies outside the predefined range of values, it leads to a negative verification result, with the consequence that the sensor data is not stored in a data block of the blockchain.In this sense, according to a further embodiment, the sensor data is stored in a data block of a blockchain if the test result value lies within a predetermined value range. This embodiment has the advantage that a history of events can be made more traceable. If something happened while the vehicle was driving, for example, if an accident occurred or an error occurred, the external environment of the vehicle can be reconstructed using the sensor data stored in the blockchain. This can, for example, facilitate the clarification of a situation that led to an accident.

[0015] The sensor data from the blockchain can be decrypted and merged. Based on the fused, decrypted sensor data, a comprehensive understanding of the vehicle's environment is generated.

[0016] Data fusion is the process of combining information or data from different sensors or data sources to create a more comprehensive, accurate, and coherent representation of a specific physical or logical condition or event. In the case of vehicles that collect sensor data from cameras, radar systems, and other sensors, data fusion refers to the integration and processing of this information to generate a comprehensive picture of the vehicle's external environment. Data fusion in automotive engineering plays a crucial role, helping to improve vehicle safety, navigation, and efficiency. By combining data from different sensors, vehicles can better perceive what is happening around them and make informed decisions, for example, in collision avoidance, parking, or autonomous driving.There are various techniques and algorithms for data fusion, including Bayesian filters, Kalman filters, and neural networks, depending on the requirements and the available sensors. Data fusion makes it possible to leverage the strengths of different sensors while compensating for weaknesses or uncertainties to create a reliable and coherent perception of a vehicle's surroundings. In this sense, according to a further embodiment, the sensor data stored in the data block of the blockchain is decrypted into decrypted sensor data. The decrypted sensor data can be fused into fused sensor data. The representation of the vehicle's external environment can be generated based on the decrypted and fused sensor data.

[0017] The consensus algorithm, or the computers that execute it, can be implemented on hardware in the vehicle. Additionally or alternatively, the consensus algorithm can also be implemented outside the motor vehicle. In this way, a comprehensive consensus algorithm can be implemented for the entire traffic system. In this sense, according to a further embodiment, the computers of the decentralized computer network are arranged inside the vehicle and / or outside the vehicle.

[0018] The above-described checking of the sensor data enables various driver assistance functions. The fused, decrypted sensor data can therefore be used in various driver assistance systems and functions. In this sense, according to a further embodiment, a driver assistance system is implemented based on the decrypted and fused sensor data.

[0019] In the following, embodiments of the invention are explained in more detail with reference to the schematic drawing, wherein identical or similar elements are provided with the same reference numerals. Fig. 1 a vehicle with several sensors and a driver assistance system for controlling the vehicle, Fig. 2 a decentralized computer network for checking sensor data generated by the sensors according to Fig. 1 are generated, and Fig. 3 shows a flow chart of an embodiment of a method for checking the sensor data obtained by the sensors according to Fig. 1 are generated for the execution of the driver assistance system according to Fig. 1.

[0020] Fig. 1 shows a vehicle, in particular a motor vehicle 1, which can be a passenger car, for example. A vehicle occupant 2 can be transported in the motor vehicle 1. The vehicle occupant 2 can intervene in the control of the motor vehicle 1, but this is not necessary, since the motor vehicle 1 can be configured to drive autonomously. For this purpose, the motor vehicle 1 has, for example, a driver assistance system 16, explained in more detail below, for at least partially autonomous driving. Corresponding (semi-)autonomous driving functions can be instructed by means of a central processor unit in the form of a central computer 3. The central computer 3 comprises a processor 4, a memory unit 5, and a network device 6. A computer program product 7 is stored on the memory unit 5.

[0021] Fig. 1 further shows a first decentralized processor unit in the form of a first decentralized computer 8. The first decentralized computer 8 comprises a processor 9, a memory unit 10 and a network device 11. A computer program product 12 is stored on the memory unit 10. The first decentralized computer 8 can access sensor data 13, e.g., by means of a network device 11. In particular, during validation, decentralized processor units 8 and 25.1 to 25.5 of a decentralized or peer-to-peer computer network 24 (e.g., the one defined by Fig. 1) check the sensor data 13 before they are used by the driver assistance system 16 to create a representation of an external environment 14 of the motor vehicle 1 and optionally to control the motor vehicle 1 based on the representation of the external environment 14 of the motor vehicle 1.

[0022] The driver assistance system 16 comprises in the embodiment shown a processor 17, a memory unit 18, a network device 19 and an actuator 21. The driver assistance system 16 can be controlled based on sensor data 13 generated by a plurality of sensors 15, 20. In the embodiment shown by Fig. 1, the motor vehicle 1 comprises, purely by way of example, a first sensor 15, e.g., a camera system, and a second sensor 20, e.g., a radar system. A computer program product 22 is stored on the memory unit 18. Purely by way of example, the computer program product 22 can contain instructions to access the sensor data 13, to generate a representation of the surroundings 14 of the motor vehicle 1 based thereon, and in turn to control the actuator 21 based thereon so that the autonomous driving function is carried out. The autonomous driving function can, for example, involve the motor vehicle 1 being steered, accelerated, or braked. The actuator 21 can, for example, be an engine, a brake, or a steering system of the motor vehicle 1.

[0023] Fig. Figure 2 shows the decentralized computer network 24, which is partially arranged within the motor vehicle 1. The peer-to-peer computer network 24 comprises several decentralized computers 8 and 25.1 to 25.5 that communicate with each other. Within the decentralized computer network 24, all computers 8 and 25 are interconnected and have equal rights. The first decentralized computer 8, in the example according to Fig. 2 a computer of the decentralized computer network 24 arranged inside the motor vehicle 1. The further decentralized computers 25.1 to 25.5 of the decentralized computer network 24 can be arranged inside or outside the motor vehicle 1 and can communicate with each other and with the first decentralized computer 8 via network devices. In particular, the same computer program product 12 can be stored on memory units of the further decentralized computers 25 as is also stored on the first decentralized computer 8. In this way, in particular the first decentralized computer 8 and the further decentralized computers 25 can be instructed by the computer program product 12 to carry out the testing of the sensor data 13, which will be explained below in particular with reference to Fig. 3 is explained in more detail.

[0024] In the Fig. 1, the first sensor 15 generates first sensor data 13.1 (e.g., camera data). The second sensor 20 generates second sensor data 13.2 (e.g., radar data). In a first step 100 of the method according to Fig. 3, the first sensor data 13.1 of the first sensor 15 and the second sensor data 13.2 of the second sensor 20 are collected. The sensor data 13 are composed in the embodiment according to Fig. 1 consists of the first sensor data 13.1 and the second sensor data 13.2. The sensor data 13.1 and 13.2 can be collected, for example, by the first decentralized computer 8 accessing the sensor data 13.1 and 13.2. Alternatively, this can also be done, for example, by the central processor unit 3 or by one of the other decentralized computers 25.1 to 25.5.

[0025] In a second step 200, the collected data 13.1 and 13.2 are encrypted into encrypted sensor data 13.1' and 13.2', thereby ensuring the confidentiality and integrity of the collected data 13.1 and 13.2. In a third step 300, a data block 23 is created. The data block 23 contains the encrypted sensor data 13.1' and 13.2'. Furthermore, a timestamp 26 and other relevant metadata associated with the encrypted data 13.1' and 13.2' are stored in the data block 23. In a fourth step 400, the data block 23 is transmitted to the decentralized computer network 24, specifically to each of its computers 8 and 25.1 to 25.5. Steps 200 to 400 can be carried out, for example, by means of the decentralized computer 8.

[0026] The encrypted sensor data 13.1' and 13.2' are verified by executing a consensus algorithm 27 by the computers 8 and 25.1 to 25.5 of the decentralized computer network 24. The consensus algorithm 27 is stored on the storage unit 10 of the decentralized computer 8, which is Fig.1. The consensus algorithm 27 is also stored on storage units (not shown) of the additional decentralized computers 25.1 to 25.5. Before the actual verification of the encrypted sensor data 13.1' and 13.2', in a fifth method step 500, one of the computers of the decentralized computer network 24 is selected to verify the encrypted sensor data 13.1' and 13.2' by jointly executing the consensus algorithm 27 by all computers 8 and 25.1 to 25.5 of the decentralized computer network 24. Methods such as proof-of-work or proof-of-stake can be used for this purpose. For example, the fifth additional decentralized computer 25.5 can be selected to subsequently verify the encrypted sensor data 13.1' and 13.2'.

[0027] In a sixth method step, the selected computer 25.5 checks the encrypted sensor data 13.1' and 13.2'. In doing so, the selected computer 8 determines a test result value ("score") 28 based on various criteria or aspects. For example, a reputation of the first sensor 15 and the second sensor 20 can be used as a criterion for determining the test result value 28. The reputation can be based on a history of the two sensors 15, 20, e.g., how often the two sensors 15, 20 have delivered sensor data in the past. The quality of the data delivered (quality, timeliness, etc.) by the two sensors 15, 20 can also be included as a reputation criterion in the calculation of the test result value 28. Furthermore, the compatibility or up-to-date nature of the data from the two sensors 15, 20 can be considered (e.g., software version of the data source).It can also be considered whether and, if so, how many manipulation or hacking attacks the two sensors 15, 20 have experienced, and the extent of the associated damage. The integrity of the data provided by the two sensors 15, 20 can also represent a meaningful reputation criterion. Corresponding information can be stored, for example, in data block 23 or in a blockchain 29, which computers 8 and 25.1 to 25.5 of the decentralized computer network 24 can access.

[0028] Alternatively or in addition to the reputation described above, a type of the first sensor 15 (camera) and the second sensor 20 (radar) can also be used as a criterion for calculating the check result value 28. Furthermore, alternatively or additionally, a tamper check can be performed to determine whether the sensor data 13.1', 13.2' have been tampered with, and the check result value 28 can be determined based on the result of this tamper check. Furthermore, additionally or alternatively, a duration of the data validation, the timestamp 26 of the data, or the like can be used to calculate the check result value 28. Corresponding information for the aforementioned alternative criteria or aspects can be stored, for example, in the data block 23 or in the blockchain 29.

[0029] If the test result value 28 lies within a predetermined value range as acceptable, then the data block 23 is stored in a data block 30 (which may be identical to the data block 23 or may contain additional data, e.g. that the test was performed and when) in the blockchain 29, which serves as an immutable, secure and transparent ledger in particular for the sensor data 13.1' and 13.2' of the data block 23 or 30.

[0030] In a seventh step 700, the verified, encrypted sensor data 13.1' and 13.2' stored in the blockchain 29 are decrypted, so that the original, unencrypted sensor data 13.1 and 13.2 are provided again. The decrypted sensor data 13.1 and 13.2 are merged into fused sensor data 13' in an eighth step 800. Based on the fused, decrypted sensor data 13', a comprehensive understanding of the external environment 14 of the motor vehicle 1 is generated in a ninth method step 900 by generating a particularly visual representation of the external environment 14 of the motor vehicle 1 based on the decrypted and fused sensor data 13'. This enables various driver assistance functions. Steps 700 to 900 can, for example, be executed by the driver assistance system 16.Furthermore, in a tenth step, the driver assistance system 16 can control the motor vehicle 1 based on the decrypted and fused sensor data 13', e.g. accelerate, brake or steer. Reference symbol 1 motor vehicle 2 vehicle occupants 3 central processor unit 4 processor 5 storage 6 Network setup 7 Computer program product 8 decentralized computers 9 processor 10 storage units 11 Network setup 12 Computer program product 13 Sensor data 13' fused sensor data 13.1 First sensor data of the first sensor 13.2 second sensor data of the second sensor 13.1' first encrypted sensor data of the first sensor 13.2' second encrypted sensor data of the second sensor 14 external environment motor vehicle 15 first sensor 16 Driver assistance system 17 processor 18 storage 19 Network setup 20 second sensor 21 Actuator 22 Computer program product 23 Data block 24 decentralized computer network 25.1 decentralized computer 25.2 decentralized computer 25.3 decentralized computer 25.4 decentralized computer 25.5 decentralized computers 26 timestamps 27 Consensus algorithm 28 Test result value 29 Blockchain 30 data blocks of the blockchain 100 first process step 200 second process step 300 third process step 400 fourth process step 500 fifth process step 600 sixth process step 700 seventh process step 800 eighth process step 900 ninth process step 1000 tenth process step

Claims

[1] Method for checking sensor data (13.1, 13.2) for the implementation of a driver assistance system (16) of a vehicle (1), wherein sensor data (13.1, 13.2) originating from at least one sensor (15, 20), which contain information about an external environment (14) of the vehicle (1), are checked by means of a decentralized computer network (24), wherein the sensor data (13.1, 13.2) are intended to be used in a driver assistance system (16) of a vehicle (1) in order to generate a representation of the external environment (14) of the vehicle (1). [2] Method according to claim 1, wherein the sensor data (13.1, 13.2) originate from a plurality of sensors (15, 20). [3] Method according to claim 1 or 2, wherein - first, the sensor data (13.1, 13.2) are encrypted to form encrypted sensor data (13.1', 13.2') and stored in a data block (23), and - the encrypted sensor data (13.1', 13.2') are then checked by means of the decentralized computer network (24). [4] Method according to one of the preceding claims, wherein - the decentralized computer network (24) comprises several computers (8, 25.1 to 25.5) and - checking the sensor data (13.1', 13.2') comprises executing a consensus algorithm (27) by the computers (8, 25.1 to 25.5) of the decentralized computer network (24). [5] Method according to claim 4, wherein by executing the consensus algorithm (27) - first, one of the computers (25.5) of the decentralized computer network (24) is selected which is to check the sensor data (13.1', 13.2'), and - then the selected computer (25.5) checks the sensor data (13.1', 13.2'). [6] Method according to one of the preceding claims, wherein checking the sensor data (13.1', 13.2') comprises - a reputation of the at least one sensor (15, 20) is determined and - a test result value (28) is determined based on the reputation of the at least one sensor (15, 20). [7] Method according to one of the preceding claims, wherein checking the sensor data (13.1', 13.2') comprises - a type of the at least one sensor (15, 20) is determined and - a test result value (28) is determined based on the type of sensor (15, 20). [8] Method according to one of the preceding claims, wherein checking the sensor data (13.1', 13.2') comprises - a manipulation test is carried out to check whether the sensor data (13.1', 13.2') have been manipulated, and - a test result value (28) is determined based on the result of the tamper check. [9] Method according to one of claims 6 to 8, wherein the sensor data (13.1', 13.2') are stored in a data block (30) of a blockchain (29) if the test result value (28) lies within a predetermined value range. [10] The method of claim 9, wherein - the sensor data (13.1', 13.2') stored in the data block (30) of the blockchain (29) are decrypted to decrypted sensor data (13.1, 13.2), - the decrypted sensor data (13.1, 13.2) are merged into fused sensor data (13'), and - the representation of the external environment (14) of the vehicle (1) is generated based on the decrypted and fused sensor data (13'). [11] Method according to one of the preceding claims, wherein the computers (8, 25.1 to 25.5) of the decentralized computer network (24) are arranged inside the vehicle (1) and / or outside the vehicle (1). [12] Method according to one of claims 10 or 11, wherein a driver assistance system (16) is executed based on the decrypted and fused sensor data (13').

Images