A method and device for enhancing sensor data against adversarial interference

Abstract

The invention relates to a method for enhancing sensor data (20, 21) against interference, wherein the sensor data (20, 21) are acquired from at least two sensors (10, 11), wherein the acquired sensor data (20, 21) of the at least two sensors (10, 11) are each replaced piecewise by stitching, wherein the piecewise replacement is carried out in such a way that the respectively replaced sensor data (30, 31) of the different sensors (10, 11) are mutually trustworthy, and wherein the piecewise replaced sensor data (30, 31) are output. The invention also relates to a device (1) for enhancing sensor data (20, 21) against interference, a method for operating an assistance system for a vehicle, an assistance system for a vehicle, a computer program, and a data carrier signal.

Description

[0001] The present invention relates to a method and a device for enhancing sensor data against interference, or making sensor data robust. The invention also relates to a method for operating an assistance system for a vehicle, an assistance system for a vehicle, a computer program, and a data carrier signal.

[0002] Machine learning, for example, based on neural networks, has enormous potential for use in modern vehicle assistance systems and autonomous vehicles. Deep neural network-based functions process sensor data (e.g., from cameras, radar, or lidar sensors) to derive key information. This information includes, for example, the type and location of objects in the vehicle environment, their properties, or the geometry or topology of lanes.

[0003] Among neural networks, convolutional neural networks (CNNs) in particular have proven particularly well-suited for use in image processing. Convolutional networks extract a variety of high-value features from input data (e.g., image data) in an unsupervised, phased manner. During the training phase, the convolutional network independently develops feature maps based on filter channels that locally process the input data to obtain local properties. These feature maps are then updated by other filter channels that derive even higher-value feature maps. Based on the information compressed from the input data in this way, the deep neural network ultimately derives its decision and provides it as output data.

[0004] Convolutional networks outperform classical methods in terms of feature accuracy, however they also have drawbacks. For example, attacks based on adversarial perturbations in sensor data / input data can lead to misclassification or incorrect semantic segmentation in the detected sensor data, despite no semantic changes.

[0005] According to the literature Chuan Guo et al., Countering Adversarial Images Using InputTransformations, axViv:1711.00117v3[cs.CV], 25.Jan.2018, https: / / arxiv.org / pdf / 1711.00117.pdf Quilting methods are known for eliminating adversarial interference in image data.

[0006] The object of the present invention is to improve methods and devices for enhancing sensor data against interference, in particular when using a plurality of sensors and sensor data fusion.

[0007] This object is achieved according to the invention by a method having the features of claim 1 and a device having the features of claim 9. Advantageous embodiments of the invention are given in the dependent claims.

[0008] In particular, a method for strengthening (or robustifying) sensor data against interference (or so-called interference disturbance) is provided, wherein the sensor data is obtained from at least two sensors, wherein the sensor data of the at least two sensors obtained are replaced segment by segment by means of stitching, wherein the segment by segment replacement is performed in such a way that the respectively replaced sensor data of different sensors are mutually trustworthy, and wherein the segment by segment replaced sensor data are output.

[0009] In addition, a device for enhancing sensor data against interference is provided, which includes a computing device, wherein the computing device is configured to obtain sensor data from at least two sensors, wherein the obtained sensor data of the at least two sensors are replaced segment by segment by stitching, and the segment by segment replacement is performed in such a way that the respectively replaced sensor data of different sensors are mutually trustworthy, and wherein the segment by segment replaced sensor data is output.

[0010] The method and the device achieve the goal of strengthening the sensor data provided by multiple sensors, i.e., at least two sensors, against interference when using the multiple sensors. To this end, the sensor data of the at least two sensors are replaced piece by piece by stitching. In this case, the replacement is performed piece by piece, i.e., the sensor data after the piece-by-piece replacement are mutually trusted (across sensors). The sensor data patches used for the piece-by-piece replacement during stitching are selected such that the matched replaced sensor data of the at least two sensors are mutually trusted in terms of time and space. If the sensor data thus strengthened are subsequently input as input data, for example, into a neural network, the interference originally contained in the obtained sensor data loses its effect, otherwise the semantic content in the sensor data is changed. Since the trustworthiness between the sensor data of the at least two sensors is maintained, the substantial relationship, in particular the spatial and temporal relationship or correlation, between the sensor data of the at least two sensors is not changed by the piece-by-piece replacement. The advantage is, in particular, that sensor data fusion is achieved after the method is implemented.

[0011] Stitching, in particular, involves the segment-by-segment replacement of sensor data, which can also be referred to as segment-by-segment reconstruction of sensor data (in the case of image data, the term "image stitching" can also be used). In particular, the sensor data can be of any type, meaning stitching is not limited to two-dimensional image data. The amount of replaced sensor data constitutes or is contained within the reconstructed data domain. For example, in the case of camera images, the camera image is divided into a number of partial segments. Typically, a number of small, rectangular image segments (also referred to as patches) are provided for this purpose. Each partial segment or image segment is compared with a partial segment (hereinafter referred to as a sensor data patch), for example, stored in a database. A sensor data patch can also be referred to as a data block. A sensor data patch constitutes, in particular, a subset of previously detected subsymbols of the same type of sensor data, wherein the sensor data is protected against adversarial interference. The comparison is performed based on a distance metric, for example, defined by the Euclidean distance on image element vectors or sensor data vectors. For this purpose, the partial segments or image segments are linearized into vectors. The distance is determined using a vector space norm, for example, the L2 norm. Each partial segment or image segment is replaced by the closest or most similar sensor data patch from the database. In this context, it can be stipulated that a minimum distance must be observed, or at least that there should be no identity between partial segments of sensor data and sensor data patches. If the sensor data is in another form (e.g., lidar data) or another format, the segment-by-segment replacement is performed in a similar manner. In particular, the segment-by-segment replacement is performed for all partial segments of the detected sensor data, so that replaced or reconstructed sensor data is then available. After the segment-by-segment replacement, i.e., after stitching, the effects of the interference countermeasure are eliminated or at least reduced in the replaced or reconstructed sensor data.

[0012] The "plausibility" of replaced sensor data should particularly mean that the replaced sensor data are physically plausible with one another. In particular, the probability that the respectively replaced sensor data also occur in the respectively selected combination under real conditions, i.e., in the real world, should be as high as possible (e.g., in the sense of maximum likelihood). In short, the replaced sensor data of at least two sensors should be selected such that the probability that the sensor data also actually occur in this combination is maximized. For example, if the at least two sensors are a camera and a lidar sensor, the plausibility of the replaced sensor data relative to one another means that the image segment of interest in the replaced camera data and the partial segment of the replaced lidar data that is temporally and spatially aligned with it are selected such that the sensor data are consistent, i.e., physically consistent with one another. In the example mentioned above where the at least two sensors are a camera and a lidar sensor, the partial segments of the sensor data are respectively replaced such that each replaced image segment is aligned with the replaced partial segment of the lidar data, as if the replaced image segment were likely formed by simultaneous detection of the sensor data from the camera and lidar sensor.

[0013] The at least two sensors are calibrated in particular with respect to one another in time and space, so that the sensor data of the at least two sensors are adapted to one another in time and space or have a common time and space reference point.

[0014] The sensor data from the at least two sensors can, in principle, be one-dimensional or multi-dimensional, in particular two-dimensional. For example, the sensor data can be a two-dimensional camera image from a camera or two-dimensional or three-dimensional lidar data from a lidar sensor. However, the sensor data can also generally originate from other sensors, such as radar sensors or ultrasonic sensors.

[0015] The acquired sensor data are, in particular, sensor data detected and / or emitted for functions for automatic or semi-automatic driving of a vehicle and / or for environmental awareness.

[0016] The vehicle is in particular a motor vehicle. However, the vehicle can in principle also be another land, air, water, rail or aviation vehicle, such as a drone or an air taxi.

[0017] Adversarial perturbations are, in particular, targeted perturbations of input data of a neural network, for example provided in the form of sensor data, wherein, despite the fact that the semantic content of the input data remains unchanged, the perturbation leads the neural network to infer incorrect results, such as an incorrect classification or an incorrect semantic segmentation of the input data.

[0018] Neural networks, in particular deep neural networks, in particular convolutional neural networks (CNNs), are trained, for example, for specific functions, such as functions of vehicle assistance systems, in particular for autonomous or semi-autonomous driving and / or for environmental perception, such as the perception of pedestrians or other objects in captured camera images.

[0019] The method is repeated in particular cyclically, so that in particular gradually replaced sensor data can be provided to the sensor data of the acquired sensor data stream.

[0020] The method can also be implemented as a computer-implemented method. The method can be implemented in particular by means of a data processing device. The data processing device in particular includes at least one computing device and at least one storage device.

[0021] In particular, a computer program is provided, which comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out the method steps of the method according to any of the described embodiments.

[0022] Furthermore, a data carrier signal is provided, which conveys the computer program.

[0023] The components of the device, in particular the computing device, can be designed individually or in combination as a combination of hardware and software, for example as program code executed on a microcontroller or microactuator. However, it can also be provided that the components are constructed individually or in combination as an application-specific integrated circuit (ASIC).

[0024] It may be provided that the method includes detecting sensor data by means of at least two sensors.

[0025] In one embodiment, it is provided that the segmented replaced sensor data are input into at least one function for automatic or semi-automatic driving of a vehicle and / or for environmental perception. As a result, reinforced sensor data are input to at least one function, so that the functionality provided by the at least one function can also be provided more robustly. The segmented replaced sensor data are input to at least one function, and the at least one function generates and provides in particular at least one control data and / or evaluation data based on the segmented replaced sensor data. As a result, the output of the at least one function can be reliably generated and provided in particular in the form of at least one control signal and / or evaluation signal. The at least one control signal and / or evaluation signal can, for example, be used to control or regulate an actuator of the vehicle and / or be further processed within the scope of automatic or semi-automatic driving, for example for trajectory planning. The at least one function is in particular a function provided by means of machine learning methods and / or artificial intelligence. For example, the at least one function can be provided by means of a trained artificial neural network.

[0026] In one embodiment, a database is provided for stitching, the database comprising sensor data patches generated from sensor data from at least two sensors. The sensor data patches from the at least two sensors are associated with each other in the database, so that the associated sensor data patches are mutually authentic. This ensures the plausibility of the replaced sensor data in a particularly efficient manner. The sensor data patches are used to replace the sensor data from the at least two sensors in sections, and can be stored in the database as common database entries. The sensor data patches for the at least two sensors can already be combined into a vector and stored in the database. The sensors are, for example, cameras and lidar sensors. The sensor data patches used for replacement, i.e., individual image segments, can be combined with the sensor data patches, i.e., partial segments of lidar data, that are physically authentically matched thereto, to form a common vector. Depth information derived from the lidar data is then assigned to each pixel of the camera. For example, if the image segments are each considered to consist of 8x8 pixels, then for three color channels and one depth channel, a vector with (8x8x(3+1)=)256 entries is formed. Each entry in the database includes such a vector. The sensor data of at least two sensors are then combined similarly to the stored vectors so that the distance to the vector stored in the database can be determined with the help of a distance metric, such as the L2 norm. The vector in the database that has the smallest distance to the vector to be replaced is used in the stitching for replacement. The creation of the database is particularly based on sensor data detected (in advance independently of the disclosed method), wherein the sensor data of at least two sensors are thus detected in particular simultaneously, wherein the sensors are calibrated with each other in space and time. Trusted sensor data is used here, that is, sensor data that is guaranteed to be free of countermeasure interference. For example, training data of a (deep) neural network can be used here, into which the (replaced) sensor data should be input during the application phase. Sensor data patches are generated from the trusted sensor data and stored in the database. The at least two sensors are other types of sensors, in which case the method is carried out similarly.

[0027] In an improved embodiment, it is provided that the sensor data patches used for the stitching are selected for the at least two sensors based on only a portion of the sensor data from the at least two sensors. In particular, it can be provided that the sensor data patches used to replace the sensor data of the at least two sensors are selected based on the sensor data obtained from only one sensor. This can reduce the computational effort required for the search, for example, because the comparison with the sensor data patches in the database only considers the sensor data of one sensor. If a sensor data patch with the smallest distance to the sensor data of the one sensor is found, the sensor data patches of one or more of the at least two sensors can be obtained from the found sensor data patches without further searching, given the existing association. This can accelerate the search in the database. In the example described above, the sensor data patches are stored in the data as 8x8 pixels, which include multiple image segments of a camera, together with the corresponding depth information. In this example, it can be provided that the comparison with the sensor data patches in the database is performed for only one image segment, wherein the corresponding entry in the vector is then directly used for the lidar data to replace the lidar data. Alternatively, the comparison or search for the closest sensor data patch can be performed based on lidar data, wherein, after finding the sensor data patch, the corresponding image segment in the vector entry is used. Overall, the comparison or search in the database can be accelerated. Because the sensor data patches for the at least two sensors stored in the database are linked to one another in a reliable manner, despite the accelerated search, no quality defects occur during stitching or when replacing the sensor data of the at least two sensors. After stitching, the replaced sensor data of the at least two sensors remain mutually reliable.

[0028] In one embodiment, at least one characteristic information item is obtained, wherein the segment-by-segment replacement during stitching also takes into account the at least one obtained characteristic information item. The characteristic information item can be labeled or marked. This allows, for example, entries in a database, i.e., sensor data patches stored therein, to be marked with additional information, thereby enabling them to be found more quickly later. In particular, it can be provided that a hash function is used for searching the database, thereby accelerating searches in the database because the number of entries in the database can be reduced by preselection before the comparison with the sensor data from the at least two sensors.

[0029] In an improved embodiment, it is provided that the acquired characteristic information is derived or can be derived from contextual information of the environment in which the sensor data of at least two sensors are detected or have been detected. The contextual information can include, for example, geographic coordinates (e.g., GPS coordinates), time of day and / or season, month, day of the week, weather (sun, rain, fog, snow, etc.), and / or traffic environment (city, country, highway, pedestrian zone, country road, main road, secondary road, etc.). Therefore, on the one hand, the quality of the sensor data after the segment-by-segment replacement can be improved, because the environment in which the sensor data was detected can be taken into account in the segment-by-segment replacement. In particular, the sensor data patches can be stored in the database when they are marked ("getaggt") with at least one contextual information. On the other hand, based on at least one acquired characteristic information or at least one contextual information, a preselection can be made before searching in the database so that only entries or sensor data patches that have partial or complete consistency with at least one characteristic information or at least one contextual information are considered during the search. This can accelerate the segment-by-segment replacement.

[0030] In one embodiment, the acquired sensor data is replaced piecemeal by piece while taking into account temporally and / or spatially adjacent sensor data from at least two sensors. This allows for correlations between temporally and / or spatially adjacent sensor data to be taken into account during the piecemeal replacement. In the example of camera images, it can be considered, for example, that the properties of individual image segments of a camera image generally have a high correlation with (spatially) adjacent image segments of the camera. If a series of camera images is considered, image segments of a camera image generally have a high correlation with the properties of the same image segments of (temporally) adjacent camera images. This is used for the piecemeal replacement to accelerate the method. In particular, it can be provided that the entries or sensor data patches stored in the database are labeled with each other based on temporal and / or spatial proximity. Sensor data patches stored as entries in the database can, in particular, be linked to other stored sensor data patches in terms of their temporal and / or spatial proximity. This allows for faster comparisons with sensor data patches stored in the database. For example, it may be provided that, after a sensor data patch for a certain partial segment is found in the sensor data for one of the at least two sensors, a preselection is made for other partial segments of the sensor data for the one of the at least two sensors. The preselection includes sensor data patches that are less than a predetermined temporal and / or spatial distance from the already selected sensor data patch, i.e., are in a predetermined temporal and / or spatial proximity to the already selected sensor data patch.

[0031] The features regarding the design of the device are derived from the description of the design of the method. The advantages of the device are the same as those in the design of the method.

[0032] Furthermore, a method for operating an assistance system for a vehicle is provided, wherein at least one function for automatic or semi-automatic driving of the vehicle and / or for environmental perception is provided with the aid of the assistance system, wherein sensor data is detected with the aid of at least two sensors, wherein a method according to any of the above-described embodiments is implemented, wherein the segmented replaced sensor data are input into at least one function, and wherein the at least one function generates and provides at least one control signal and / or evaluation signal based on the segmented replaced sensor data.

[0033] In addition, an assistance system for a vehicle is provided, which includes: at least two sensors configured to detect sensor data; a device according to any of the above-mentioned embodiments, wherein the assistance system is configured to provide at least one function for automatic or semi-automatic driving and / or for environmental perception of the vehicle, wherein the at least one function generates and provides at least one control signal and / or evaluation signal based on sensor data replaced segment by segment with the aid of the device.

[0034] Furthermore, a vehicle is provided, comprising at least one device and / or at least one auxiliary system according to any of the above embodiments. The vehicle is particularly a motor vehicle. However, the vehicle may also be other land, air, water, rail, or aviation vehicles, such as drones or air taxis.

[0035] The preferred embodiments of the present invention are described in more detail below with reference to the accompanying drawings. In the accompanying drawings:

[0036] Figure 1 Schematic diagram showing an embodiment of a device and an embodiment of an auxiliary system for hardening sensor data against interference;

[0037] FIG2 shows a schematic diagram for explaining suturing (prior art);

[0038] Figure 3 A schematic diagram of suturing is shown for illustrating an embodiment of the method described in this application.

[0039] exist Figure 11 shows a schematic diagram of an embodiment of a device 1 for enhancing sensor data 20, 21 against interference. The device 1 comprises a computing device 2 and a storage device 3. The device 1 can be used in particular in a vehicle, in particular a motor vehicle, to enhance input data of a neural network 50 used therein against interference. The device 1 implements the method described in this application for enhancing sensor data 20, 21 against interference.

[0040] The components of the device 1 , in particular the computing device 2 , can be designed individually or in combination as a combination of hardware and software, for example as program code executed on a microcontroller or microactuator.

[0041] Sensor data 20, 21 of two sensors 10, 11 are input to the device 1 or computing device 2. The sensors 10, 11 can be, for example, a camera and a lidar.

[0042] The computing device 2 acquires or receives the sensor data 20, 21 and replaces them piecemeal by stitching. This piecemeal replacement is performed so that the piecemeal replaced sensor data 30, 31 of the two sensors 10, 11 are mutually authentic.

[0043] The segmented sensor data 30, 31 are then output by the computing device 2. The segmented sensor data 30, 31 are then fed into an artificial neural network 50. The neural network 50 is provided by a control unit 51, for example, by a computing device of the control unit 51 providing the functionality of the neural network 50 or performing the computational operations required to provide the neural network 50. The neural network 50 provides, in particular, functions for autonomous or semi-autonomous driving of a vehicle and / or for environmental awareness. These functions are provided by one of the sensors 10, 11 and the system including the assistance system 200. The neural network 50 is trained based on these functions. Based on the segmented sensor data 30, 31, the functions provided by the neural network 50 generate at least one control signal 52 and / or evaluation signal 53, which can be fed back, for example, to an actuator (not shown) of the vehicle and / or to at least one other control unit of the vehicle.

[0044] After stitching or after segment-by-segment replacement, the sensor data 30 , 31 have the same format as the sensor data 20 , 21 , so that the device 1 can be integrated and used in sensors 10 , 11 and neural networks 50 already in use.

[0045] In particular, provision is made for stitching to provide a database 40 having sensor data patches 60, 61 generated from sensor data of sensors 10, 11, wherein the sensor data patches 60, 61 of sensors 10, 11 are linked to one another in database 40, such that the respectively linked sensor data patches 60, 61 are mutually trustworthy. Database 40 is stored, for example, in storage device 3.

[0046] To this end, database 40 is previously created using authentic, detected sensor data from two sensors 10 and 11 by generating a large number of interrelated sensor data patches from the authentic, detected sensor data. Authenticity here particularly means that the detected sensor data is guaranteed to contain no adversarial interference. Trustworthy sensor data can be, for example, camera images and lidar data. Thus, it can be provided that sensor data patches 60 and 61 each include an 8x8 pixel subsegment of a camera image and an adapted subsegment of lidar data from 8x8 measurement points. The sensors used, or authentic sensors, are here particularly calibrated with respect to one another in terms of time and space.

[0047] To perform segment-by-segment replacement during stitching, the computing device 2 operates as follows. The sensor data 20, 21 are each divided into partial segments. These partial segments are compared with sensor data patches 60, 61 stored in the database 40. Based on a distance metric, a search is performed for each partial segment to find a sensor data patch 60, 61 that has the smallest distance from the partial segment in question. For this purpose, the sensor data 20, 21 included in each partial segment and the sensor data included in the sensor data patches 60, 61 are each expressed as a vector. Using a distance metric, such as the L2 norm, the distances between the vectors can be determined and the determined distances compared with one another. If a sensor data patch 60, 61 with the smallest distance from the partial segment in question is found, the partial segment in the sensor data 20, 21 is replaced with this sensor data patch and provided as replaced sensor data 30, 31. Since the sensor data patches for the two sensors 10, 11 in the database 40 are linked to one another, the sensor data 20, 21 of the two sensors 10, 11 are replaced using the linked sensor data patches 60, 61. By using the associated sensor data patches 60 , 61 , the replaced sensor data 30 , 31 of the two sensors 10 , 11 are mutually plausible.

[0048] Provision can be made to preselect the sensor data patches 60, 61 used for both sensors 10, 11 based on the sensor data 20, 21 of only some of the sensors 10, 11. For example, the preselection can be based solely on the sensor data 20 of sensor 10. Since the sensor data patches 60, 61 are linked to one another, for each sensor data patch 60 found from the sensor data 20, the associated sensor data patch 61 for the sensor data 20 is immediately identified.

[0049] Provision can be made, characterized in that the segment-by-segment replacement of the acquired sensor data 20, 21 is performed taking into account the temporally and / or spatially adjacent sensor data 20, 21 of at least two sensors 10, 11, respectively. The sensor data patches 60, 61 in the database 40 can be associated or labeled with respect to temporal and / or spatial proximity, in particular. This allows a preselection to be made during the search for sensor data patches 60, 61, wherein temporal and / or spatial correlations are taken into account when the sensor data 20, 21 represented by the sensor data patches 60, 61 are present.

[0050] In particular, it is provided that the acquired sensor data 20 , 21 are sensor data detected and / or output for functions for automatic or semi-automatic driving of a vehicle and / or for driver assistance of a vehicle and / or for environment detection and / or environment perception.

[0051] If there are more than two sensors 10, 11, the method for obtaining the sensor data 30, 31 for all sensors 10, 11 is carried out in a similar manner. In particular, after carrying out the method, the replaced sensor data 30, 31 of all sensors 10, 11 are mutually trustworthy. In particular, it can be provided that, as an alternative or in addition, cameras and lidar can also be used as further sensors.

[0052] FIG2 illustrates stitching using a camera image 22 as an example in the prior art. Sensor data 20, here a camera image 22, is broken down into a plurality of sub-segments 23. Within the scope of a stitching step 100, a search is performed in database 40 for a sensor data patch 60 that has the minimum distance relative to the sub-segment 23, using a distance metric. A sensor data patch 60 is an image segment having the same dimensions as the sub-segment 23, i.e., an image segment having the same number of pixels. The distance metric is, for example, the L2 norm, which is applied to the vectors generated by linearizing the image segments. In the stitching step 100, each sub-segment 23 is replaced by the corresponding sensor data patch 60 having the minimum distance thereto. It can be provided that the minimum distance must be maintained. In this way, all sub-segments 23 are replaced by sensor data patches 60 from database 40. This results in replaced sub-segments 24, which form the merged, replaced sensor data 30 or replaced camera image 25.

[0053] exist Figure 3 , a schematic diagram illustrating stitching according to an embodiment of the method described herein is shown, using as an example sensor data 20 in the form of a camera image 22 and sensor data 21 in the form of lidar data 26. The stitching itself is performed in the same manner as described above in conjunction with FIG. However, the segment-by-segment replacement is performed such that the corresponding replaced sensor data 30, 31 of the sensors, i.e., the camera and lidar sensor, are mutually trustworthy. To this end, a plausibility check is also performed in the stitching step 100. In particular, within the scope of the stitching step 100, sensor data patches 60, 61 are also determined or selected from the database 40 so that the sensor data patches 60, 61 are mutually trustworthy. In the present example, this means, in particular, that the replaced partial segment 24 of the camera sensor data 20 is matched in a physically trustworthy manner to the replaced partial segment 28 of the lidar data 26. In short, the replaced partial segments 24, 28 or replaced camera image 25 and replaced lidar data 29 are coordinated with one another and should be substantially or physically consistent. In particular, the image shown in replaced camera image 25 and the depth profile of replaced lidar data 29 should then match one another in a plausible manner.

[0054] For this purpose, it is particularly provided that the sensor data patches 60, 61 are already stored in association with one another in the database 40. For example, the database 40 can be created before the method described herein is carried out by generating sensor data patches 60, 61 for two (or more) sensors simultaneously, wherein partial segments are respectively generated from simultaneously detected authentic sensor data and are stored jointly or in association with one another as sensor data patches 60, 61 in the database 40. For example, the individual partial segments for the two sensors are combined to form a common vector and stored as a common or associated sensor data patch 60, 61.

[0055] It can also be provided that at least one characteristic information item 15 is obtained, wherein the segment-by-segment replacement in the stitching step 100 also takes into account the at least one obtained characteristic information item 15. For example, when searching in the database 40, a preselection of the sensor data patches 60, 61 is performed based on the obtained characteristic information item 15, thereby accelerating the search for the sensor data patch 60, 61 having the smallest distance.

[0056] In an improved embodiment, it may be provided that the acquired characteristic information 15 is derived or can be derived from context information 16 of the environment in which the sensor data 20, 21 of the at least two sensors are or have been detected. The context information 16 may, for example, be geographic coordinates (e.g., GPS coordinates), time of day and / or season, month, day of the week, weather (sun, rain, fog, snow, etc.), and / or traffic conditions (urban, rural, highway, pedestrian zone, rural road, main road, secondary road, etc.). This context information 16 may, for example, be detected or otherwise provided by at least one environmental sensor. In a vehicle, the context information may be retrieved in the vehicle control via the control area network (CAN) bus. Using the environmental sensor 16, for example, a preselection may be made from sensor data patches 60, 61, thereby accelerating the search for the closest sensor data patch 60, 61. For this purpose, it is provided that the sensor data patches 60, 61 are each tagged ("getaggt") with the corresponding characteristic of the context information and stored in the database 40.

[0057] List of Reference Numerals

[0058] 1 Equipment

[0059] 2 Computing devices

[0060] 3 Storage devices

[0061] 10 Sensors (Cameras)

[0062] 11 Sensor (LiDAR sensor)

[0063] 15 Feature Information

[0064] 16 Contextual Information

[0065] 20 sensor data

[0066] 21 Sensor Data

[0067] 22 camera images

[0068] 23 Partial fragment

[0069] 24 Replaced fragments

[0070] 25 Replaced camera image

[0071] 26 LiDAR data

[0072] 27 Partial fragment

[0073] 28 Replaced fragments

[0074] 29 Replaced lidar data

[0075] 30 Sensor data after replacement

[0076] 31 Sensor data after replacement

[0077] 40 Database

[0078] 50 Neural Networks

[0079] 51 Control Equipment

[0080] 52 control signals

[0081] 53 Evaluating Signals

[0082] 60 sensor data patches

[0083] 61 Sensor Data Patch

[0084] 100 Suture Steps

[0085] 200 Assistance Systems

Claims

1. A method for enhancing sensor data (20, 21) against interference, wherein: The sensor data (20, 21) are obtained from at least two sensors (10, 11), wherein the obtained sensor data (20, 21) of the at least two sensors (10, 11) are replaced section by section by stitching, wherein the section by section replacement is performed in such a way that the respectively replaced sensor data (30, 31) of the different sensors (10, 11) are mutually credible, and wherein the section by section replaced sensor data (30, 31) are output, wherein for stitching a database (40) is provided, which has at least Sensor data patches (60, 61) generated from sensor data (20, 21) of two sensors (10, 11), wherein the sensor data patches (60, 61) of at least two sensors (10, 11) are associated with one another in a database (40) such that the respectively associated sensor data patches (60, 61) are mutually trustworthy, for which purpose the database is created based on detected sensor data, wherein the sensor data of at least two sensors are detected simultaneously, wherein the at least two sensors are spatially and temporally calibrated with one another.

2. The method according to claim 1, characterized in that The segment-wise replaced sensor data (30, 31) are fed into at least one function for automatic or semi-automatic driving of a vehicle and / or for environmental awareness.

3. The method according to claim 2, characterized in that Sensor data patches (60, 61) for at least two sensors (10, 11) to be used when stitching are selected based on only a portion of the sensor data (20, 21) of the at least two sensors (10, 11).

4. The method according to claim 1, wherein At least one characteristic information (15) is obtained, wherein the segment-by-segment replacement during stitching also takes into account the at least one characteristic information (15) obtained.

5. The method according to claim 4, characterized in that The acquired characteristic information (15) is derived or can be derived from context information (16) of an environment in which sensor data (20, 21) of at least two sensors (10, 11) are or have been detected.

6. The method according to claim 1, characterized in that The acquired sensor data (20, 21) are replaced piecemeal by piece taking into account the temporally and / or spatially adjacent sensor data (20, 21) of at least two sensors (10, 11).

7. A method for operating an assistance system (200) for a vehicle, wherein: At least one function for automatic or semi-automatic driving of a vehicle and / or for environmental awareness is provided with the aid of the assistance system (200), wherein sensor data (20, 21) are detected with the aid of at least two sensors (10, 11), wherein a method according to any one of claims 1 to 6 is implemented, wherein the segmented replaced sensor data (30, 31) are input to at least one function, and wherein the at least one function generates and provides at least one control signal (52) and / or evaluation signal (53) based on the segmented replaced sensor data (30, 31).

8. A device for enhancing sensor data (20, 21) against interference, the device comprising a computing device (2), wherein the computing device (2) is configured to obtain sensor data (20, 21) of at least two sensors (10, 11), wherein the obtained sensor data (20, 21) of the at least two sensors (10, 11) are replaced piece by piece by stitching, and the piece by piece replacement is performed in such a way that the respectively replaced sensor data (30, 31) of different sensors (10, 11) are mutually trustworthy, and wherein The sensor data (30, 31) are output after segment-by-segment replacement, wherein for stitching, a database (40) is provided, which has sensor data patches (60, 61) generated from sensor data (20, 21) of at least two sensors (10, 11), wherein the sensor data patches (60, 61) of at least two sensors (10, 11) are associated with each other in the database (40) such that the respectively associated sensor data patches (60, 61) are mutually trustworthy, for which purpose the database is created based on the detected sensor data, wherein the sensor data of at least two sensors are detected simultaneously, wherein the at least two sensors are spatially and temporally calibrated with each other.

9. An auxiliary system (200) for a vehicle, the auxiliary system comprising: At least two sensors (10, 11) configured to detect sensor data (20, 21), a device (1) according to claim 8, wherein the assistance system (200) is configured to provide at least one function for automatic or semi-automatic driving of a vehicle and / or for environmental perception, wherein the at least one function generates and provides at least one control signal (52) and / or evaluation signal (53) based on the sensor data (30, 31) replaced in sections by means of the device (1).

10. A computer program product comprising instructions which, when said computer program is executed by a computer, cause the computer to carry out the method steps of the method according to any one of claims 1 to 7.

Images