Method for detecting anomalies in vehicle sensor data, use of the method to detect a spoofing attack on a vehicle, and advanced driver assistance system

Abstract

A method for detecting anomalies in image data captured by a vehicle sensor includes receiving first and second image frames, and first and second observation clouds generated based on LiDAR data capt

Description

TECHNICAL FIELD

[0001] Various embodiments relate to a computer-implemented method for detecting anomalies in vehicle sensor data. Various embodiments also relate to use of the aforementioned method to detect a spoofing attack on a vehicle. Various embodiments also relate to an Advanced Driver Assistance System (ADAS). BACKGROUND

[0002] An autonomous vehicle typically uses a set of sensors to navigate and monitor its environment. These sensors may include Global Positioning System (GPS), inertial measurement unit (IMU), cameras and LiDAR. These sensors may be susceptible to sensor anomalies that cause errors in computations of the vehicle’s pose, velocity, and obstacle semantics. These sensor anomalies can result from comer cases. Comer cases may result from mechanical failures in sensors and the complexity of the driving environment. These anomalies may also occur due to malicious manipulating, i.e., spoofing attacks. While anomaly-detection solutions have been developed to address these threats, the existing solutions are often constrained by their narrow focus on specific types of anomalies. Further, these solutions generally require significant computational resource from the graphics processing unit (GPU), resulting in considerable detection latency. As such, there is a need for an improved method to detect anomalies in sensor data of vehicles and to detect spoofing attacks on vehicles. SUMMARY

[0003] According to various embodiments, there is provided a computer-implemented method for detecting anomalies in vehicle sensor data. The method includes receiving image data comprising a first image frame and a second image frame succeeding the first image frame. The method further includes receiving a first observation point cloud generated based on data captured by a LiDAR sensor on a first scan, and further receiving a second observation point cloud generated based on data captured by the LiDAR sensor on a second scan succeeding the second image frame. All scans between the first scan and the second scan occur during a time interval between the first image frame and the second image frame. The method further includes determining a motion transformation matrix based on the first image frame and the second image frame. The method further includes determining an interpolated transformation matrix based on the motion transformation matrix, and further based on when the first scan and the second scan occurred. The method further includes generating a projected point cloud based on the first observation point cloud and further based on the interpolated transformation matrix. The method further includes determining an aggregated error based on the projected point cloud and the second observation point cloud. The method further includes detecting anomalies in the image data based on the aggregated error.

[0004] According to various embodiments, there is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the abovementioned method for detecting anomalies in vehicle sensor data.

[0005] According to various embodiments, there is provided a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the abovementioned method for detecting anomalies in vehicle sensor data.

[0006] According to various embodiments, there is provided a use of the abovementioned method to detect a spoofing attack on a vehicle.

[0007] According to various embodiments, there is provided a data processing apparatus comprising means for carrying out the abovementioned computer-implemented method for detecting anomalies in vehicle sensor data.

[0008] According to various embodiments, there is provided an ADAS that includes a camera and a LiDAR sensor. The camera is configured to generate the image data. The LiDAR sensor is configured to capture data for generating each of the first observation point cloud and the second observation point cloud. The ADAS further includes the abovementioned data processing apparatus.

[0009] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0011] FIG. 1 shows a block diagram of an anomaly detection framework according to various embodiments.

[0012] FIG. 2 shows a block diagram of another anomaly detection framework according to various embodiments.

[0013] FIG. 3 shows a flow diagram of a method for detecting anomalies in vehicle sensor data according to various embodiments.

[0014] FIG. 4 shows a simplified block diagram of a data processing apparatus according to various embodiments.

[0015] FIG. 5 shows a simplified block diagram of an ADAS according to various embodiments. DESCRIPTION

[0016] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0017] It will be understood that any property described herein for a specific device may also hold for any device described herein. It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0018] The term “coupled” (or “connected”) herein may be understood as electrically coupled or as mechanically coupled, for example attached or fixed, or just in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words: coupling without direct contact) may be provided.

[0001] In this context, the device as described in this description may include a memory which is for example used in the processing carried out in the device. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0019] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0020] The present disclosure contemplates a solution to detect anomalies in vehicle sensor data, particularly in autonomous vehicles, by checking the spatio-temporal consistency of extracted features from camera images, and correcting for distortions in the camera data using LiDAR data. The solution may include an end-to-end, real-time detection framework that utilizes direct visual method of camera-lidar fusion to identify a broad range of anomalies with high interpretability. The detection framework may be referred to, in the present disclosure, as a method for detecting anomalies in image data. The method may include assessing spatiotemporal consistency with high frequency and low fidelity to register point clouds. Subsequently, a LiDAR-based method may be used to refine these estimates at a lower frequency, additionally correcting any distortions in the point clouds caused by drifts in the visual detection. This dual-sensor approach may ensure accurate detection of sensor anomalies, even in conditions of rapid motion or poor lighting.

[0021] According to various embodiments, the detection framework may formalize spatiotemporal consistency to enhance the interpretability of the detection framework. To this end, the detection framework may define and validate spatio-temporal consistency conflicts that occur during anomaly scenarios. These conflicts may be formulated as an optimization problem. The defined optimization problem may be transformed into a streamlined, end-to-end system. This system may incorporate low-weight modules while optimizing data fusion and overall system performance. The detection framework may achieve real-time detection.

[0002] According to various embodiments, the detection framework may be effective across various sensor spoofing scenarios, by leveraging on the fused camera-LiDAR data and the established spatio-temporal consistency conflicts to accurately identify and mitigate different types of anomalies.

[0022] FIG. 1 shows a block diagram of an anomaly detection framework 100 according to various embodiments. The anomaly detection framework 100 may be configured to perform a method for detecting anomalies in vehicle sensor data, such as image data. The anomaly detection framework 100 may include performing a plurality of modules. The plurality of modules may include a feature extraction module 102, a feature prediction and generation module 104, a match pairs searching module 106 and an offset error computation module 108. The anomaly detection framework 100 may detect anomalies based on determining the spatiotemporal consistency of the features in image data.

[0023] The feature extraction module 102 may be configured to extract two-dimensional (2D) feature points from images using suitable feature extraction algorithms.

[0024] According to an embodiment, the feature extraction module 102 may extract the 2D feature extraction by selecting a pixel p as the center and by searching for 16 pixels on a circular pattern with a radius of 3 pixels. If the difference in brightness between the searched pixels and p exceeds a predetermined threshold (for example, 20%) for 12 consecutive pixels, / ? may be considered as a keypoint and selected as a feature. Keypoints are typically located in comers, edges, and blocks within the images. To ensure scale and rotation invariance in the vehicle’s movement, the extracted keypoints may be further processed using multi-scale image pyramids and intensity centroid algorithms. An image pyramid represents a single image at multiple scales, with each level being a downsampled version of the original image. All keypoints extracted from various levels may be collected. The intensity centroid algorithm leverages the fact that a comer’s intensity is the offset from its center to determine the orientation. A descriptor, which is a 256-bit binary feature vector consisting of only Is and 0s, may be used to further differentiate the keypoints. Each bit may be computed by comparing the brightness of surrounding pixels based on a predefined pattern.

[0025] In some embodiments, the algorithm employed for the 2D feature extraction may be the Oriented FAST and Rotated BRIEF (ORB) algorithm disclosed in “Orb: An efficient alternative to sift or surf” by Rublee et. al. published in IEEE ICCV, 2011. The ORB may provide an advantage over other feature extraction algorithms due to its real-time performance without requiring GPUs and its good invariance to changes in viewpoint and illumination.

[0026] In other embodiments, the algorithm employed for the 2D feature extraction may be one of Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Accelerated-KAZE (A-KAZE). SIFT is disclosed in “Distinctive image features from scaleinvariant keypoints,” by Lowe, published in the International Journal of Computer Vision, 2004. SURF is disclosed in “Surf: Speeded up robust features” by Bay et.aL, published in ECCV, 2006. A-KAZE is disclosed in “Fast explicit diffusion for accelerated features in nonlinear scale spaces” by Alcantarilla et. al., published in British Machine Vision Conference, 2013.

[0027] According to various embodiments, the feature prediction and generation module 104 may be configured to predict three-dimensional (3D) feature points yt by utilizing the historical environment states [y0, ..., yt_t] and generating counterfeit 3D feature points y't from spoofed sensor data. To estimate the normal state xt and yt, the mapping mechanism from vision-based SLAM may be used to convert stable keypoints from the current frame into map points. The mapping mechanism may triangulate the current frame with another adjacent frame. Historical map points may be stored in a map dataset.

[0028] A keypoint may be deemed stable if it satisfies two criteria: the number of frames observing the map point is more than two since its creation, and the ratio of the number of frames that can track the map point to the number of frames that can observe this map point is greater than 25%. If the current frame can find a matching key point for the map point, it may be defined as trackable, and if only the map point can be observed, it may be defined as observable. In addition, the objects in adjacent frames are assumed to move uniformly in a short period of time, thus the pose of the current frame xt can be estimated based on the pose of the previous frame xt_± and related control outputs ut_r. The environment state of the current frame yt may also be predicted in the same method. To determine the spoofed state x't and y't, keypoints extracted in the current frame may be combined with perceived data from spoofed sensors and formulate corresponding strategies for different spatial consistency conflicts. The estimation of xt 1- and yt is based on the distribution of keypoints in the captured images and is not affected by spoofed data. The vision-based SLAM is disclosed in “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras” by Mur-Artal et. al., is published in IEEE Transactions on Robotics, 2017.

[0029] The generation of map points for each frame may occupy substantial computational and storage resources. To mitigate this performance overhead, the ORB-SLAM framework may be used, where the conversion of stable observed keypoints to map points for a frame is performed only when the current frame is identified as a keyframe. The determination of keyframes is typically based on whether the environmental changes surrounding the vehicle are substantial. This approach may significantly mitigate the frequency of calculations required. Moreover, the method has no need for constructing an environment map, as a window mechanism is established wherein only map points generated by continuous keyframes are stored. Based on this way, the growth in storage requirements due to map expansion may be considerably reduced.

[0030] According to various embodiments, the match pair searching module 106 may be configured to may identify keypoint matching pairs from the 2D extracted keypoints zt and 3D predicted points yt or spoofed points y't. To do this, 3D map points in the physical world may be projected to the 2D coordinate in the current frame with an estimated pose, which is formalized as: m?, Mf M? t t ’ M? 1)' With respect to Ic = fx 0' 0' 0 cx fy cy 0 1 m = (m1,m2, mi = (mf,m^ (1)

[0031] (M*, M?, are the 3D map point and (m*, m?} are its projected coordinates in the current frame. Ic is the camera intrinsic matrix containing the focal length along x-axis fx and y-axis fy, and the optical center offset along x-axis cx and y-axis cy.

[0032] The corresponding keypoint kt in the current frame may be identified for each projected map point mi. All map points may be traversed to create a container Kt for each map point, storing all keypoints within a circular area of radius r from the map point. Each extracted keypoint k may be allocated to one level lk of multi-scale image pyramids. Since M is transferred from keypoints, each m also owns a level lm. The keypoint matching follows a rule that level lk. of the selected keypoints kt must be the same as level lm of the corresponding map point or the difference is 1, which is formalized as: Kt = {kj = - krf + (m^ - krf <r2} (2) such that lk. e [lm. — 1, lm., lm. + 1}, for each kj G Kt

[0033] For each keypoint in the container, the feature searching function computes the Hamming distance of descriptors between all keypoints kj G KL and the related map point mi. The keypoint with the shortest distance kbest G KL will form a pair with the map point mt and be put into the corresponding histogram bin according to the rotation angle difference between them. The histogram bins may be twelve containers for storing matching pairs of different angular differences (interval: 30°). The three bins with the most matching pairs (mi, khest) may be selected as inputs to the model, while others may be discarded. The process may be formulated as: (mi, khest) e {(mi, khest)\Bin!dii((mi, khest)) eTop3BinIdx} With respect to / c^^min HDist( / cy. m^, kj e Kt tj z \anglem-angle best\ Binldx(mi, kDest) =------——---- HDist denotes the function of computing the Hamming distance and Binldx denotes the function of the bin index. Top3BinIdx is the set of top three bin indices.

[0034] After the aforementioned procedures, the necessary components for determining the spatio-temporal error may be determined. These necessary components may include the observed data zt, the normal predicted pose xt, and the environment state yt, as well as the deceived pose x't and the deceived environment state y't.

[0035] According to various embodiments, the offset error computation module 108 may be configured to optimize the spatio-temporal error using a nonlinear optimizer. A suitable optimizer may be the g2o, which is based on the Levenberg-Marquardt implementation. The g2o optimizer is disclosed in “G2o: A general framework for graph optimization” by Kummerle et. al., published in IEEE ICRA, 2011. The objective of the optimizer is to minimize all error edges by traversing an error edge for each matching pair, with the intention of discovering an appropriate pose xt and an appropriate environment state yt. Following multiple optimizations, the total mean of the error may be calculated. The matching pair whose error exceeds the threshold as an outlier may be identified as an anomaly.

[0036] As described with respect to FIG. 1, the feature extraction module 102 may output 2D features that represent current observations of the vehicle zt. These features may be used to model the current state of the observed environment.

[0037] The observation zt may also be calculated from the 3D states estimated at the previous time Xf-t and the odometry information at the current time. The 3D estimated state x represents these feature points in world coordinates. The odometry transformation from the last to the current time is captured by a transformation matrix Tt in the Lie group SE(3): se(3) = jr = s] e ir4x4|s e so(3), t e ir3x3} (4)

[0038] The Lie group SE(3) is a specific group, which is a finite-dimensional real smooth manifold, in which the group operations of multiplication and inversion are smooth maps.

[0039] R and s are the 3x3 rotation matrices and 3x1 translation matrices, respectively. The projection of 2D coordinates pt approximate zt using the formula: Pj =^K(Rxt-i+s) (5)

[0040] K denotes fixed camera parameters and d represents the distance to each feature point. Noise in the observations leads to an error et between zt and pt: et = zt - Pt (6)

[0041] To optimize the camera pose, errors from each pair of matched feature points (zk, pk) may be aggregated to formulate a least-squares problem: 1V 1 T = argmm-> zk - -K^Rx^ + s) k=l (7)

[0042] However, in the event of sensor anomaly, the motion transformation is disrupted, and the 3D estimated feature points xt_r are inaccurately projected into incorrect positions, significantly increasing the error in the optimization problem. The distance between the correct feature points zt and their incorrectly projected position p't may become enlarged.

[0043] According to various embodiments, another anomaly detection framework 200 may be provided. The anomaly detection framework 200 may include the anomaly detection framework 100. The anomaly detection framework 200 may include incorporating the LiDAR sensor data into Equation (7). The superscript m herein denotes different LiDAR scan sweeps. Pm represents the LiDAR point cloud captured during the mth sweep, herein also referred to as an observation point cloud. Each Pm may be compared with the LiDAR point cloud from the previous sweep, Pm-1. The visual motion transformation for sweep m is denoted as Tm, and the starting time of this sweep is indicated by tm. For any laser point i received at time the drift between tm and may be interpolated linearly. This is expressed as:

[0044] The projection pm may be calculated using the previous observations Pm-1 motion transformation T™ between the two LiDAR scans: pm = (8) and the (9)

[0045] Here, / (■) represents the function that projects the previous point cloud Pm-1 using the motion transformation T™, which includes adjustments derived from the visual camera data.

[0046] The error e™ between the observation point cloud and its projection may be expressed as: em = pm _ pm (w)

[0047] Consequently, Equation (7) is revised to integrate both camera image data and LiDAR sensor data, yielding a new optimization problem: i T™ = arg min |^ || Pm (H) 1 k=l

[0048] By optimizing this least-squares problem, the positional and orientational discrepancies may be effectively estimated. The derived error e™ may serve as a crucial indicator for detecting various sensor anomalies.

[0049] The anomaly detection framework 200 may utilize the full potential of data fusion from both camera and LiDAR, thereby enhancing the reliability and accuracy of the anomaly detection process. The anomaly detection framework 200 may provide several technical advantages, including enhanced environment robustness, improved detection success rate and increased robustness against adaptive attacks.

[0050] The anomaly detection framework 200 may be able to achieve consistent performance under a broad spectrum of environmental conditions, including variable lighting, fog, and snow. By integrating direct visual methods or camera-lidar fusion data, the anomaly detection framework 200 may enhance the accuracy of the optimization process. This integration significantly boosts the success rate in detecting various sensor anomalies. The anomaly detection framework 200 may demonstrate superior robustness against diverse types of adaptive attacks. For example, camera blinding attacks may be effectively identified using LiDAR data, while feature manipulation attacks may be mitigated through the robust feature extraction capabilities of the data fusion process.

[0051] FIG. 2 shows a block diagram of the anomaly detection framework 200 according to various embodiments. The anomaly detection framework 200 may be configured to perform a method for detecting anomalies in vehicle sensor data. The anomaly detection framework 200 may include a plurality of modules. The plurality of modules may include a point cloud generation module 202, a motion merge module 204, a feature projection module 206, and a second offset error computation module 208.

[0052] The point cloud generation module 202 may be configured to generate a respective observation point cloud based on data captured in each scan performed by a LiDAR sensor. In other words, the point generation module 202 may generate a mth observation point cloud Pm for the data captured at the mth scan, i.e., mth sweep. The point cloud generation module 202 may extract 3D features from the LiDAR data, and may generate the observation point clouds based on the 3D features.

[0053] The motion merge module 204 may be configured to determine an interpolated transformation matrix T™ between starting time of the mth scan and time tt, based on a motion transformation matrix T. Time refers to the scan receiving time close to the second image frame receiving time m + 1. The motion transformation matrix may be determined based on image data, using equation (7) described with respect to FIG. 1. The motion merge module 204 may determine the interpolated transformation matrix using equation (8).

[0054] The feature projection module 206 may be configured to generate a projected point cloud pm based on Pm-1 and T™ between two LiDAR scans. The projected point cloud pm may be obtained using equation (9).

[0055] The offset error computation module 208 may be configured to compare the projected point cloud pm to the observation point cloud Pm, to determine an aggregated error defined by the equation (10). The offset error computation module 208 may further minimize the aggregated error using an optimizer, for example, like used in the offset error computation module 108. The anomaly detection framework 200 may determine presence or absence of anomalies based on the minimized aggregated error.

[0056] FIG. 3 shows a flow diagram of a method 300 for detecting anomalies in vehicle sensor data according to various embodiments. The method 300 may include processes 302, 304, 306, 308, 310, 312 and 314. The process 302 may include receiving image data that includes a first image frame and a second image frame. The second image frame may succeed the first image frame. In other words, the first and second image frames may be successive image frames. The second image frame may come after the first image frame. The first image frame may capture visual information on the surroundings of a vehicle at an earlier time t — 1. The second image frame may capture visual information on the surroundings of the vehicle at a later time t.

[0057] The process 304 may include receiving a first observation point cloud generated based on data captured by a LiDAR sensor on a first scan, and further receiving a second observation point cloud (Pm) generated based on data captured by the LiDAR sensor on a second scan. The second scan may succeed the second image frame. The second scan may occur after a plurality of scans after the first scan. The plurality of scans between the first scan and the second scan may occur during the time interval between the first image frame and the second image frame. The second scan may come after the second image frame. The first observation point cloud and the second observation point cloud may be outputs of the point cloud generation module 202. The first and second scans may be the LiDAR scans that are the closest in time, to the first and second image frames. The process 306 may include determining a motion transformation matrix (T) based on the first image frame and the second image frame. The process 306 may be performed by the anomaly detection method 100. The process 308 may include determining an interpolated transformation matrix (T™) based on the motion transformation matrix, and further based on when the first scan and the second scan occurred. The process 308 may be performed by the motion merge module 204. The process 310 may include generating a projected point cloud (pm) based on the first observation point cloud and further based on the interpolated transformation matrix. The process 310 may be performed by the feature projection module 206. The process 312 may include determining an aggregated error based on the projected point cloud and the second observation point cloud. The process 312 may be performed by the offset error computation module 208. The process 314 may include detecting anomalies in the image data based on the aggregated error.

[0058] The method 300 may be at least partially performed by the anomaly detection framework 100 and the anomaly detection framework 200. The method 300 synthesizes data from multiple sensors to generate a spatio-temporal conflict profile. By doing so, the capability to identify and counter sensor anomalies is improved, ensuring a more secure and reliable autonomous navigation system. This may be a significant step forward in the field of ADS perception robustness against sensor anomalies. By advancing beyond the limitations of existing sensor anomaly detection methods, the method 300 incorporates the strengths of camera and LiDAR data and resolves the challenges posed by single-point failures and variable environmental conditions. Through modelling of spatio-temporal consistency and the innovative fusion of multi-modal sensor data, the method 300 provides high accuracy and strong robustness in the detection of real-world sensor anomalies.

[0059] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, the method 300 may further include generating the first observation point cloud based on data captured by the LiDAR sensor on the first scan, and generating, the second observation point cloud based on data captured by the LiDAR sensor on the second scan. This may be carried out by the point cloud generation module 202.

[0060] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, in the method 300, determining the motion transformation matrix may include extracting features from each of the first image frame and the second image frame, for example, performed by the feature extraction module 102. Determining the motion transformation matrix may further include projecting features from the first image frame to a time when the second image frame was captured, to result in a projected image frame. This may be performed by the feature prediction and generation module 104. Determining the motion transformation matrix may further include minimizing an error between the projected image frame and the second image frame to result in a minimized error. This may be performed by the match pairs searching module 106 and the offset error computation module 108. Determining the motion transformation matrix may further include determining the motion transformation matrix based on the minimized error.

[0061] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, determining the aggregated error may include matching features points in the projected point cloud to corresponding feature points in the second observation point cloud to result in a plurality of pairs of matched feature points. Determining the aggregated error may further include computing, for each pair of matched feature points, a respective error between the pair of matched features points. Determining the aggregated error may further include aggregating the respective errors of the plurality of pairs of matched feature points to result in the aggregated error.

[0062] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, detecting anomalies in the image data may include minimizing the aggregated error, to result in a minimized error, and comparing the minimized error to an error threshold.

[0063] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, detecting anomalies in the image data may further include detecting presence of anomalies based on determining that the minimized error exceeds the error threshold.

[0064] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, minimizing the aggregated error may include minimizing the aggregated error, by a nonlinear optimization solver.

[0065] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the LiDAR sensor may scan at a lower rate than an image frame rate of the image data.

[0066] FIG. 4 shows a simplified block diagram of a data processing apparatus 400 according to various embodiments. The data processing apparatus 400 may include means for carrying out the method 300. For example, the data processing apparatus 400 may include at least one processor 404 and a computer-readable medium 402. The processor 404 and the computer-readable medium 402 may be coupled by coupling line 406, for example, electrically and / or mechanically.

[0067] According to various embodiments, the computer-readable medium 402 may include instructions which, when executed by a computer, cause the computer to carry out the method 300.

[0068] According to various embodiments, a computer program may include instructions which, when the computer program is executed by a computer, cause the computer to carry out the method 300.

[0069] According to various embodiments, the method 300 may be used to detect a spoofing attack on a vehicle.

[0070] FIG. 5 shows a simplified block diagram of an ADAS 500 according to various embodiments. The ADAS 500 may include a camera 502, a LiDAR sensor 504 and the data processing apparatus 400. The camera 502 may be configured to generate the image data. The LiDAR sensor may be configured to capture data for generating each of the first observation point cloud and the second observation point cloud.

[0071] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0072] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0073] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0074] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A computer-implemented method (300) for detecting anomalies in vehicle sensor data, the method comprising:receiving image data comprising a first image frame and a second image frame succeeding the first image frame;receiving a first observation point cloud generated based on data captured by a LiDAR sensor on a first scan, and further receiving a second observation point cloud generated based on data captured by the LiDAR sensor on a second scan succeeding the second image frame the first scan,wherein all scans between the first scan and the second scan occur during a time interval between the first image frame and the second image frame;determining a motion transformation matrix based on the first image frame and the second image frame;determining an interpolated transformation matrix based on the motion transformation matrix, and further based on when the first scan and the second scan occurred;generating a projected point cloud based on the first observation point cloud and further based on the interpolated transformation matrix;determining an aggregated error based on the projected point cloud and the second observation point cloud; anddetecting anomalies in the image data based on the aggregated error.

2. The method (300) of claim 1, further comprising:generating the first observation point cloud based on data captured by the LiDAR sensor on the first scan; andgenerating the second observation point cloud based on data captured by the LiDAR sensor on the second scan.

3. The method (300) of any preceding claim, wherein determining the motion transformation matrix comprisesextracting features from each of the first image frame and the second image frame, projecting features from the first image frame to a time when the second image frame was captured, to result in a projected image frame;minimizing an error between the projected image frame and the second image frame, to result in a minimized error; anddetermining the motion transformation matrix based on the minimized error.

4. The method (300) of any preceding claim, wherein the determining the aggregated error comprisesmatching features points in the projected point cloud to corresponding feature points in the second observation point cloud to result in a plurality of pairs of matched feature points,computing, for each pair of matched feature points, a respective error between the pair of matched features points, andaggregating the respective errors of the plurality of pairs of matched feature points to result in the aggregated error.

5. The method (300) of any preceding claim, wherein detecting anomalies in the image data comprisesminimizing the aggregated error, to result in a minimized error, andcomparing the minimized error to an error threshold.

6. The method (300) of claim 5, wherein detecting anomalies in the image data further comprisesdetecting presence of anomalies based on determining that the minimized error exceeds the error threshold.

7. The method (300) of claim 5, wherein minimizing the aggregated error comprises minimizing the aggregated error, by a nonlinear optimization solver.

8. The method (300) of any preceding claim, wherein the LiDAR sensor scans at a lower rate than an image frame rate of the image data.

9. A computer-readable medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method (300) of any preceding claims.

10. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (300) of any one of claims 1 to 8.

11. Use of the method (300) of any one of claims 1 to 8 to detect a spoofing attack on a vehicle.

12. A data processing apparatus (400) comprising means for carrying out the computer-implemented method (300) of any one of claims 1 to 8.

13. An advanced driver assistance system (500) comprising:a camera (502) configured to generate the image data;a LiDAR sensor (504) configured to capture data for generating each of the first observation point cloud and the second observation point cloud; andthe data processing apparatus (400) of claim 12.

Images