Attack detection method and apparatus for autonomous driving system, device, and storage medium
By fusion detection of the information data of three-dimensional and two-dimensional sensors in the autonomous driving system, and using IoU numerical analysis, the problem of attack detection in the existing technology is not universal and accurate, and the abnormal situation of multiple sensors is realized The accurate identification and positioning of the system improves the security of the system.
Patent Information
- Application Number
- PCT/CN2023/137634
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2023-12-08
- Publication Date
- 2024-11-28
AI Technical Summary
The attack detection methods of existing autonomous driving systems are not universal and have low accuracy, and cannot effectively detect abnormal situations of multiple sensors, which may lead to wrong driving control strategies and threaten the lives and safety of other vehicles and pedestrians.
A method is adopted to obtain the information data of the target area through the three-dimensional sensor and the two-dimensional sensor respectively, and input it into the pre-trained detection model for fusion detection, calculate the IoU value to locate the attacked sensor, and use the three-dimensional sensor and the two-dimensional sensor. The time and space correlations are used to detect attacks to improve the accuracy of detection.
The accuracy and universality of multiple sensor attack detection of autonomous driving systems can be realized, and abnormal situations of three-dimensional and two-dimensional sensors can be effectively identified, which improves safety and avoids potential dangers caused by inaccurate detection.
Smart Images

Figure CN2023137634_28112024_PF_FP_ABST
Abstract
Description
Attack detection method, device, equipment and storage medium for autonomous driving system Technical Field
[0001] The present application relates to the field of sensor detection technology, and in particular to an attack detection method, device, equipment and storage medium for an autonomous driving system. Background Art
[0002] Autonomous driving technology is a comprehensive approach that integrates cutting-edge technologies such as artificial intelligence, sensing, mapping, and computing. Its functionality relies on onboard positioning sensor systems. These systems primarily consist of devices such as lidar, visual cameras, millimeter-wave radar, and the Global Positioning System (GPS). These systems provide the autonomous vehicle's planning and decision-making modules with a wealth of positioning data, including information such as vehicle speed, attitude, and position. The safety of autonomous vehicle route planning and decision-making relies on the safety of these positioning sensor systems. Any anomaly in the positioning sensor system could result in the sensor acquiring erroneous positioning information, leading to incorrect driving control strategies and threatening the safety of other vehicles and the lives of drivers and pedestrians.
[0003] At present, the main method for detecting sensor attacks in unmanned driving technology is to combine it with a specific single-sensor target detection algorithm and perform attack detection on this basis. However, this method is not universal. For example, the attack detection method paired with a target detection algorithm based on lidar cannot be applied to attack detection scenarios using a target detection algorithm based on camera. In addition, the attack detection results of the detection method for a single type of sensor are not very accurate.
[0004] Summary of the Invention
[0005] In view of this, the present application provides an attack detection method, device, equipment and storage medium for an autonomous driving system to solve the problem that the existing attack detection methods for autonomous driving systems are not universal and have low accuracy.
[0006] To solve the above technical problems, a technical solution adopted in the present application is: to provide an attack detection method for an autonomous driving system, which includes: respectively obtaining three-dimensional information data and two-dimensional information data of a target area using a three-dimensional sensor and a two-dimensional sensor; respectively inputting the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a two-dimensional detection model to obtain a first three-dimensional detection frame and a first two-dimensional detection frame for each target object in the target area; inputting the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame for each target object in the target area, and performing a coordinate system change on the second three-dimensional detection frame to obtain a second two-dimensional detection frame; calculating a first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculating a second IoU value between the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array; performing attack detection based on the first array and the second array and locating the attacked sensor.
[0007] As a further improvement of the present application, before inputting the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model, it also includes: performing a chi-square comparison on the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment to obtain a chi-square comparison value; judging whether the chi-square comparison value exceeds a preset threshold; when the chi-square comparison value exceeds the preset threshold, performing a histogram matching on the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment to obtain enhanced two-dimensional information data at the current moment, and replacing the two-dimensional information data at the current moment; when the chi-square comparison value does not exceed the preset threshold, maintaining the two-dimensional information data at the current moment.
[0008] As a further improvement of the present application, the coordinate system of the second three-dimensional detection frame is changed to obtain a second two-dimensional detection frame, including: obtaining the three-dimensional center point coordinates of the second three-dimensional detection frame; using the three-dimensional center point coordinates to confirm the homogeneous coordinates of four points on the second three-dimensional detection frame that are in the same plane as the three-dimensional center point; using the pre-acquired camera projection matrix, camera rotation matrix, sensor to camera coordinate system rotation matrix and homogeneous coordinate calculation to obtain four two-dimensional coordinate points; and using the four two-dimensional coordinate points to construct a second two-dimensional detection frame.
[0009] As a further improvement of the present application, the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object is calculated to obtain a first array, including: confirming the target first two-dimensional detection frame and the target second two-dimensional detection frame corresponding to each target object; respectively calculating the first area and the second area of the target first two-dimensional detection frame and the target second two-dimensional detection frame, and the third area of the overlapping area of the target first two-dimensional detection frame and the target second two-dimensional detection frame; using the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object; and constructing the first array using the first IoU values corresponding to all target objects.
[0010] As a further improvement of the present application, the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object is calculated to obtain a second array, including: confirming the target first three-dimensional detection frame and the target second three-dimensional detection frame corresponding to each target object; calculating the first volume and the second volume of the target first three-dimensional detection frame and the target second three-dimensional detection frame respectively; calculating the bottom surface area of the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame; confirming the height of the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame; using the bottom surface area and the height to calculate the third volume of the overlapping area; using the first volume, the second volume, and the third volume to calculate the second IoU value corresponding to each target object; and constructing a second array using the second IoU values corresponding to all target objects.
[0011] As a further improvement of the present application, attack detection and locating the attacked sensor based on the first array and the second array include: respectively determining whether there are outliers in the first array and the second array; when there are outliers in the first array and no outliers in the second array, confirming that the three-dimensional sensor is attacked; when there are no outliers in the first array and there are outliers in the second array, confirming that the two-dimensional sensor is attacked; when there are outliers in both the first array and the second array, confirming that the three-dimensional sensor and / or the two-dimensional sensor is attacked; when there are outliers in both the first array and the second array, confirming that the three-dimensional sensor and the two-dimensional sensor are attacked; when there are no outliers in both the first array and the second array, confirming that the three-dimensional sensor and the two-dimensional sensor are not attacked.
[0012] As a further improvement of this application, the three-dimensional sensor includes a lidar, the two-dimensional sensor includes a camera, the three-dimensional detection model is constructed based on the PointPillar algorithm, the two-dimensional detection model is constructed based on the YOLOv3 algorithm, and the fusion detection model is constructed based on the AVOD algorithm.
[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an attack detection device for an autonomous driving system, which includes: an acquisition module, used to respectively use a three-dimensional sensor and a two-dimensional sensor to obtain three-dimensional information data and two-dimensional information data of a target area; a first detection module, used to input the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a two-dimensional detection model, respectively, to obtain a first three-dimensional detection frame and a first two-dimensional detection frame for each target object in the target area; a second detection module, used to input the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model, to obtain a second three-dimensional detection frame for each target object in the target area, and to perform a coordinate system transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame; a calculation module, used to calculate a first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculate a second IoU value between the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array; and an analysis module, used to perform attack detection and locate the attacked sensor based on the first array and the second array.
[0014] In order to solve the above technical problems, another technical solution adopted in this application is: to provide a computer device, which includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the steps of the attack detection method of the autonomous driving system as described in any one of the above items.
[0015] In order to solve the above technical problems, another technical solution adopted in this application is: to provide a storage medium storing program instructions of an attack detection method for an autonomous driving system that can implement any of the above items.
[0016] The beneficial effects of the present application are as follows: the attack detection method of the autonomous driving system of the present application performs target detection on the three-dimensional information data and the two-dimensional information data obtained by the three-dimensional sensor and the two-dimensional sensor respectively to obtain a first three-dimensional detection frame and a first two-dimensional detection frame, and performs fusion target detection on the three-dimensional information data and the two-dimensional information data to obtain a second three-dimensional detection frame, and performs coordinate transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame, and then performs IoU numerical calculation on the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each object to obtain a first array, and performs IoU numerical calculation on the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each object to obtain a second array, and analyzes the first array and the second array to confirm whether the sensor is attacked. It uses the temporal and spatial correlation of the three-dimensional sensor and the two-dimensional sensor to perform attack detection, thereby improving the accuracy of attack detection, and can realize attack detection by using the three-dimensional sensor and the two-dimensional sensor as the basic configuration of the autonomous driving vehicle, and is no longer limited to attack detection by the target detection algorithm using one of the sensors as the main one, and has higher universality. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] FIG1 is a schematic diagram of a flow chart of an attack detection method for an autonomous driving system according to an embodiment of the present invention;
[0018] FIG2 is a schematic diagram of a second three-dimensional detection frame according to an embodiment of the present invention;
[0019] FIG3 is a schematic diagram of overlapping areas of two-dimensional detection frames according to an embodiment of the present invention;
[0020] FIG4 is a schematic diagram of functional modules of an attack detection device for an automatic driving system according to an embodiment of the present invention;
[0021] FIG5 is a schematic structural diagram of a computer device according to an embodiment of the present invention;
[0022] FIG6 is a schematic structural diagram of a storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0024] The terms "first," "second," and "third" in this application are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.
[0025] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0026] FIG1 is a flow chart of an attack detection method for an autonomous driving system according to an embodiment of the present invention. It should be noted that the method of the present invention is not limited to the flow sequence shown in FIG1 if substantially the same results are achieved. As shown in FIG1 , the attack detection method for an autonomous driving system includes the following steps:
[0027] Step S101: using a three-dimensional sensor and a two-dimensional sensor to obtain three-dimensional information data and two-dimensional information data of a target area respectively.
[0028] In this embodiment, the three-dimensional sensor is preferably a laser radar. It should be noted that the three-dimensional sensor in the embodiment of the present invention is not limited to a laser radar, and other sensor devices that can be used to obtain three-dimensional information data are also included in the scope of protection of the present invention. The two-dimensional sensor is preferably a camera. It should be noted that the two-dimensional sensor in the embodiment of the present invention is not limited to a camera, and other sensor devices that can be used to obtain two-dimensional information data are also included in the scope of protection of the present invention.
[0029] Specifically, this embodiment uses the 3D and 2D sensors on the vehicle to simultaneously collect 3D and 2D information data from the target area when the vehicle is driven using the autonomous driving system. It should be noted that during attack detection, the detection range is limited to a target area, which is the overlapping area of the 3D and 2D sensor fields of view. For example, for a lidar and camera, this overlapping area is the lidar's FOV (Field of View). By projecting the camera's image size onto the lidar's field of view, the 3D information data collected by the lidar and the 2D information data collected by the camera are obtained.
[0030] Step S102: input the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a two-dimensional detection model respectively to obtain a first three-dimensional detection frame and a first two-dimensional detection frame of each target object in the target area.
[0031] The 3D detection model is built based on the PointPillar algorithm. Specifically, the 3D sensor uses the PointPillars (3D Object Proposal Generation and Detection from Point Cloud) algorithm to detect 3D targets and obtain the first 3D target detection frame. The input data is the original data point cloud data of the 3D sensor, that is, the 3D information data, in the format of [x, y, z, intensity]. The original feature of the 3D sensor is a point cloud set, which can be represented as a vector (64-line laser radar, each line scans 1800 points per circle) Where X represents the horizontal coordinate of the point cloud, Y represents the vertical coordinate of the point cloud, Z represents the height of the point cloud, and intensity represents the reflection intensity of the point cloud. Using a two-stage approach, using PointNet++ as the backbone network, the segmentation task is first completed to determine the label of each 3D point. For each point classified as the foreground, the feature is used to generate a box. The box is then optimized. The target object detection result is expressed as a vector Indicates. Where x lidar =[X v Y v Z v ] T Indicates the center point value of the target object in the three-dimensional sensor coordinate system, [L, W, H] T Indicates the length, width and height of the first 3D detection box.
[0032] Among them, the two-dimensional detection model is built based on the YOLOv3 algorithm. Specifically, the YOLOv3 algorithm is used to perform two-dimensional target detection on the two-dimensional information data obtained by the two-dimensional sensor to obtain the target center point. For example, taking the camera as an example, the collected picture is input into the two-dimensional detection model in RGB format. The two-dimensional detection model calls the deep convolutional neural network algorithm to extract features from the RGB image. Common structures in the neural network algorithm include convolution layer, pooling layer, activation layer, dropout layer, BN (batch normalization) layer, fully connected layer, etc. Finally, the features extracted from the picture can effectively describe the information of the target object. The input data format is the RGB image of the camera, and the output is the center point coordinates and the detection frame width (width) and height (height) in the image coordinate system, in the format of where X camera Indicates the value of the x direction in the image coordinate system, Y camera Represents the y-direction value, w represents the width of the first two-dimensional detection frame, and h represents the height of the first two-dimensional detection frame. After obtaining the first two-dimensional detection frame, the coordinates of the four points of the first two-dimensional detection frame can be further obtained as follows: X camera,1 ,=X camera,3 ,=X camera -w / 2; X camera,2 ,=X camera,4 ,=X camera +w / 2; Y camera,1 ,=Y camera,2 ,=Y camera +h / 2; Y camera,3 ,=Y camera,4 ,=Y camera -h / 2;
[0033] where X camera,i ,i=1,2,3,4 respectively represent the coordinate values of the upper left corner, upper right corner, lower left corner and lower right corner of the first two-dimensional detection frame in the x direction of the image coordinate system. camera,i ,i=1, 2, 3, 4 respectively represent the coordinate values of the upper left corner, upper right corner, lower left corner, and lower right corner of the first two-dimensional detection box in the y direction of the image coordinate system.
[0034] Step S103: Input the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame of each target object in the target area, and perform a coordinate system transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame.
[0035] Specifically, the fusion detection model is built based on the AVOD algorithm (Joint 3D Proposal Generation and Object Detection from View Aggregation). After obtaining 3D and 2D information data, the 3D and 2D information data are input into the AVOD algorithm for target detection, resulting in a second 3D detection frame. The AVOD algorithm fuses 3D and 2D information data. When the 3D information data is point cloud data acquired by a lidar, the AVOD algorithm only uses the top and front views of the point cloud, which reduces the amount of computation without losing too much information. A 3D candidate region is then generated, and the features and candidate regions are fused to output the final second 3D detection frame.
[0036] Furthermore, in order to improve the defense capability of the two-dimensional sensor, in some embodiments, before step S103, the following steps are further included:
[0037] 1. Perform a chi-square comparison on the two-dimensional information data of the previous moment and the two-dimensional information data of the current moment to obtain the chi-square comparison value.
[0038] 2. Determine whether the chi-square comparison value exceeds the preset threshold.
[0039] 3. When the chi-square comparison value exceeds the preset threshold, the two-dimensional information data at the current moment is histogram matched with the two-dimensional information data at the previous moment to obtain the enhanced two-dimensional information data at the current moment and replace the two-dimensional information data at the current moment.
[0040] 4. When the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.
[0041] In this embodiment, the purpose of image enhancement is to improve the fusion detection model's defense against 2D sensor attacks. A chi-square comparison is performed between the 2D information data at the previous moment and the 2D information data at the current moment to obtain a chi-square comparison value. A determination is then made as to whether the chi-square comparison value exceeds a preset threshold. If the chi-square comparison value exceeds the preset threshold, a histogram matching is performed between the current and previous 2D information data to obtain enhanced 2D information data for the current moment, which replaces the current 2D information data. If the chi-square comparison value does not exceed the preset threshold, the current 2D information data is maintained.
[0042] Furthermore, in step S103, the step of performing coordinate system transformation on the second three-dimensional detection frame to obtain the second two-dimensional detection frame specifically includes:
[0043] 1. Get the 3D center coordinates of the second 3D detection frame.
[0044] 2. Use the 3D center point coordinates to confirm the homogeneous coordinates of the four points on the second 3D detection frame that are in the same plane as the 3D center point.
[0045] 3. Use the pre-acquired camera projection matrix, camera rotation matrix, sensor-to-camera coordinate system rotation matrix and homogeneous coordinates to calculate the four two-dimensional coordinate points.
[0046] 4. Use the four two-dimensional coordinate points to construct a second two-dimensional detection frame.
[0047] Specifically, in this embodiment, four corner points (including only x coordinates and y coordinates) and two heights are used to describe a second three-dimensional detection frame. As shown in FIG2 , the second three-dimensional detection frame is represented by vectors Indicates that c i =[x ci y ci ] T , i=1, 2, 3, 4 represent the x- and y-direction coordinates of the four vertices of the second three-dimensional detection frame, h1 represents the height of the bottom surface of the second three-dimensional detection frame from the plane formed by the x-axis and the y-axis in the three-dimensional coordinate system, and h2 represents the height of the top surface of the second three-dimensional detection frame from the plane formed by the x-axis and the y-axis in the three-dimensional coordinate system.
[0048] Specifically, after obtaining the second three-dimensional detection frame, the second three-dimensional detection frame is projected into the image coordinate system to obtain the second two-dimensional detection frame. Please refer to Figure 2. First, the vector form of the second three-dimensional detection frame is converted into the conventional three-dimensional detection frame format Y 3D融合检测 =[X v Y v Z v ] T , where: Z v =(h2-h1) / 2; X v =x c1 -x c2 ; Y v =y c1 -y c2 ;
[0049] Among them, x ci Indicates c i The x-axis coordinate of the point, y ci Indicates c i The y-axis coordinate of the point, i=1, 2, 3, 4, [X v Y v Z v ] T Indicates the coordinates of the center point of the second 3D detection box.
[0050] Then, the coordinate system of the four points near the center of the second 3D detection frame (points A, B, C, and D in Figure 2) is transformed in sequence to obtain four points in the image coordinate system. The specific transformation process is as follows: 点A齐次坐标 =[X v,1 ,Y v,1 ,Z v,1 ] T =[X v -(x c2 -x c1 ) / 2,Y v ,h2]; Y 点B齐次坐标 =[X v,2 ,Y v,2 ,Z v,2 ] T =[X v +(x c2 -x c1 ) / 2,Y v ,h2]; Y 点c齐次坐标 =[X v,3 ,Y v,3 ,Z v,3 ] T =[X v -(x c2 -x c1 ) / 2,Y v ,h1]; Y 点D齐次坐标 =[X v,4 ,Y v,4 ,Z v,4 ] T =[X v +(x c2 -x c1 ) / 2,Y v ,h1]; Y=P*R*Tr v elo t o c am*Y 点n齐次坐标 , n=A, B, C, D;
[0051] Among them, Y represents the coordinate of the second two-dimensional detection frame, Y 点n齐次坐标 Represents the homogeneous vector of the target object, P is the camera projection matrix, R is the camera rotation matrix, Tr v elo t o c am is the rotation matrix from the 3x4 three-dimensional coordinate system to the camera coordinate system.
[0052] Step S104: Calculate a first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculate a second IoU value between the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array.
[0053] Specifically, the IoU value can be used to characterize the inconsistency between the two sets of detection frames. The IoU value is calculated using the first two-dimensional detection frame and the second two-dimensional detection frame to obtain a first IoU value that characterizes the inconsistency between the first two-dimensional detection frame and the second two-dimensional detection frame. Multiple target objects correspond to multiple first IoU values, which are constructed into a first array. The IoU value is calculated using the first three-dimensional detection frame and the second three-dimensional detection frame to obtain a second IoU value that characterizes the inconsistency between the first three-dimensional detection frame and the second three-dimensional detection frame. Multiple target objects correspond to multiple second IoU values, which are constructed into a second array. It should be noted that the two sets of detection frames correspond to the same target area, and the target object of the target area is constant. Therefore, the number of elements in the first array and the second array is equal, and each target object has a corresponding element in the first array or the second array.
[0054] Furthermore, in step S104, the step of calculating the first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain the first array specifically includes:
[0055] 1. Confirm the target first two-dimensional detection frame and target second two-dimensional detection frame corresponding to each target object.
[0056] Specifically, since there are often multiple target objects in the scene, there will be multiple detection frames. Therefore, when confirming the target first and second two-dimensional detection frame corresponding to each target object, first select a first two-dimensional detection frame as the target first two-dimensional detection frame, and then calculate the Euclidean distance between the center point of the target first two-dimensional detection frame and the center point of each second two-dimensional detection frame, and select the second two-dimensional detection frame with the smallest Euclidean distance as the target second two-dimensional detection frame.
[0057] 2. Calculate the first area, the second area of the first two-dimensional detection frame of the target, the second two-dimensional detection frame of the target, and the third area of the overlapping area of the first two-dimensional detection frame of the target and the second two-dimensional detection frame of the target.
[0058] Specifically, by obtaining the coordinates of the four vertices of the two-dimensional detection frame, the first area and the second area of the target first two-dimensional detection frame and the target second two-dimensional detection frame are calculated based on the coordinates of the four vertices. The calculation formula is as follows: S A =|x a2 -x a1 |×|y a2 -ya1 |; S B =|x b2 -x b1 |×|y b2 -y b1 |;
[0059] Among them, S A Represents the first area, A1(x a1 ,y a1 )、B1(x a1 ,y a2 )、C1(x a2 ,y a1 )、D1(x a2 ,y a2 ) represents the coordinates of the four vertices of the first two-dimensional detection box of the target. S B Indicates the second area, A2(x b1 ,y b1 )、B2(x b1 ,y b2 )、C2(x b2 ,y b1 )、D2(x b2 ,y b2 ) represents the coordinates of the four vertices of the second two-dimensional detection box of the target.
[0060] Take Figure 3 as an example to illustrate that when there is an overlapping area between two detection frames, the overlapping area is also a rectangular frame. The four vertices of the rectangular frame can be obtained according to the four vertex coordinates of the overlapping target first two-dimensional detection frame and the target second two-dimensional detection frame. The coordinate of its upper left corner is A2(x b1 ,y b1 ), the coordinate of the lower left corner is E(x b1 ,y a2 ), the coordinate of the upper right corner is F(x a2 ,y b1 ), the coordinate of the lower right corner is D1(x a2 ,y a2 ). Then, the third area of the overlapping area is calculated using the coordinates of the four vertices of the overlapping area.
[0061] 3. Use the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object.
[0062] Specifically, the first IoU value=the third area / (the first area+the second area-the third area).
[0063] 4. Construct the first array using the first IoU values corresponding to all target objects.
[0064] Specifically, the first array is represented as: I = [i1, i2, ..., in ], n is the number of the first IoU values.
[0065] Furthermore, in step S104, the step of calculating the second IoU value between the first 3D detection frame and the second 3D detection frame corresponding to each target object to obtain a second array specifically includes:
[0066] 1. Confirm the target first 3D detection frame and target second 3D detection frame corresponding to each target object.
[0067] Specifically, the target first 3D detection frame and the target second 3D detection frame are also confirmed by using the Euclidean distance between the center point coordinates of the first 3D detection frame and the second 3D detection frame.
[0068] 2. Calculate the first volume and second volume of the first 3D detection frame and the second 3D detection frame of the target respectively.
[0069] Specifically, the length, width and height of the three-dimensional detection frame can be determined according to the coordinates of the eight vertices of the three-dimensional detection frame, and the volume of the three-dimensional detection frame can be calculated using the length, width and height.
[0070] 3. Calculate the bottom area of the overlapping area between the first 3D detection frame of the target and the second 3D detection frame of the target.
[0071] 4. Confirm the height of the overlapping area between the first 3D detection frame of the target and the second 3D detection frame of the target.
[0072] 5. Use the bottom area and height to calculate the third volume of the overlapping area.
[0073] It should be understood that the overlapping area of the first and second 3D detection frames is also a cubic structure, and its volume can be calculated by multiplying the base area by the height. Since the target objects corresponding to the first and second 3D detection frames are both located on the ground, that is, the bottom surfaces of the first and second 3D detection frames are in the same plane, the vertex coordinates of the bottom surface of the overlapping area of the first and second 3D detection frames can be determined based on the four vertex coordinates of the bottom surface of the first and second 3D detection frames and the four vertex coordinates of the bottom surface of the second 3D detection frame. The base area of the overlapping area can then be calculated based on the vertex coordinates of the bottom surface of the overlapping area. This method is the same as the method for calculating the overlapping area in the two-dimensional detection frame and will not be repeated here. As for the height of the overlapping area, since the target objects corresponding to the first and second 3D detection frames are both located on the ground, that is, the bottom surfaces of the first and second 3D detection frames are in the same plane, the height of the overlapping area of the first and second 3D detection frames is the smaller of the height values in the first and second 3D detection frames. After obtaining the base area and height of the overlapping area of the first and second 3D detection frames, the third volume of the overlapping area can be calculated.
[0074] 6. Use the first volume, second volume, and third volume to calculate the second IoU value corresponding to each target object.
[0075] Specifically, the second IoU value = the third volume / (first volume + second volume - third volume)
[0076] 7. Construct the second array using the second IoU values corresponding to all target objects.
[0077] Specifically, the first array is represented as: U = [u1, u2, ..., u n ], n is the number of the second IoU values.
[0078] Step S105: Perform attack detection based on the first array and the second array and locate the attacked sensor.
[0079] Specifically, after obtaining the first array and the second array, an inconsistency detection is performed on the two arrays to confirm whether the sensor is attacked and to locate whether the attacked three-dimensional sensor or two-dimensional sensor.
[0080] Furthermore, step S105 specifically includes:
[0081] 1. Determine whether there are outliers in the first array and the second array respectively.
[0082] 2. When there are outliers in the first array and no outliers in the second array, it is determined that the three-dimensional sensor is attacked.
[0083] 3. When there is no outlier in the first array and there is an outlier in the second array, it is determined that the two-dimensional sensor is attacked.
[0084] 4. When outliers exist in both the first array and the second array, it is determined that the three-dimensional sensor and / or the two-dimensional sensor is attacked.
[0085] 5. When there are no outliers in the first array and the second array, it is confirmed that the three-dimensional sensor and the two-dimensional sensor are not attacked.
[0086] The attack detection method for an autonomous driving system according to an embodiment of the present invention performs target detection on three-dimensional information data and two-dimensional information data acquired by a three-dimensional sensor and a two-dimensional sensor, respectively, to obtain a first three-dimensional detection frame and a first two-dimensional detection frame, performs fusion target detection on the three-dimensional information data and the two-dimensional information data to obtain a second three-dimensional detection frame, performs coordinate transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame, then performs an Intersection over Union (IoU) calculation on the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each object to obtain a first array, performs an IoU calculation on the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each object to obtain a second array, and analyzes the first array and the second array to determine whether the sensor is under attack. The method utilizes the temporal and spatial correlation between three-dimensional and two-dimensional sensors for attack detection, thereby improving the accuracy of attack detection. Furthermore, attack detection can be achieved using three-dimensional and two-dimensional sensors, which are basic configurations of autonomous vehicles, and is no longer limited to target detection algorithms that utilize only one sensor as the primary sensor, thus having higher universality.
[0087] Figure 4 is a schematic diagram of the functional modules of an attack detection device for an autonomous driving system according to an embodiment of the present invention. As shown in Figure 4 , the attack detection device 20 for an autonomous driving system includes an acquisition module 21 , a first detection module 22 , a second detection module 23 , a calculation module 24 , and an analysis module 25 .
[0088] An acquisition module 21 is configured to acquire three-dimensional information data and two-dimensional information data of a target area using a three-dimensional sensor and a two-dimensional sensor respectively;
[0089] The first detection module 22 is configured to input the 3D information data and the 2D information data into a pre-trained 3D detection model and a pre-trained 2D detection model, respectively, to obtain a first 3D detection frame and a first 2D detection frame for each target object in the target area;
[0090] The second detection module 23 is configured to input the 3D information data and the 2D information data into a pre-trained fusion detection model to obtain a second 3D detection frame for each target object in the target area, and perform a coordinate system transformation on the second 3D detection frame to obtain a second 2D detection frame;
[0091] a calculation module 24 configured to calculate a first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain a first array, and to calculate a second IoU value between the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array;
[0092] The analysis module 25 is configured to perform attack detection based on the first array and the second array and locate the attacked sensor.
[0093] Optionally, before the second detection module 23 executes the operation of inputting the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model, it is also used to: perform a chi-square comparison on the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment to obtain a chi-square comparison value; determine whether the chi-square comparison value exceeds a preset threshold; when the chi-square comparison value exceeds the preset threshold, perform a histogram matching on the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment to obtain enhanced two-dimensional information data at the current moment, and replace the two-dimensional information data at the current moment; when the chi-square comparison value does not exceed the preset threshold, maintain the two-dimensional information data at the current moment.
[0094] Optionally, the second detection module 23 performs a coordinate system change on the second three-dimensional detection frame to obtain a second two-dimensional detection frame, specifically including: obtaining the three-dimensional center point coordinates of the second three-dimensional detection frame; using the three-dimensional center point coordinates to confirm the homogeneous coordinates of four points on the second three-dimensional detection frame that are in the same plane as the three-dimensional center point; using the pre-acquired camera projection matrix, camera rotation matrix, sensor to camera coordinate system rotation matrix and homogeneous coordinate calculation to obtain four two-dimensional coordinate points; and using the four two-dimensional coordinate points to construct a second two-dimensional detection frame.
[0095] Optionally, the calculation module 24 performs an operation of calculating the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain the first array, specifically including: confirming the target first two-dimensional detection frame and the target second two-dimensional detection frame corresponding to each target object; respectively calculating the first area and the second area of the target first two-dimensional detection frame and the target second two-dimensional detection frame, and the third area of the overlapping area of the target first two-dimensional detection frame and the target second two-dimensional detection frame; using the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object; and constructing the first array using the first IoU values corresponding to all target objects.
[0096] Optionally, the calculation module 24 performs an operation of calculating the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array, specifically including: confirming the target first three-dimensional detection frame and the target second three-dimensional detection frame corresponding to each target object; calculating the first volume and the second volume of the target first three-dimensional detection frame and the target second three-dimensional detection frame respectively; calculating the bottom surface area of the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame; confirming the height of the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame; using the bottom surface area and the height to calculate the third volume of the overlapping area; using the first volume, the second volume, and the third volume to calculate the second IoU value corresponding to each target object; and constructing a second array using the second IoU values corresponding to all target objects.
[0097] Optionally, the analysis module 25 performs attack detection based on the first array and the second array and locates the attacked sensor, specifically including: determining whether there are outliers in the first array and the second array respectively; when there are outliers in the first array and no outliers in the second array, confirming that the three-dimensional sensor is attacked; when there are no outliers in the first array and there are outliers in the second array, confirming that the two-dimensional sensor is attacked; when there are outliers in both the first array and the second array, confirming that the three-dimensional sensor and / or the two-dimensional sensor is attacked; when there are no outliers in both the first array and the second array, confirming that the three-dimensional sensor and the two-dimensional sensor are attacked.
[0098] Optionally, the three-dimensional sensor includes a lidar, the two-dimensional sensor includes a camera, the three-dimensional detection model is constructed based on a PointPillar algorithm, the two-dimensional detection model is constructed based on a YOLOv3 algorithm, and the fusion detection model is constructed based on an AVOD algorithm.
[0099] For other details about the technical solutions for implementing each module in the attack detection device of the autonomous driving system in the above embodiment, please refer to the description of the attack detection method of the autonomous driving system in the above embodiment, which will not be repeated here.
[0100] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.
[0101] Please refer to Figure 5, which is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. As shown in Figure 5, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. The memory 32 stores program instructions. When the program instructions are executed by the processor 31, the processor 31 performs the steps of the attack detection method for the autonomous driving system described in any of the above embodiments.
[0102] The processor 31 may also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip having signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.
[0103] Refer to Figure 6, which is a schematic diagram of the structure of the storage medium of an embodiment of the present invention. The storage medium of the embodiment of the present invention stores program instructions 41 that can implement the attack detection method of the above-mentioned autonomous driving system, wherein the program instructions 41 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, or a tablet.
[0104] In the several embodiments provided in this application, it should be understood that the disclosed computer devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0105] In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for detecting an attack on an autonomous driving system, characterized in that: It includes: Using a three-dimensional sensor and a two-dimensional sensor to obtain three-dimensional information data and two-dimensional information data of the target area respectively; Inputting the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a two-dimensional detection model respectively to obtain a first three-dimensional detection frame and a first two-dimensional detection frame of each target object in the target area; Inputting the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame of each target object in the target area, and performing a coordinate system transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame; Calculate a first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculate a second IoU value between the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array; Attack detection is performed based on the first array and the second array, and the attacked sensor is located.
2. The attack detection method for an automatic driving system according to claim 1, characterized in that: Before inputting the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model, the method further includes: Perform a chi-square comparison on the two-dimensional information data of the previous moment and the two-dimensional information data of the current moment to obtain a chi-square comparison value; Determining whether the chi-square comparison value exceeds a preset threshold; When the chi-square comparison value exceeds a preset threshold, the two-dimensional information data at the current moment is histogram matched with the two-dimensional information data at the previous moment to obtain enhanced two-dimensional information data at the current moment, and replace the two-dimensional information data at the current moment; When the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.
3. The attack detection method for an automatic driving system according to claim 1, characterized in that: The step of changing the coordinate system of the second three-dimensional detection frame to obtain a second two-dimensional detection frame includes: Obtaining the three-dimensional center point coordinates of the second three-dimensional detection frame; Using the coordinates of the three-dimensional center point, confirming the homogeneous coordinates of four points on the second three-dimensional detection frame that are in the same plane as the three-dimensional center point; Calculate four two-dimensional coordinate points using the pre-acquired camera projection matrix, camera rotation matrix, sensor-to-camera coordinate system rotation matrix and the homogeneous coordinates; The second two-dimensional detection frame is constructed using the four two-dimensional coordinate points.
4. The attack detection method for an automatic driving system according to claim 1, characterized in that: The first IoU value between the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object is calculated to obtain a first array, including: Confirming a target first two-dimensional detection frame and a target second two-dimensional detection frame corresponding to each target object; Calculating respectively a first area and a second area of the first two-dimensional detection frame of the target, the second two-dimensional detection frame of the target, and a third area of an overlapping area of the first two-dimensional detection frame of the target and the second two-dimensional detection frame of the target; Calculate a first IoU value corresponding to each target object using the first area, the second area, and the third area; The first array is constructed using the first IoU values corresponding to all target objects.
5. The attack detection method for an automatic driving system according to claim 1, characterized in that: The calculating the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array includes: Confirming a target first three-dimensional detection frame and a target second three-dimensional detection frame corresponding to each target object; Calculating a first volume and a second volume of a first three-dimensional detection frame of the target and a second three-dimensional detection frame of the target respectively; Calculating the bottom surface area of the overlapping region of the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target; Determining a height of an overlapping area between the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target; Calculate the third volume of the overlapping area using the bottom surface area and the height; Calculate a second IoU value corresponding to each target object using the first volume, the second volume, and the third volume; The second array is constructed using the second IoU values corresponding to all target objects.
6. The attack detection method for an automatic driving system according to claim 1, characterized in that: The performing attack detection based on the first array and the second array and locating the attacked sensor includes: Determine whether the first array and the second array have outliers respectively; When there are outliers in the first array and there are no outliers in the second array, confirming that the three-dimensional sensor is attacked; When there is no outlier in the first array and there is an outlier in the second array, confirming that the two-dimensional sensor is attacked; When outliers exist in both the first array and the second array, it is confirmed that the three-dimensional sensor and / or the two-dimensional sensor is attacked; When there are no outliers in the first array and the second array, it is confirmed that the three-dimensional sensor and the two-dimensional sensor are not attacked.
7. The attack detection method for an automatic driving system according to claim 1, characterized in that: The three-dimensional sensor includes a laser radar, the two-dimensional sensor includes a camera, the three-dimensional detection model is constructed based on a PointPillar algorithm, the two-dimensional detection model is constructed based on a YOLOv3 algorithm, and the fusion detection model is constructed based on an AVOD algorithm.
8. An attack detection device for an automatic driving system, characterized in that: It includes: An acquisition module, used to respectively acquire three-dimensional information data and two-dimensional information data of the target area using a three-dimensional sensor and a two-dimensional sensor; A first detection module, used to input the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a two-dimensional detection model, respectively, to obtain a first three-dimensional detection frame and a first two-dimensional detection frame of each target object in the target area; A second detection module is used to input the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame of each target object in the target area, and perform a coordinate system transformation on the second three-dimensional detection frame to obtain a second two-dimensional detection frame; a calculation module, configured to calculate a first IoU value between a first two-dimensional detection frame and a second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculate a second IoU value between a first three-dimensional detection frame and a second three-dimensional detection frame corresponding to each target object to obtain a second array; An analysis module is used to perform attack detection based on the first array and the second array and locate the attacked sensor.
9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the steps of the attack detection method for the autonomous driving system as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: Program instructions are stored that can implement an attack detection method for an autonomous driving system as described in any one of claims 1-7.
Citation Information
Patent Citations
Target detection and tracking method based on two-dimensional picture and three-dimensional point cloud fusion
CN111626217A
Three-dimensional target detection system and method based on point cloud and image data
CN113111978A
Target detection method and device for automatic driving and computer readable storage medium
CN114187579A
Object detection method and device and electronic equipment
CN114596358A
Attack detection method based on automatic driving, terminal equipment and storage medium
CN115453589A
Cited By
Target detection method and device based on multi-modal fusion, equipment and storage medium
CN120808296A
