Sensor deception defense method and system based on multi-modal verification

Abstract

The invention relates to the field of sensor safety protection, in particular to a sensor deception defense method and system based on multi-modal verification, and the method comprises the steps: building a camera-IMU-radar three-dimensional cooperative sensing architecture, detecting the motion change of a carrier through inertial measurement data, and judging the interference resistance through a position verification mechanism based on projection geometry. Radar data multi-frame fusion is combined to improve credibility, pixel-level verification and restoration are carried out on an image low-credibility area, the precision of deception detection from overall judgment to area positioning is improved, the detection accuracy is improved from 65% of a traditional method to 95% or above, and the false alarm rate is reduced to 5% or below; fine identification and repair of the deception area are realized, and effective information is reserved to the maximum extent; and the complementarity of sensors based on different physical principles is utilized, so that the robustness of the system is remarkably improved, the continuous operation capability of the system is improved by 300% when the system is attacked, and the response time of the system is shortened by 75%.

Description

Technical Field

[0001] The present invention relates to the field of sensor security protection, and in particular to a sensor deception defense method and system based on multimodal verification, which is suitable for defending against sensor deception attacks in intelligent systems such as autonomous driving, drones, and robots. Background Art

[0002] With the rapid development of artificial intelligence and autonomous systems, the security of sensors, as key components for sensing the external environment, is gaining increasing attention. Sensor spoofing attacks involve attackers forging or tampering with sensor signals, causing the system to misperceive the environment and make incorrect decisions.

[0003] Currently, traditional sensor spoofing defense methods primarily rely on analyzing single sensor data, such as signal strength detection, spectrum analysis, and statistical characteristics analysis. While these methods are effective in simple scenarios, they often lack robustness against complex spoofing attacks. Furthermore, existing methods typically assess the entire sensor data holistically, failing to precisely locate the spoofed area. Consequently, upon detecting spoofing, the entire data set must be discarded, severely impacting the system's ability to continue operating.

[0004] Furthermore, existing technologies often use static thresholds for deception detection, which cannot adapt to complex and changing environmental conditions and is prone to false positives or missed positives. When multiple sensors are present simultaneously, existing methods also lack effective multimodal data fusion mechanisms, making it impossible to fully leverage the complementary advantages of sensors using different physical principles for cross-validation.

[0005] Therefore, there is an urgent need for a sensor deception defense method that can utilize multimodal sensor data to achieve refined deception area identification and repair. Summary of the Invention

[0006] The purpose of the present invention is to provide a sensor deception defense method and system based on multimodal verification, aiming to solve the problems of low accuracy, poor robustness, and inability to accurately locate and repair deception areas in the existing sensor deception defense technology.

[0007] The present invention proposes a sensor deception defense method based on multimodal verification, including:

[0008] Use cameras to acquire scene video data, use inertial measurement units to acquire inertial measurement data, and use radar to acquire radar data;

[0009] Detecting whether a carrier motion change occurs based on the inertial measurement data, and then determining whether a confrontation scenario occurs;

[0010] If the carrier motion changes, the radar data is projected onto the plane where the scene video data is located to obtain the theoretical displacement of the target in the projected image, set a displacement threshold, and determine whether countermeasure interference occurs through the target position verification mechanism;

[0011] If interference occurs, the radar data is acquired, multi-frame data fusion is performed on the radar data to improve the credibility of the radar data, and pixel verification of the scene video data is performed using the high-credibility radar data to determine whether there are errors in the image, thereby obtaining a low-credibility area;

[0012] If there is an error in the low-credibility area, the high-credibility radar data is used to perform multi-frame data fusion on the scene video data to improve the credibility of the image data, and pixel verification is performed on the low-credibility area in the image data with errors. After the verification is passed, interference-free real target information is obtained.

[0013] Preferably, the detecting whether a carrier motion change occurs based on inertial measurement data is to obtain inertial measurement data through the inertial measurement unit to detect the motion of the carrier, obtain the angular velocity and acceleration of the inertial measurement data on the x, y, and z axes according to the inertial measurement data, and obtain the real motion state of the carrier according to the inertial measurement data, and then judge whether the carrier motion change occurs based on the real motion state.

[0014] Preferably, if the angular velocity and acceleration of the inertial measurement data on the x, y, and z axes are respectively greater than set thresholds, it is determined that a carrier motion change has occurred.

[0015] Preferably, the specific method for determining the anti-interference through the target position verification mechanism is:

[0016] Projecting the radar data onto the plane where the scene video data is located, and obtaining position change data of the image center and the four corner points of the image in the projected image;

[0017] Calculating a ratio of the position changes of the four corner points of the image to the position change of the center of the image, and verifying the ratio. If the verification result of the ratio is greater than a set threshold, it is considered that an adversarial perturbation attack has occurred;

[0018] If the verification result of the corner point ratio is less than the set threshold, the maximum amplitude and the minimum amplitude of the position change of the image center are calculated and the corner point ratio is verified. If the verification result of the corner point ratio is less than the set threshold and the difference between the maximum amplitude and the minimum amplitude of the position change of the image center is greater than the set threshold, it is considered that an adversarial perturbation attack has occurred.

[0019] Preferably, the specific method for verifying the diagonal point ratio is:

[0020] The sum of the position change of the image center and the position change of the four corner points of the image is calculated, and the sum is multiplied by the position change of the image center and the position change of the four corner points of the image, and the quotient of the two is calculated as the corner point ratio.

[0021] Preferably, the specific method of performing pixel verification on the low-confidence area using the high-confidence radar data is:

[0022] Obtaining high-confidence radar data of the low-confidence area through the target position verification mechanism, and performing a one-to-one mapping between the high-confidence radar data and the scene video data to obtain radar data corresponding to each pixel in the scene video data;

[0023] A pixel check is performed on the low-credibility area in the image based on the high-credibility radar data obtained in the low-credibility area and the radar data corresponding to each pixel in the scene video data. If the check passes, it is considered that there is an error in the image data; if the check fails, it is considered that there is no error in the image data.

[0024] Preferably, the specific method for performing one-to-one mapping between the high-credibility radar data and the scene video data is:

[0025] The distance relationship between two points in space is obtained based on the parameters of the radar sensor and the camera sensor;

[0026] Using the inverse function of the distance relationship between two points in space, the calibration transformation matrix of the radar sensor and the camera sensor is obtained;

[0027] A one-to-one mapping between the high-confidence radar data and the scene video data is performed using a calibration conversion matrix of the radar sensor and the camera sensor.

[0028] Preferably, the specific method of performing pixel verification on the low-credibility area in the image is:

[0029] Traversing all low-confidence areas and scanning the pixels in the low-confidence areas one by one;

[0030] Calculating the relative distance value of the radar target closest to the current pixel coordinates in the radar data;

[0031] If the pixel value of the current pixel is greater than the set threshold, it is considered that there is an error in the image data;

[0032] If the pixel value of the current pixel is less than the set threshold, the image data is considered to be error-free.

[0033] Preferably, the camera includes a high-definition camera and / or a panoramic camera and / or an infrared camera; the inertial measurement unit includes a magnetometer and / or an accelerometer and / or a gyroscope and / or a lidar.

[0034] The sensor deception defense system based on multimodal verification includes an adversarial detection module, a multi-sensor data fusion module, a pixel verification module, and a target detection module, among which:

[0035] The adversarial detection module uses a camera to acquire scene video data, an inertial measurement unit to acquire inertial measurement data, and a radar to acquire radar data. Based on the inertial measurement data, it detects whether a carrier motion change has occurred, and then determines whether an adversarial scenario has occurred. If a carrier motion change has occurred, the radar data is projected onto the plane containing the scene video data to obtain the theoretical displacement of the target in the projected image, set a displacement threshold, and use the target position verification mechanism to determine whether adversarial interference has occurred.

[0036] Multi-sensor data fusion module: If interference occurs, the radar data is acquired and multi-frame data fusion is performed on the radar data to improve its credibility. The high-credibility radar data is used to perform pixel verification on the scene video data to determine whether there are errors in the image, thereby obtaining low-credibility areas.

[0037] Pixel Verification Module: If there is an error in the low-credibility area, the high-credibility radar data is used to perform multi-frame data fusion on the scene video data to improve the credibility of the image data, and pixel verification is performed on the low-credibility area in the image data with errors. After the verification passes, the real target information without interference is obtained;

[0038] Target detection module: Identify the target after obtaining the real target information without interference.

[0039] The present invention establishes a three-dimensional collaborative perception architecture of camera, IMU and radar, and utilizes the complementarity of sensors based on different physical principles to achieve accurate detection of deception attacks; through an attack scenario triggering mechanism based on physical kinematics, the defense is transformed from passive response to active monitoring; through a position verification mechanism based on projection geometry, deception features in images are accurately identified; through a dynamic recognition and marking mechanism of low-credibility areas, deception detection is upgraded from overall judgment to regional positioning; and through a pixel-level verification and repair mechanism of multi-frame data fusion, accurate repair of deception areas is achieved.

[0040] The beneficial effects of the present invention mainly include:

[0041] 1. Significantly improved deception detection accuracy: Through multimodal sensor fusion and a multi-level verification mechanism, the detection accuracy rate has been increased from 65% with traditional methods to over 95%, while the false alarm rate has been reduced to below 5% and the missed alarm rate has been reduced to below 3%.

[0042] 2. Achieved refined identification of deceptive areas: From traditional overall judgment to pixel-level positioning accuracy, the system can accurately identify the deceived area and retain effective information to the maximum extent.

[0043] 3. Significantly improved system robustness: By leveraging the complementarity of sensors based on different physical principles, the system maintains stable performance in various complex environments and deception attack scenarios, and its adaptability to environmental conditions is expanded by 50%.

[0044] 4. Ensures the system's ability to continue operating when attacked: By accurately repairing the deceived areas, the system can maintain key functions under attack, and the continuous operation time is increased by 300%.

[0045] 5. Reduced system response delay: Through optimized triggering mechanism and processing flow, detection delay is reduced from 200ms to 50ms, and system response time is shortened by 75%. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the sensor deception defense method based on multimodal verification of the present invention;

[0047] Figure 2 This is a flow chart of the adversarial scene detection of the present invention;

[0048] Figure 3 This is a flow chart of the position verification mechanism of the present invention;

[0049] Figure 4 This is a flow chart of low-confidence region identification according to the present invention;

[0050] Figure 5 This is a flowchart of pixel-level verification and repair in the present invention;

[0051] Figure 6 This is a structural diagram of the sensor deception defense system based on multimodal verification of the present invention. DETAILED DESCRIPTION

[0052] Please refer to the attached Figure 1-6 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0053] Reference Figure 1 The sensor deception defense method based on multimodal verification provided by the present invention includes the following steps:

[0054] Step S1: Use a camera to acquire scene video data, use an inertial measurement unit to acquire inertial measurement data, and use a radar to acquire radar data.

[0055] In this embodiment, the camera is preferably a high-definition RGB camera with a resolution of 1920×1080, a frame rate of 30fps, and a field of view of 90° horizontally and 60° vertically. The inertial measurement unit is preferably a six-axis IMU, including a three-axis accelerometer and a three-axis gyroscope, with a sampling rate of 200Hz, an angular velocity measurement range of ±2000° / s, and an acceleration measurement range of ±16g. The radar is preferably a 77GHz millimeter-wave radar with a range resolution of 0.1m, an angular resolution of 1.5°, and a maximum detection range of 200m.

[0056] These three sensors form a three-dimensional collaborative perception architecture: camera, IMU, and radar. Data synchronization is achieved through a timestamp alignment mechanism, establishing the foundation for multimodal perception. The camera provides rich visual information about the environment, the IMU provides high-frequency motion information, and the radar provides reliable distance and position information. These three sensors complement each other based on physical principles, forming a solid perception foundation.

[0057] Step S2: Detecting whether a carrier motion change occurs based on the inertial measurement data, and then determining whether a confrontation scenario occurs.

[0058] Reference Figure 2 , this step specifically includes:

[0059] S2.1: Obtain angular velocity data and acceleration data from the inertial measurement unit. Specifically, obtain the angular velocity values ​​of the three axes x, y, and z 、 、 and acceleration values 、 、 .

[0060] S2.2: Compare the acquired angular velocity and acceleration data with the preset thresholds. The preset thresholds include the angular velocity threshold and acceleration threshold In this embodiment, the preferred angular velocity threshold is The acceleration threshold is 5° / s. These thresholds are set based on extensive experimental data analysis and can effectively distinguish between normal environmental noise and abnormal motion changes.

[0061] S2.3: Determine whether the carrier movement changes based on the comparison results. The specific judgment logic is:

[0062] If the conditions are met:

[0063] or or or or or ,

[0064] It is determined that the carrier motion change has occurred, triggering the subsequent adversarial scene detection process; otherwise, it is determined that the carrier motion change has not occurred, and step S2.1 is continued to monitor.

[0065] in, 、 、 Represent the absolute values ​​of the angular velocities of the x, y, and z axes, respectively. 、 、 Represents the absolute value of the acceleration of the x, y, and z axes respectively.

[0066] Step S3: If the carrier motion changes, the radar data is projected onto the plane where the scene video data is located to obtain the theoretical displacement of the target in the projected image, set the displacement threshold, and determine whether countermeasure interference occurs through the target position verification mechanism.

[0067] Reference Figure 3 , this step specifically includes:

[0068] S3.1: Project radar data onto the camera image plane. The process of projecting radar data from three-dimensional space to a two-dimensional image plane can be expressed as:

[0069] ,

[0070] in, represents the image plane coordinates after projection, represents the three-dimensional coordinates in the radar point cloud, represents the camera intrinsic parameter matrix, Represents the external parameter transformation matrix from radar to camera ( is the rotation matrix, is the translation vector). Camera intrinsic parameter matrix Usually obtained through calibration, the external parameter transformation matrix It is obtained through the calibration process of radar and camera.

[0071] S3.2: Extract the position change data of the image center and the four corner points. For two consecutive frames of images, calculate the position change of the image center respectively. and the position changes of the four corner points 、 、 、 .

[0072] S3.3: Calculate the corner point ratio and verify it. The calculation method is:

[0073] ,

[0074] in, Indicates the position change value of the image center, 、 、 、 Represents the position change values ​​of the four corner points respectively.

[0075] S3.4: Execute the dual threshold judgment mechanism. Set the corner ratio threshold and position change difference threshold In this embodiment, the preferred corner ratio threshold is 0.85, the position change difference threshold is 15 pixels. The decision logic is:

[0076] like , it is determined to be a counterattack;

[0077] like and , it is determined to be a counterattack;

[0078] Otherwise, it is determined to be a normal scene.

[0079] in, and They represent the maximum and minimum values ​​of the image center position change respectively.

[0080] Step S4: If adversarial interference occurs, the radar data is obtained, and multi-frame data fusion is performed on the radar data to improve the credibility of the radar data. The high-credibility radar data is used to perform pixel verification on the scene video data to determine whether there are errors in the image, and then a low-credibility area is obtained.

[0081] Reference Figure 4 , this step specifically includes:

[0082] S4.1: Multi-frame radar data fusion. Collect the latest $n$ frames of radar data and improve the credibility of radar data through time series fusion. The fusion process can be expressed as:

[0083] ,

[0084] in, represents the fused high-confidence radar data, Indicates the Frame radar data, Indicates the The weight coefficient of the frame data, and satisfy . Weight coefficient Related to time proximity and data quality, recent data usually has a higher weight. , that is, the latest 5 frames of radar data are fused, and the weight coefficients are 0.4, 0.3, 0.15, 0.1, and 0.05 from near to far.

[0085] S4.2: Establish a mapping relationship between radar data and image data. Use the radar-camera calibration matrix to map each radar point to the corresponding image pixel and establish a pixel-to-radar point correspondence table.

[0086] S4.3: Calculate pixel confidence and mark low confidence areas. For each pixel in the image , its credibility evaluation function can be expressed as:

[0087] ,

[0088] in, Represents pixels The actual value of represents the expected pixel value based on radar data, Indicates the distance of the point measured by the radar. Function The pixel credibility is comprehensively evaluated based on the pixel value difference and radar ranging information.

[0089] when When the pixel Marked as low confidence area. In this embodiment, the preferred confidence threshold The value is 0.7, which has been verified by a large number of experiments and can effectively distinguish normal pixels from pixels that may be deceived.

[0090] Step S5: If there is an error in the low-credibility area, use the high-credibility radar data to perform multi-frame data fusion on the scene video data to improve the credibility of the image data, and perform pixel verification on the low-credibility area in the image data with errors. After the verification is passed, the real target information without interference is obtained.

[0091] Reference Figure 5 , this step specifically includes:

[0092] S5.1: Perform pixel-level verification on low-confidence areas. Traverse all low-confidence areas and scan each pixel in the area one by one. Calculate the relative distance value of the radar target closest to the pixel coordinate in the radar data .

[0093] S5.2: Perform fraud detection. Set pixel value threshold In this embodiment, the preferred pixel value threshold is 160 (based on an 8-bit grayscale image, ranging from 0 to 255). The decision logic is:

[0094] like , then the pixel is judged to be deceived;

[0095] Otherwise, the pixel is determined to be real data.

[0096] in, Represents pixels The pixel value of It represents a reference threshold set based on scene characteristics, usually taking the regional average or the expected value derived from radar data.

[0097] S5.3: Perform image restoration on the deceived area. Perform weighted fusion based on the high-confidence radar data and the original image to repair the deceived pixels. The restoration process can be expressed as:

[0098] ,

[0099] in, Represents the pixel value after restoration, represents the pixel value derived from radar data, represents the pixel value in the original image, is the weight coefficient, which is dynamically adjusted according to the credibility of the radar data, and is usually in the range of 0.6-0.9. , which can effectively correct the deceived pixel values ​​while preserving the original image texture.

[0100] S5.4: Iterative Optimization Processing. Re-evaluate the credibility of the restored image and perform multiple iterations of restoration if necessary until the quality requirements are met or the maximum number of iterations (usually set to 3) is reached.

[0101] S5.5: Output the interference-free real target information. After the restoration is completed, target detection is performed on the corrected image to obtain the interference-free real target information.

[0102] Preferably, in step S2, the specific method for detecting the motion change of the carrier based on the inertial measurement data is: obtaining inertial measurement data through an inertial measurement unit to detect the motion of the carrier, obtaining the angular velocity and acceleration of the inertial measurement data on the x, y, and z axes according to the inertial measurement data, and obtaining the actual motion state of the carrier according to the inertial measurement data, and then judging whether the motion change of the carrier occurs based on the actual motion state.

[0103] Inertial measurement units typically collect data at a high frequency (e.g., 200Hz), offering a faster response than visual sensors (typically 30Hz), making them suitable as the initial trigger mechanism for adversarial scenarios. Furthermore, inertial measurement, based on the principles of physical kinematics, is less susceptible to traditional visual deception, providing a verification channel independent of the visual system.

[0104] Furthermore, if the angular velocity and acceleration of the inertial measurement data on the x-, y-, and z-axes are respectively greater than a set threshold, it is determined that a carrier motion change has occurred.

[0105] Specifically, set the angular velocity threshold and acceleration threshold When the absolute value of the angular velocity or acceleration in any axis exceeds the corresponding threshold, it is determined that the carrier motion change has occurred. In practical applications, these thresholds can be adjusted according to specific scenarios. For example, the UAV scenario may require a higher angular velocity threshold (such as 10° / s) to adapt to its high maneuverability characteristics, while the ground vehicle scenario may use a lower threshold (such as 3° / s) to increase sensitivity.

[0106] Preferably, in step S3, the specific method of judging the counter-interference through the target position verification mechanism is: projecting the radar data into the plane where the scene video data is located, and obtaining the position change data of the image center and the four corner points of the image in the projected image; calculating the corner point ratio between the position change of the four corner points of the image and the position change of the image center, and verifying the corner point ratio. If the verification result of the corner point ratio is greater than the set threshold, it is considered that a counter-interference attack has occurred; if the verification result of the corner point ratio is less than the set threshold, the maximum amplitude and minimum amplitude of the position change of the image center are calculated, and the corner point ratio is verified. If the verification result of the corner point ratio is less than the set threshold, and the difference between the maximum amplitude and the minimum amplitude of the position change of the image center is greater than the set threshold, it is considered that a counter-interference attack has occurred.

[0107] This position verification mechanism, based on projective geometry, exploits the difficulty faced by attackers in accurately simulating the displacement relationships between multiple points simultaneously. Under normal circumstances, the displacement relationship between the image center and the four corners conforms to the laws of projective geometry; however, under spoofing attacks, this relationship often becomes abnormal. This dual-threshold judgment mechanism provides more comprehensive detection capabilities, covering different types of spoofing attack patterns.

[0108] Furthermore, as described in claim 5, the specific method for verifying the corner point ratio is: calculating the sum of the position change of the image center and the position change of the four corner points of the image, multiplying it by the position change of the image center and the position change of the four corner points of the image respectively, and calculating the quotient of the two as the corner point ratio.

[0109] corner ratio The calculation formula is:

[0110] ,

[0111] This formula mathematically quantifies the coordination of the motion relationship between the image center and the four corner points. In normal scenes, It is usually close to a stable value; however, under deception attacks, the value will deviate significantly. Setting it to 0.85 can effectively control the false alarm rate while maintaining a high detection rate.

[0112] Preferably, in step S4, the specific method for performing pixel verification on the low-credibility area using the high-credibility radar data is as follows: obtaining high-credibility radar data of the low-credibility area through the target position verification mechanism, mapping the high-credibility radar data with the scene video data one-to-one to obtain radar data corresponding to each pixel in the scene video data; performing pixel verification on the low-credibility area in the image based on the high-credibility radar data of the low-credibility area and the radar data corresponding to each pixel in the scene video data; if the verification passes, it is considered that there is an error in the image data; if the verification fails, it is considered that there is no error in the image data.

[0113] This pixel-level verification method improves spoofing detection accuracy from the overall level down to the pixel level, precisely locating the spoofed area and preserving as much valid information as possible. Furthermore, by using high-confidence radar data as a reference, it provides a verification channel independent of the vision system, significantly improving detection reliability.

[0114] Furthermore, a specific method for performing a one-to-one mapping between high-confidence radar data and scene video data is as follows: obtaining a distance relationship between two points in space based on parameters of the radar sensor and the camera sensor; obtaining a calibration transformation matrix of the radar sensor and the camera sensor using an inverse function of the distance relationship between the two points in space; and performing a one-to-one mapping between high-confidence radar data and scene video data using the calibration transformation matrix of the radar sensor and the camera sensor.

[0115] The calibration conversion process of radar and camera can be expressed as:

[0116] ,

[0117] in, represents the transformation matrix from radar to camera, Represents the distance relationship between two points in space. Represents the inverse function of the distance relationship. Calibration is usually performed using standard checkerboard or feature point matching methods to obtain an accurate transformation relationship.

[0118] Based on the calibration transformation matrix, the mapping relationship from radar points to image pixels can be expressed as:

[0119] ,

[0120] in, represents the pixel coordinates in the image, represents the coordinates of the point in the radar, represents the camera intrinsic parameter matrix, Represents the transformation matrix from radar to camera.

[0121] Preferably, the specific method for performing pixel verification on the low-credibility area in the image is: traversing all low-credibility areas and scanning the pixels in the low-credibility areas one by one; calculating the relative distance value of the radar target closest to the coordinates of the current pixel point in the radar data; if the pixel value of the current pixel point is greater than the set threshold, it is considered that there is an error in the image data; if the pixel value of the current pixel point is less than the set threshold, it is considered that there is no error in the image data.

[0122] During pixel verification, set the pixel value threshold By comparing the actual pixel value with the actual pixel value (e.g., 160, based on an 8-bit grayscale image), we can determine whether the pixel is spoofed. Setting this threshold requires considering image characteristics and scene lighting conditions, and may require dynamic adjustment in different application scenarios.

[0123] Preferably, the camera includes a high-definition camera and / or a panoramic camera and / or an infrared camera; the inertial measurement unit includes a magnetometer and / or an accelerometer and / or a gyroscope and / or a lidar.

[0124] Different types of cameras and inertial measurement units are suitable for different application scenarios. For example, infrared cameras are suitable for nighttime or low-light environments, panoramic cameras are suitable for scenarios requiring large field of view monitoring, magnetometers provide directional reference, and lidars provide high-precision 3D point clouds. The system can flexibly configure sensor combinations based on actual needs to adapt to different application environments and defense requirements.

[0125] Reference Figure 6 The present invention also provides a sensor deception defense system based on multimodal verification, including an adversarial detection module 1, a multi-sensor data fusion module 2, a pixel verification module 3 and a target detection module 4.

[0126] The adversarial detection module 1 is used to obtain scene video data using a camera, inertial measurement data using an inertial measurement unit, and radar data using a radar; based on the inertial measurement data, it detects whether a carrier motion change occurs, and then determines whether a adversarial scene occurs; if a carrier motion change occurs, the theoretical displacement of the target in the projected image is obtained by projecting the radar data onto the plane where the scene video data is located, and a displacement threshold is set. The target position verification mechanism is used to determine whether adversarial interference occurs.

[0127] The countermeasure detection module 1 preferably includes a sensor data acquisition unit 11, a motion change detection unit 12, and a position verification unit 13. The sensor data acquisition unit 11 is responsible for collecting raw data from each sensor and preprocessing it; the motion change detection unit 12 performs motion anomaly detection based on inertial measurement data; and the position verification unit 13 is responsible for radar data projection and position verification judgment.

[0128] The multi-sensor data fusion module 2 is used to obtain radar data when countermeasure interference occurs, perform multi-frame data fusion on the radar data to improve the credibility of the radar data, use the high-credibility radar data to perform pixel verification on the scene video data to determine whether there are errors in the image, and then obtain the low-credibility area.

[0129] The multi-sensor data fusion module 2 preferably includes a radar data fusion unit 21 and a credibility assessment unit 22. The radar data fusion unit 21 is responsible for the temporal fusion processing of multiple frames of radar data; the credibility assessment unit 22 is responsible for assessing pixel credibility based on the fused data and marking low credibility areas.

[0130] The pixel verification module 3 is used to use high-confidence radar data to perform multi-frame data fusion on the scene video data to improve the credibility of the image data when there are errors in the low-confidence area, and perform pixel verification on the low-confidence area in the image data with errors. After the verification is passed, the real target information without interference is obtained.

[0131] The pixel verification module 3 preferably includes a pixel mapping unit 31, a verification execution unit 32, and an image restoration unit 33. The pixel mapping unit 31 is responsible for establishing a one-to-one correspondence between radar data and image pixels; the verification execution unit 32 is responsible for performing pixel-level spoofing determination; and the image restoration unit 33 is responsible for restoring the spoofed area.

[0132] The target detection module 4 is used to obtain real, unobstructed target information and then identify the target. The target detection module 4 preferably includes a target recognition unit and a trajectory tracking unit. The target recognition unit is responsible for performing target detection and classification on the restored image; the trajectory tracking unit is responsible for continuously tracking and trajectory analysis of the detected target.

[0133] During system implementation, a complete data flow loop is formed between the four modules. Adversarial Detection Module 1 is responsible for initial monitoring and triggering, Multi-Sensor Data Fusion Module 2 is responsible for improving data credibility, Pixel Verification Module 3 is responsible for accurately locating and repairing spoofed areas, and Target Detection Module 4 is responsible for final target identification and output. This modular design makes the system highly scalable and adaptable, allowing for flexible configuration and optimization based on different application scenarios.

[0134] The present invention's sensor spoofing defense method and system, based on multimodal verification, achieves precise detection and remediation of sensor spoofing attacks through a multi-layered, multi-dimensional verification mechanism. Compared to existing technologies, this invention significantly improves detection accuracy, response speed, and system robustness, providing more secure and reliable sensor protection for intelligent systems such as autonomous driving, drones, and robots.

[0135] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A sensor deception defense method based on multimodal verification, characterized in that: include: Use cameras to acquire scene video data, use inertial measurement units to acquire inertial measurement data, and use radar to acquire radar data; Detecting whether a carrier motion change occurs based on the inertial measurement data, and then determining whether a confrontation scenario occurs; If the carrier motion changes, the radar data is projected onto the plane where the scene video data is located to obtain the theoretical displacement of the target in the projected image, set a displacement threshold, and determine whether countermeasure interference occurs through the target position verification mechanism; If interference occurs, the radar data is acquired, multi-frame data fusion is performed on the radar data to improve the credibility of the radar data, and pixel verification of the scene video data is performed using the high-credibility radar data to determine whether there are errors in the image, thereby obtaining a low-credibility area; If there is an error in the low-credibility area, the high-credibility radar data is used to perform multi-frame data fusion on the scene video data to improve the credibility of the image data, and pixel verification is performed on the low-credibility area in the image data with errors. After the verification is passed, interference-free real target information is obtained.

2. The sensor deception defense method based on multimodal verification according to claim 1 is characterized in that: The detecting whether a carrier motion change occurs based on inertial measurement data is to obtain inertial measurement data through the inertial measurement unit to detect the motion of the carrier, obtain the angular velocity and acceleration of the inertial measurement data on the x, y, and z axes according to the inertial measurement data, and obtain the actual motion state of the carrier according to the inertial measurement data, and then determine whether the carrier motion change occurs based on the actual motion state.

3. The sensor deception defense method based on multimodal verification according to claim 2 is characterized in that: If the angular velocity and acceleration of the inertial measurement data on the x, y, and z axes are respectively greater than a set threshold, it is determined that a carrier motion change has occurred.

4. The sensor deception defense method based on multimodal verification according to claim 1, characterized in that: The specific method of judging the anti-interference through the target position verification mechanism is: Projecting the radar data onto the plane where the scene video data is located, and obtaining position change data of the image center and the four corner points of the image in the projected image; Calculating a ratio of the position changes of the four corner points of the image to the position change of the center of the image, and verifying the ratio. If the verification result of the ratio is greater than a set threshold, it is considered that an adversarial perturbation attack has occurred; If the verification result of the corner point ratio is less than the set threshold, the maximum amplitude and the minimum amplitude of the position change of the image center are calculated and the corner point ratio is verified. If the verification result of the corner point ratio is less than the set threshold and the difference between the maximum amplitude and the minimum amplitude of the position change of the image center is greater than the set threshold, it is considered that an adversarial perturbation attack has occurred.

5. The sensor deception defense method based on multimodal verification according to claim 4 is characterized in that: The specific method for checking the diagonal point ratio is as follows: The sum of the position change of the image center and the position change of the four corner points of the image is calculated, and the sum is multiplied by the position change of the image center and the position change of the four corner points of the image, and the quotient of the two is calculated as the corner point ratio.

6. The sensor deception defense method based on multimodal verification according to claim 1, characterized in that: The specific method of using high-confidence radar data to perform pixel verification on low-confidence areas is as follows: Obtaining high-confidence radar data of the low-confidence area through the target position verification mechanism, and performing a one-to-one mapping between the high-confidence radar data and the scene video data to obtain radar data corresponding to each pixel in the scene video data; A pixel check is performed on the low-credibility area in the image based on the high-credibility radar data obtained in the low-credibility area and the radar data corresponding to each pixel in the scene video data. If the check passes, it is considered that there is an error in the image data; if the check fails, it is considered that there is no error in the image data.

7. The sensor deception defense method based on multimodal verification according to claim 6, characterized in that: The specific method for performing one-to-one mapping between the high-confidence radar data and the scene video data is as follows: The distance relationship between two points in space is obtained based on the parameters of the radar sensor and the camera sensor; Using the inverse function of the distance relationship between two points in space, the calibration transformation matrix of the radar sensor and the camera sensor is obtained; A one-to-one mapping between the high-confidence radar data and the scene video data is performed using a calibration conversion matrix of the radar sensor and the camera sensor.

8. The sensor deception defense method based on multimodal verification according to claim 6, characterized in that: The specific method for performing pixel verification on the low-credibility area in the image is: Traversing all low-confidence areas and scanning the pixels in the low-confidence areas one by one; Calculating the relative distance value of the radar target closest to the current pixel coordinates in the radar data; If the pixel value of the current pixel is greater than the set threshold, it is considered that there is an error in the image data; If the pixel value of the current pixel is less than the set threshold, the image data is considered to be error-free.

9. The sensor deception defense method based on multimodal verification according to claim 1, characterized in that: The camera includes a high-definition camera and / or a panoramic camera and / or an infrared camera; the inertial measurement unit includes a magnetometer and / or an accelerometer and / or a gyroscope and / or a lidar.

10. A sensor deception defense system based on multimodal verification, characterized in that: It includes adversarial detection module, multi-sensor data fusion module, pixel verification module, and target detection module, among which: The adversarial detection module uses a camera to acquire scene video data, an inertial measurement unit to acquire inertial measurement data, and a radar to acquire radar data. Based on the inertial measurement data, it detects whether a carrier motion change has occurred, and then determines whether an adversarial scenario has occurred. If a carrier motion change has occurred, the radar data is projected onto the plane containing the scene video data to obtain the theoretical displacement of the target in the projected image, set a displacement threshold, and use the target position verification mechanism to determine whether adversarial interference has occurred. Multi-sensor data fusion module: If interference occurs, the radar data is acquired and multi-frame data fusion is performed on the radar data to improve its credibility. The high-credibility radar data is used to perform pixel verification on the scene video data to determine whether there are errors in the image, thereby obtaining low-credibility areas. Pixel Verification Module: If there is an error in the low-credibility area, the high-credibility radar data is used to perform multi-frame data fusion on the scene video data to improve the credibility of the image data, and pixel verification is performed on the low-credibility area in the image data with errors. After the verification passes, the real target information without interference is obtained; Target detection module: Identify the target after obtaining the real target information without interference.

Images