Anti-attack detection method and system based on cross-modal attention mechanism

Abstract

The invention relates to the field of artificial intelligence, in particular to an anti-attack detection method and system based on a cross-modal attention mechanism. The method comprises the steps of collecting multi-modal data, performing sparse feature extraction and reconstruction on the multi-modal data to obtain radar sparse features and visual sparse features, and fusing the radar sparse features and the visual sparse features based on a cross-modal attention mechanism to obtain cross-modal fusion features. The cross-modal fusion features are input into an auto-encoder SLAE sharing a potential space for consistency learning, radar reconstruction features and visual reconstruction features are obtained, and reconstruction errors are calculated to realize detection and identification of countermeasure attacks; and the accuracy of anti-attack detection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to adversarial attack detection methods and systems based on cross-modal attention mechanisms. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous intelligent drones, with their powerful capabilities in target recognition and dynamic tracking, have been widely applied in key areas such as unmanned inspection and emergency rescue, becoming a core carrier of the low-altitude economy. The reliable operation of these intelligent systems highly depends on the accurate perception and analysis of multimodal sensor data, such as visual images and radar point clouds. Deep learning algorithms, in particular, have shown significant advantages in processing single-modal data (such as target classification in visual images and obstacle detection in radar point clouds), driving the improvement of drones' autonomous decision-making capabilities.

[0003] However, physical adversarial attacks on artificial intelligence models have become a serious challenge threatening drone security. These attacks, by implanting adversarial perturbations (such as special patterns or artificial light sources) into target objects or the environment, can directly deceive single-modal perception models such as vision or radar, causing deviations in target identification and localization, and ultimately leading to risks such as drone mission failure and loss of control. For example, attackers can interfere with visual recognition by covering ground targets with specific textures, or use electromagnetic signals to interfere with the spatial positioning accuracy of radar point clouds, causing drones to misjudge environmental information.

[0004] Existing defenses against physical adversarial attacks are still limited to optimization at the single-modal level. For example, adversarial training methods for the visual modality improve model robustness by injecting attack samples, while filtering algorithms for the radar modality enhance data stability through noise reduction. These single-modal detection algorithms can only identify anomalies in data from a single type of sensor, and their defense logic relies on learning the distribution patterns of single-modal data.

[0005] However, in real-world scenarios, attackers have begun employing bimodal cooperative attack strategies: by simultaneously interfering with visual images and radar point clouds (e.g., simultaneously covering the target with visual disturbances and releasing radar wave interference signals), they cause the perception data of both modalities to exhibit false characteristics and form a false consistency with each other. In this situation, single-modal detection algorithms, unable to verify the authenticity of data across modalities, struggle to identify such cooperative attacks. For example, a visual modality detection algorithm might misjudge its own perception as normal due to the false normality of radar data, and vice versa. Furthermore, single-modal detection algorithms typically require extensive data updates and training for a specific modality, resulting in significant computational overhead. This makes them unsuitable for the real-time and lightweight requirements of resource-constrained platforms such as drones, further limiting their defensive effectiveness in complex bimodal attack scenarios.

[0006] Therefore, given the technical bottleneck that single-modal detection algorithms cannot cope with bimodal adversarial attacks, it is urgent to explore a detection method that can integrate multimodal data and verify consistency across modalities in order to improve the security protection capabilities of intelligent drones in complex attack scenarios. Summary of the Invention

[0007] To address the technical bottleneck that single-modal detection algorithms cannot handle bimodal adversarial attacks, this application provides an adversarial attack detection method and system based on a cross-modal attention mechanism.

[0008] Firstly, this application provides an adversarial attack detection method based on a cross-modal attention mechanism:

[0009] S1: Collect multimodal data, including radar point cloud data and depth vision data;

[0010] S2: Perform sparse feature extraction and reconstruction on the radar point cloud data and depth visual data respectively to obtain radar sparse features and visual sparse features.

[0011] S3: The radar sparse features and visual sparse features are fused based on a cross-modal attention mechanism to obtain cross-modal fused features; the cross-modal attention mechanism includes quality-aware gating and cross-attention mechanism, the quality-aware gating is used to dynamically allocate initial weights based on data quality, and the cross-attention mechanism is used to fine-tune the fusion weights based on the semantic correlation between modalities;

[0012] S4: Input the cross-modal fusion features into a shared latent space autoencoder (SLAE) for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space, and the decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features.

[0013] S5: Calculate the first Euclidean distance between the radar sparse feature and the radar reconstructed feature, and the second Euclidean distance between the visual sparse feature and the visual reconstructed feature, and use the sum of the first Euclidean distance and the second Euclidean distance as the reconstruction error;

[0014] S6: If the reconstruction error is greater than a preset threshold, it is determined that there is an adversarial attack; otherwise, it is determined that there is no adversarial attack.

[0015] The above technical solution acquires complementary information through multimodal data acquisition (radar point cloud + depth vision), filters noise through sparse feature extraction, dynamically fuses features using a cross-modal attention mechanism (quality-aware gating + cross-attention), learns modal consistency through a shared latent space autoencoder SLAE, and finally judges adversarial attacks through reconstruction errors. The complementarity of multimodal data addresses the vulnerability of single modalities to attacks, with the 3D spatial information of radar and the texture information of vision forming cross-validation; the cross-modal attention mechanism enables dynamic weight adjustment, filtering low-quality data (quality-aware gating) and strengthening semantic associations (cross-attention), ensuring the reliability of fused features; the shared latent space autoencoder learns cross-modal consistency in normal modes, making modal conflicts caused by adversarial attacks significantly reflected through reconstruction errors, achieving effective detection of physical adversarial attacks, and exhibiting strong generalization ability without relying on specific attack patterns.

[0016] Optionally, in step S2, the sparse feature extraction and reconstruction includes:

[0017] A complete dictionary was constructed, and the sparse coefficients were obtained by solving an optimization problem, namely:

[0018]

[0019] Where x is the input feature, D is the overcomplete dictionary, α is the sparsity coefficient, and λ is the sparsity control parameter;

[0020] The input features are reconstructed based on the sparse coefficients to obtain the radar sparse features or visual sparse features.

[0021] By employing the above technical solution, sparse coefficients are extracted through an overcomplete dictionary and an optimization problem. Features are then reconstructed based on these sparse coefficients, preserving key information while suppressing noise. Sparse feature extraction achieves feature sparsity by constraining λ||α||1 using L1 regularization, retaining only the most crucial information for the task (such as the spatial contour of radar and the target edge in vision), while filtering out random noise and redundant information. The reconstructed features exhibit higher stability, providing clean input for subsequent cross-modal fusion, avoiding fusion bias caused by noise interference, and indirectly improving the accuracy of adversarial attack detection.

[0022] Optionally, the quality-aware gating dynamic allocation of initial weights includes:

[0023] The quality indicators of the radar sparse features are evaluated, and the quality indicators include at least one of point density, maximum detection range, and field of view.

[0024] The quality metrics of the visual sparse features are evaluated, including at least one of contrast, sharpness, and dynamic range;

[0025] Initial weights are assigned to the radar sparse features and visual sparse features based on the evaluation results, with higher quality features corresponding to larger initial weights.

[0026] By adopting the above technical solutions, quality perception gating evaluates the quality indicators of radar point clouds and depth vision, assigns higher initial weights to high-quality features, and dynamically adapts to environmental changes. For example, when the quality of visual images deteriorates at night, it automatically reduces the visual weights and increases the radar weights (radar is not affected by lighting), avoiding low-quality data from dominating the fusion results. It also ensures that the modal data participating in the fusion have basic reliability from a physical perspective, reduces false detections caused by temporary sensor failures or environmental interference (such as heavy fog or backlight), and improves the environmental adaptability of the method.

[0027] Optionally, the cross-attention mechanism fine-tunes the fusion weights, including:

[0028] Construct query Q, key K, and value V for the radar sparse features and visual sparse features respectively;

[0029] The semantic correlation between the radar sparse features and the visual sparse features is calculated to obtain the cross-modal attention weights;

[0030] Based on the cross-modal attention weights, the radar sparse features and visual sparse features are weighted and summed to obtain the cross-modal fusion features. The calculation formula is as follows:

[0031]

[0032] Where F fused For cross-modal fusion features, A cross For cross-modal attention weights, V vision V represents the value of the visual sparse feature. radar This represents the value of the radar sparse feature.

[0033] By adopting the above technical solution, the cross-attention mechanism calculates the semantic correlation between modalities by constructing Q / K / V representations, fine-tunes the weights based on the correlation, and finally outputs fused features. It strengthens modal consistency at the semantic level. For example, if the 3D contour detected by radar matches the target category recognized by vision, the weight is automatically increased, and vice versa, making the fused features closer to the semantics of the real scene. It solves the problem that traditional fixed-weight fusion cannot cope with modal semantic conflicts, is sensitive to adversarial attack scenarios such as vision being deceived but radar functioning normally, and improves the accuracy of attack detection.

[0034] Optionally, the shared encoder is a three-layer fully connected network, wherein the weights of the first two layers are shared in the radar feature processing branch and the visual feature processing branch, which are used to converge the radar sparse features and the visual sparse features to the same distribution pattern; the third layer is a dimension compression layer, which is used to compress high-dimensional features into the same latent space.

[0035] By adopting the above technical solution, the first two layers of the shared encoder share weights, forcing radar and visual features to converge to the same distribution; the third layer is compressed into a shared latent space to filter modality-specific noise; weight sharing forces the two heterogeneous features (radar point cloud is three-dimensional coordinates, and vision is two-dimensional pixels) to align core features (such as the shape and position of the target) during the transformation process, solving the semantic gap caused by the difference in cross-modal data structure; the shared latent space ensures that the feature representations of the two modalities are highly similar in normal scenarios, making it easier to capture modal inconsistencies caused by adversarial attacks (such as latent representation shifts), and improving the autoencoder's sensitivity to anomalies.

[0036] Optionally, the decoder includes a radar decoder and a visual decoder, with the radar decoder and visual decoder using independent parameters; the radar decoder is used to reconstruct the spatial structure features of the radar point cloud, and the visual decoder is used to reconstruct the texture detail features of the depth vision.

[0037] By adopting the above technical solution, the decoder includes a radar decoder (focusing on spatial structure reconstruction) and a visual decoder (focusing on texture detail reconstruction) with independent parameters, which are adapted to modal characteristics respectively; the reconstruction effect is optimized for modal differences: the radar decoder focuses on restoring the spatial distribution of point clouds (such as the distance and outline of the target), and the visual decoder focuses on restoring the texture and color of the image (such as the surface features of the target), making the reconstructed features closer to the original input; the improved reconstruction accuracy can reduce the misjudgment error in normal scenes, while making the abnormal error caused by adversarial attacks more significant, and improving the signal-to-noise ratio of reconstruction error judgment.

[0038] Optionally, the preset threshold is determined by the following method: in a normal scenario without adversarial attacks, multiple sets of the multimodal data are collected, the reconstruction error corresponding to each set of data is calculated, and the maximum value of all reconstruction errors is taken as the preset threshold.

[0039] By adopting the above technical solution, the preset threshold is determined based on the maximum value of multiple reconstruction errors under normal scenarios, ensuring that the error of normal data is lower than the threshold.

[0040] Optionally, it also includes a lightweight optimization step, which includes:

[0041] Sparse pruning is performed on the shared encoder and decoder, removing the 30% of neurons with the lowest weight magnitudes;

[0042] The model weights are quantized to 8-bit, converting the 32-bit floating-point weights into 8-bit integer weights.

[0043] The cross-attention mechanism is optimized using low-rank decomposition and window attention to reduce computational complexity.

[0044] By adopting the above technical solutions, on the one hand, sparse pruning technology is used to scientifically and rationally remove 30% of redundant neurons, reducing unnecessary computational resource consumption; on the other hand, 8-bit quantization technology is used to successfully compress the weight volume to 1 / 4 of its original size. Furthermore, low-rank decomposition and window attention are combined to optimize the computational structure, comprehensively improving computational efficiency. This better meets the real-time detection needs of UAVs (such as dynamic environmental response during low-altitude flight), broadening the practical application scenarios of the method.

[0045] Secondly, this application provides an adversarial attack detection system based on a cross-modal attention mechanism; the system includes modules for executing the adversarial attack detection method in the first aspect or any possible implementation of the first aspect:

[0046] The system includes:

[0047] The acquisition module is used to acquire multimodal data, including radar point cloud data and depth vision data;

[0048] The feature extraction module is used to extract and reconstruct sparse features from the radar point cloud data and the depth visual data respectively, to obtain radar sparse features and visual sparse features.

[0049] The feature fusion module is used to fuse the radar sparse features and visual sparse features based on a cross-modal attention mechanism to obtain cross-modal fused features. The cross-modal attention mechanism includes a quality-aware gating mechanism and a cross-attention mechanism. The quality-aware gating mechanism is used to dynamically allocate initial weights based on data quality, and the cross-attention mechanism is used to fine-tune the fusion weights based on the semantic relevance between modalities.

[0050] The feature reconstruction module is used to input the cross-modal fusion features into an autoencoder SLAE with a shared latent space for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space, and the decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features.

[0051] The calculation module is used to calculate the first Euclidean distance between the radar sparse features and the radar reconstructed features, and the second Euclidean distance between the visual sparse features and the visual reconstructed features, and to use the sum of the first Euclidean distance and the second Euclidean distance as the reconstruction error.

[0052] The judgment module is used to determine that there is an adversarial attack if the reconstruction error is greater than a preset threshold; otherwise, it determines that there is no adversarial attack.

[0053] Thirdly, this application provides a computer device including a processor, a memory, and a communication bus, wherein the communication bus is used to realize a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the anti-attack detection method as described in any of the preceding claims.

[0054] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the anti-attack detection method described above.

[0055] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the anti-attack detection method described above. Attached Figure Description

[0056] Figure 1 A schematic diagram of a cross-modal attention mechanism-based adversarial attack detection method provided in this application embodiment;

[0057] Figure 2 A schematic diagram of an adversarial attack detection framework based on a cross-modal attention mechanism is provided for an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of a cross-attention mechanism architecture provided in an embodiment of this application;

[0059] Figure 4 A schematic diagram of an adversarial attack detection system based on a cross-modal attention mechanism is provided for an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0063] refer to Figure 1 An adversarial attack detection method based on cross-modal attention mechanism mainly includes the following steps:

[0064] S1: Collect multimodal data, including radar point cloud data and depth vision data;

[0065] It should be understood that radar point cloud data is a dataset composed of a large number of discrete points generated after a target or environment is scanned in three dimensions by radar sensors (such as millimeter-wave radar and lidar). Each point contains three-dimensional spatial coordinates (x, y, z), and some radars also record the reflection intensity of the point (related to the target material and surface roughness), which can reflect the target's spatial position, outline, distance, and other geometric information. For example, when a UAV scans ground buildings using lidar, the generated point cloud can accurately represent the height, shape, and relative distance of the building to the UAV.

[0066] Specifically, by equipping a drone with a lidar sensor, the drone can emit a laser beam and receive the reflected signal to calculate the laser propagation time to obtain the target distance. Combined with the drone's own positioning information (such as GPS positioning), three-dimensional point coordinates can be generated and point cloud data streams can be output in real time.

[0067] Similarly, depth vision data refers to fused data that includes two-dimensional visual information (such as texture, color, and target outline) and depth information (distance from a pixel to the camera). It retains the semantic features of traditional images (such as target category and texture details) while supplementing spatial distance information through the depth channel, enabling collaborative perception of planar vision and three-dimensional ranging. For example, in pedestrian images captured by a drone camera, each pixel not only contains RGB color values ​​but also indicates the actual distance between the pedestrian and the drone. The specific methods for acquiring radar point cloud data and depth vision data can be any existing method, and this embodiment is not limited thereto.

[0068] S2: Perform sparse feature extraction and reconstruction on radar point cloud data and depth visual data respectively to obtain radar sparse features and visual sparse features.

[0069] In an optional embodiment of this application, an overcomplete dictionary D can be constructed, and sparse coefficients can be obtained by solving an optimization problem, wherein the optimization problem is:

[0070]

[0071] Where x is the input feature (i.e., radar point cloud data, depth vision data), D is the overcomplete dictionary, α is the sparsity coefficient, and λ is the sparsity control parameter (the larger λ is, the sparser α is).

[0072] The input features are reconstructed based on sparse coefficients to obtain radar sparse features or visual sparse features.

[0073] It should be understood that any existing method can be used to solve the optimization problem, and this embodiment does not impose any restrictions on this, as long as the sparse coefficients α of the pronunciation can be obtained. For example, the Basis Pursuit Denoising (BPDN) algorithm can be used, which transforms the L1 regularization term into a constraint condition and solves the constrained least squares problem through an iterative algorithm.

[0074] In an optional embodiment of this application, after the sparse coefficients are obtained by the above algorithm, the sparse features can be achieved by a linear combination of the dictionary D and the sparse coefficients:

[0075]

[0076] As a sparse feature, α * The optimal sparsity coefficients are obtained to optimize the solution of the problem.

[0077] In an optional embodiment of this application, the K-SVD algorithm can be used for offline training to learn a complete dictionary D from a large number of samples, enabling it to sparsely represent target features (such as the spatial structure of radar point clouds and the texture edges of visual images). Due to the differences in features across modalities, such as radar point cloud data emphasizing three-dimensional geometric features and depth vision data emphasizing texture primitives, independently trained dictionaries can be used for radar point clouds and depth vision respectively.

[0078] The sparsity coefficient α is obtained by solving the optimization problem. * Essentially, it involves the activation weights of dictionary atoms. For example, in radar point clouds, an atom corresponds to a vertical plane feature; if the coefficient of that atom is non-zero, it indicates the presence of a vertical plane structure in the input. Similarly, in visual images, an atom corresponds to an edge texture, and its coefficient reflects the intensity of that texture. By summing the activated dictionary atoms according to their coefficients, sparse features are obtained. Due to α * It is sparse, with only a few atoms participating in the reconstruction, thus preserving key information and suppressing noise. That is, through sparse representation, random noise (such as radar measurement errors and visual image noise) is naturally filtered out during the reconstruction process because it cannot be effectively represented by dictionary atoms. The reconstructed sparse features retain only key information matching dictionary atoms (such as target contours and texture primitives), weakening irrelevant details, thus improving the stability of subsequent cross-modal fusion. Furthermore, the dimensionality of the sparse coefficients is much lower than that of the original features, reducing the computational complexity of subsequent processing, thus better adapting to the resource-constrained environment of UAVs.

[0079] S3: Based on the cross-modal attention mechanism, radar sparse features and visual sparse features are fused to obtain cross-modal fused features; the cross-modal attention mechanism includes quality-aware gating and cross-attention mechanism. Quality-aware gating is used to dynamically allocate initial weights based on data quality, and cross-attention mechanism is used to fine-tune the fusion weights based on the semantic correlation between modalities.

[0080] See also Figure 2 This embodiment can employ a cross-modal attention mechanism during the fusion process to dynamically adjust the modal contribution weights based on both data quality and semantic relevance. This can better adapt to environmental changes and improve detection accuracy.

[0081] In an optional embodiment of this application, the quality-aware gating can evaluate the quality indicators of radar sparse features and visual sparse features separately, and assign initial weights to the radar sparse features and visual sparse features based on the evaluation results, with higher-quality features corresponding to larger initial weights. Specifically, the evaluation process of data quality is described in detail below using the evaluation of the quality indicators of radar sparse features as an example:

[0082] First, calculate the score for each quality indicator:

[0083] For point density assessment, the point density score can be calculated using the following formula:

[0084]

[0085] Different regions are assigned different weights, such as high weights for central regions and low weights for peripheral regions; point density scores are calculated by counting the number of valid points within a region.

[0086] For the assessment of maximum detection range, the corresponding score can be calculated using the following formula:

[0087]

[0088] In the formula, d max Indicates the radar's nominal maximum detection range; d 实际 This represents the actual maximum detection distance in the current frame, for example, the maximum distance among 95% of valid points; k represents the attenuation coefficient, which controls the rate at which the score decreases with distance.

[0089] For field-of-view evaluation, coverage is calculated by marking whether each grid contains valid points. If the coverage is below a threshold (e.g., 30%), it is considered poor quality. The corresponding score can be calculated using the following formula:

[0090]

[0091] Based on the scores of each quality indicator, the comprehensive score of radar sparse features is calculated using the following formula:

[0092] q radar = ω1 × point density score + ω2 × detection range score + ω3 × field of view score

[0093] The weights of each quality indicator can be flexibly set. For example, for static scenarios (such as building monitoring), the density weight can be emphasized and the field of view weight can be weakened. For example, it can be set as: w1=0.6, w2=0.3, w3=0.1; for dynamic scenarios (such as drone obstacle avoidance), the detection distance can be emphasized. For example, it can be set as: w1=0.4, w2=0.5, w3=0.1.

[0094] To better understand, the following example, evaluating the quality metrics of visual sparse features, will be used to explain the data quality assessment process in detail:

[0095] First, calculate the scores for each quality metric of the visual sparse feature:

[0096]

[0097] In the formula, σ(I) represents the standard deviation of the image grayscale value, and μ(I) represents the mean of the image grayscale value; the higher the score, the better the contrast (e.g., uniform lighting scene scores high, backlighting scene scores low).

[0098]

[0099] The second derivative of the pixel value is calculated by applying the Laplacian operator to the image; the variance of the derivative (i.e., the Laplacian variance in the formula) is calculated, and the larger the variance, the clearer the image (the sharper the edges); the score is normalized to the 0-1 range by setting a threshold (e.g., 200) through experiments.

[0100]

[0101] Effective pixels refer to pixels with grayscale values ​​in the range of [10, 245] (excluding overly dark / overly bright pixels); if the proportion of effective pixels is less than 60%, the weight is reduced.

[0102] The overall data quality score for visually sparse features can be calculated using the following formula:

[0103] q vision = ω4 × Contrast Score + ω5 × Sharpness Score + ω6 × Dynamic Range Score

[0104] The weights of each quality indicator can be flexibly set. For example, for daytime scenes, the focus can be on sharpness, such as setting w4=0.3, w5=0.5, w6=0.2; for nighttime scenes, the focus can be on contrast, such as setting w4=0.5, w5=0.3, w6=0.2.

[0105] The quality perception gating dynamically assigns initial weights to the radar sparse features and visual sparse features based on their combined scores using the following formula:

[0106]

[0107] In the formula, W radar W represents the initial weights of the radar. vision This represents the initial visual weights, where ∈ is a smoothing factor to avoid zero denominators; for example, ∈ is set to 0.01.

[0108] In an optional embodiment of this application, if W radar <0.3 and W vision If W is ≥0.7, then visual data can be completely trusted; if W vision If the value is less than 0.2 (e.g., in dense fog at night), visual data is disabled, and only radar is relied upon.

[0109] This embodiment uses quality-aware gating to quantitatively evaluate data quality and dynamically assign weights, achieving quality-aware fusion of radar and visual features. This provides reliable input for subsequent cross-modal consistency learning and helps support the accuracy of adversarial attack detection.

[0110] refer to Figure 3 In an optional embodiment of this application, the cross-attention mechanism for fine-tuning the fusion weights includes:

[0111] Construct query Q, key K, and value V for radar sparse features and visual sparse features respectively; when generating Q (query), K (key), and V (value), pre-weight the features using the initial weights:

[0112]

[0113] In the formula, This represents the linear transformation weight matrix from radar sparse features to the query space, and its function is to transform the radar sparse features F... r Project onto the query feature space; The weight matrix represents the linear transformation from visual features to the key space, and its function is to transform the visual sparse features F... v Projected onto the key feature space.

[0114] Pre-weighting features with initial weights weakens the influence of poor-quality features before they enter cross-attention, thus preventing low-quality features from dominating semantic association calculations.

[0115] Then, the semantic correlation between radar sparse features and visual sparse features is calculated to obtain cross-modal attention weights. Based on the cross-modal attention weights, the radar sparse features and visual sparse features are weighted and summed to obtain the cross-modal fusion feature. The calculation formula is as follows:

[0116]

[0117] Where F fused For cross-modal fusion features, A cross For cross-modal attention weights, For the transpose of cross-modal attention weights, V vision V is the value matrix of visual sparse features. radar Let Q be the value matrix of the sparse features of the radar, where softmax() represents the normalization function. radar The query vector representing the sparse features of radar, Q vision A query vector representing visually sparse features. The transpose of the key vector representing visual sparse features. The transpose of the key vector representing the sparse features of the radar is given, and d is a scaling factor used to prevent excessively large gradients from affecting convergence.

[0118] S4: Input the cross-modal fusion features into the autoencoder SLAE with a shared latent space for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space, and the decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features.

[0119] In an optional embodiment of this application, the shared encoder is a three-layer fully connected network, wherein the weights of the first two layers are shared in the radar feature processing branch and the visual feature processing branch, which is used to converge the radar sparse features and the visual sparse features to the same distribution pattern; the third layer is a dimension compression layer, which is used to compress high-dimensional features into the same latent space.

[0120] Specifically, the first two layers of the shared encoder share weights in the radar feature processing branch and the visual feature processing branch, and use the ReLU activation function for nonlinear transformation. The number of neurons in the first two layers can be set according to actual needs and the dimension of the input cross-modal fusion features. For example, if the dimension of the input cross-modal fusion features is B, the number of neurons in the first layer is set to B1 = 2B, and the number of neurons in the second layer is set to B2 = B. This shared weight design forces different modal features to gradually converge to the same distribution pattern during the transformation process, so as to facilitate subsequent comparison and learning in the same latent space.

[0121] The third layer of the shared encoder is a dimensionality compression layer, which compresses high-dimensional features into a dense latent space. Its number of neurons is significantly less than the first two layers, and it is used to further filter modality-specific noise, retaining only cross-modal common features.

[0122] It should be noted that the core principle of the shared encoder is to achieve alignment and dimensionality reduction of features from different modalities through weight sharing and layer-by-layer feature transformation. Specifically, the first two layers share weights, enabling radar and visual features to learn similar feature representations during the transformation process. For example, for the shape features of a target, the three-dimensional contour shown by radar point clouds and the two-dimensional edge information shown by visual images will gradually form similar feature expressions after being processed by the neural network layers with shared weights, thus solving the semantic gap problem caused by differences in cross-modal data structures to a certain extent. Through the nonlinear transformation of the first two layers, preliminary feature extraction and fusion are performed on the input features to uncover more representative features. The third layer, the dimensionality compression layer, further maps high-dimensional features to a low-dimensional latent space, enabling features from different modalities to be uniformly represented in the same space, facilitating subsequent consistency learning and reconstruction operations.

[0123] In this embodiment, the decoder includes a radar decoder and a visual decoder, which use independent parameters. The radar decoder is used to reconstruct the spatial structure features of the radar point cloud, and the visual decoder is used to reconstruct the texture detail features of the depth vision.

[0124] The decoder includes a radar decoder and a visual decoder, each with independent parameters. They are used to reconstruct the features mapped to the latent space, recovering representations similar to the original radar and visual features. The specific structure can be designed as a fully connected network or a convolutional neural network, depending on the actual needs, to adapt to the characteristics of different modal features.

[0125] For example, a radar decoder uses a multi-layer fully connected network with a structure of d→B3→B4→B5, where d is the dimension of the latent spatial features of the input, B3 and B4 are the number of neurons in the intermediate layers, and B5 is the dimension of the reconstructed radar features, used to reconstruct the spatial structural features of the radar point cloud.

[0126] Visual decoders can employ convolutional neural network structures, containing several convolutional layers, pooling layers, and upsampling layers, to reconstruct texture details from depth vision. For example, texture information in latent spatial features is first extracted using convolutional layers, and then the resolution of the image is gradually restored using upsampling layers, ultimately outputting reconstructed features similar to the original visual image.

[0127] Based on the characteristics of radar point cloud data, the radar decoder utilizes the weight parameters of a fully connected network to progressively expand and transform the latent spatial features, recovering information such as the three-dimensional spatial coordinates and reflection intensity of the radar point cloud, thereby reconstructing the spatial structural features of the radar point cloud. For example, by learning the mapping relationship between latent spatial features and original radar point cloud features, feature vectors in the latent space are mapped to three-dimensional space to generate the reconstructed radar point cloud.

[0128] Visual decoders utilize convolution and upsampling operations in convolutional neural networks to recover information such as texture, color, and contours from latent spatial features. Convolutional layers are used to extract local texture patterns from latent spatial features, while upsampling layers are used to restore low-resolution feature maps to the resolution of the original image, thereby reconstructing the texture details of depth vision.

[0129] In this embodiment, the cross-modal fused features obtained through a cross-modal attention mechanism are used as input to a shared encoder. The shared encoder first performs a nonlinear transformation on the input features using the first two fully connected layers with shared weights, causing the radar and visual features to gradually converge in feature representation. Then, a third dimensionality compression layer compresses the high-dimensional features into a dense latent space, resulting in a unified latent space feature representation. Next, the latent space features are fed into the radar decoder and the visual decoder respectively. The radar decoder, based on its own weight parameters, progressively transforms and expands the latent space features to reconstruct the spatial structure features of the radar point cloud, obtaining the radar reconstructed features. The visual decoder then uses convolutional neural network operations to recover the texture detail features of the visual image from the latent space features, obtaining the visual reconstructed features.

[0130] S5: Calculate the first Euclidean distance between the radar sparse features and the radar reconstructed features, and the second Euclidean distance between the visual sparse features and the visual reconstructed features. The sum of the first Euclidean distance and the second Euclidean distance is taken as the reconstruction error, as shown in the following formula:

[0131]

[0132] In the formula, L recon F represents the reconstruction error. r This indicates the sparse characteristics of radar. F represents the radar reconfiguration characteristics. v Indicates visual sparseness. This represents visual reconstruction features.

[0133] In abnormal situations, modal inconsistencies caused by tampering or attacks will manifest as a shift in the underlying representation, leading to significant reconstruction errors and triggering anomaly detection mechanisms.

[0134] S6: If the reconstruction error is greater than the preset threshold, it is determined that there is an adversarial attack; otherwise, it is determined that there is no adversarial attack.

[0135] In an optional embodiment of this application, the preset threshold is determined as follows: Under normal circumstances without adversarial attacks, multiple sets of multimodal data are collected, the reconstruction error corresponding to each set of data is calculated, and the maximum value of all reconstruction errors is taken as the preset threshold. This ensures that the errors of normal data are all below the threshold.

[0136] In an optional embodiment of this application, during model optimization, this scheme proposes to employ sparse pruning and 8-bit quantization techniques to reduce computational redundancy and optimize storage efficiency. The sparse pruning technique is based on the importance of weight magnitudes, removing redundant connections. An importance score is defined. Among them W ijThis is the weight matrix. The pruning strategy analyzes the neuron weight distribution in the later stages of training and removes neurons that have a smaller impact on the detection results, such as removing I... ij The lowest 30% weights are used to generate a sparse matrix, thereby reducing the number of parameters by more than 30%; while 8-bit quantization compresses the model weights from 32-bit floating-point numbers to 8-bit integers, as shown in the following formula:

[0137]

[0138] Among them W int8 W represents the quantized 8-bit integer weights. float This represents the original 32-bit floating-point weights, where μ is the mean of the weight distribution and σ is the standard deviation of the weight distribution. The range of 8-bit signed integers is [-128, 127]. Here, [-127, 127] is used to avoid overflow. After multiplying by 127, the standardized weights are scaled to a range close to [-1, 1] and mapped to 8-bit integers. round() represents rounding to the nearest integer to convert the scaled floating-point result into the nearest integer, resulting in 8-bit integer weights.

[0139] This process compresses the model size to one-quarter of its original size, effectively reducing memory usage and improving computational throughput. Furthermore, to reduce the computational complexity of the autoencoder, a bottleneck layer is introduced in the hidden layers. This retains the most critical information through dimensionality reduction, thereby optimizing the information compression process.

[0140] In addition to structural optimization, improvements will also be made to attention computation. Traditional Transformer computation is computationally complex, which is unacceptable for real-time demanding UAV missions. Therefore, in optional embodiments of this application, low-rank decomposition and window attention can also be employed to reduce computational complexity. The low-rank decomposition method approximates the decomposition of the attention matrix. Approximately The complexity is reduced from O(n 2 The computational complexity of attention calculations is reduced to O(nk), thus decreasing the computational burden. Window attention, through a local computation mechanism, performs attention operations only within adjacent windows, avoiding redundant computation of global features. These optimization methods significantly reduce the computational complexity of attention calculations and improve inference speed.

[0141] This embodiment innovatively overcomes the limitations of traditional fixed-weight fusion techniques by employing quality-aware gating. In nighttime scenes, visual data is significantly affected by illumination, while radar point clouds remain stable. In this case, the cross-attention mechanism automatically reduces the influence weight of visual data and increases the contribution of the radar point cloud to ensure the robustness of the system's perception. Conversely, in daylight conditions with ample illumination, visual data provides richer semantic information, so the cross-attention mechanism enhances the contribution of visual data to optimize the overall accuracy of perception.

[0142] This embodiment utilizes an autoencoder for anomaly detection against adversarial attacks, offering the following advantages: 1) As an unsupervised learning method, the autoencoder can learn the distribution patterns of perceptual data from unlabeled data, making it highly suitable for scenarios lacking a large number of labeled samples. By learning normal perceptual patterns, the autoencoder can effectively identify anomalous perceptions without relying on manually labeled data; 2) This model achieves comprehensive anomaly detection by fusing data from different modalities. By combining communication link features and spatial location features, the model can perform a more comprehensive analysis of anomalies in the robot's perception system, improving detection accuracy; 3) The autoencoder model has a simple structure, few parameters, and low computational overhead. This makes the model suitable for deployment on small UAV platforms with limited computing resources, enabling real-time anomaly detection while ensuring system efficiency and robustness.

[0143] In addition, this embodiment employs sparse pruning technology to scientifically and rationally remove 30% of redundant neurons, reducing unnecessary computational resource consumption. Furthermore, it utilizes 8-bit quantization technology to successfully compress the weight volume to 1 / 4 of its original size. Simultaneously, it combines low-rank decomposition and window attention to optimize the computational structure, comprehensively improving computational efficiency.

[0144] Based on the above embodiments of the adversarial attack detection method, this embodiment also provides an adversarial attack detection system based on a cross-modal attention mechanism, which can be used to implement the steps of the above method.

[0145] refer to Figure 4 The anti-attack detection system includes:

[0146] The acquisition module 41 is used to acquire multimodal data, including radar point cloud data and depth vision data;

[0147] Feature extraction module 42 is used to extract and reconstruct sparse features from the radar point cloud data and the depth visual data respectively, to obtain radar sparse features and visual sparse features.

[0148] The feature fusion module 43 is used to fuse the radar sparse features and visual sparse features based on a cross-modal attention mechanism to obtain cross-modal fused features. The cross-modal attention mechanism includes quality-aware gating and cross-attention mechanism. The quality-aware gating is used to dynamically allocate initial weights based on data quality, and the cross-attention mechanism is used to fine-tune the fusion weights based on the semantic correlation between modalities.

[0149] The feature reconstruction module 44 is used to input the cross-modal fusion features into an autoencoder SLAE in a shared latent space for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space, and the decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features.

[0150] The calculation module 45 is used to calculate the first Euclidean distance between the radar sparse features and the radar reconstructed features, and the second Euclidean distance between the visual sparse features and the visual reconstructed features, and to use the sum of the first Euclidean distance and the second Euclidean distance as the reconstruction error.

[0151] The judgment module 46 is used to determine that there is an adversarial attack if the reconstruction error is greater than a preset threshold; otherwise, it determines that there is no adversarial attack.

[0152] Various variations and specific examples of the methods provided in the above embodiments are also applicable to the system of this embodiment. Through the foregoing detailed description of the methods, those skilled in the art can clearly understand the implementation method of the system in this embodiment. For the sake of brevity, they will not be described in detail here.

[0153] To better execute the above methods, embodiments of this application also provide a computer device, such as... Figure 5 As shown, the computer device includes a processor 51, a memory 52, and a communication bus 53 for enabling communication between the processor and the memory.

[0154] Computer devices can be implemented in various forms, including mobile phones, in-vehicle computers, in-vehicle terminals, drone computers, and other devices.

[0155] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the methods provided in the above embodiments; the data storage area may store data involved in the methods provided in the above embodiments.

[0156] Optionally, the memory may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory exists independently and is connected to the processor via a communication bus, or the memory is integrated with the processor.

[0157] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0158] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0159] In an alternative embodiment, the computer device may further include a communication interface (not shown) for communication with other devices.

[0160] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the methods of the above embodiments.

[0161] This application also provides a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing any of the methods described in this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium (such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc.).

[0162] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for detecting adversarial attacks based on a cross-modal attention mechanism, characterized in that, The method includes: S1: Collect multimodal data, including radar point cloud data and depth vision data; S2: Perform sparse feature extraction and reconstruction on the radar point cloud data and depth visual data respectively to obtain radar sparse features and visual sparse features. S3: The radar sparse features and visual sparse features are fused based on a cross-modal attention mechanism to obtain cross-modal fused features; the cross-modal attention mechanism includes quality-aware gating and cross-attention mechanism, the quality-aware gating is used to dynamically allocate initial weights based on data quality, and the cross-attention mechanism is used to fine-tune the fusion weights based on the semantic correlation between modalities; S4: Input the cross-modal fusion features into an autoencoder SLAE with a shared latent space for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space. The decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features. S5: Calculate the first Euclidean distance between the radar sparse feature and the radar reconstructed feature, and the second Euclidean distance between the visual sparse feature and the visual reconstructed feature, and use the sum of the first Euclidean distance and the second Euclidean distance as the reconstruction error; S6: If the reconstruction error is greater than a preset threshold, it is determined that there is an adversarial attack; otherwise, it is determined that there is no adversarial attack.

2. The method as described in claim 1, characterized in that, In step S2, the sparse feature extraction and reconstruction includes: A complete dictionary was constructed, and the sparse coefficients were obtained by solving an optimization problem, namely: Where x is the input feature, D is the overcomplete dictionary, α is the sparsity coefficient, and λ is the sparsity control parameter; The input features are reconstructed based on the sparse coefficients to obtain the radar sparse features or visual sparse features.

3. The method as described in claim 1, characterized in that, In step S3, the dynamic allocation of initial weights for quality-sensing gating includes: The quality indicators of the radar sparse features are evaluated, and the quality indicators include at least one of point density, maximum detection range, and field of view. The quality metrics of the visual sparse features are evaluated, including at least one of contrast, sharpness, and dynamic range; Initial weights are assigned to the radar sparse features and visual sparse features based on the evaluation results, with higher quality features corresponding to larger initial weights.

4. The method as described in claim 1, characterized in that, In step S3, the cross-attention mechanism fine-tunes the fusion weights, including: Construct query Q, key K, and value V for the radar sparse features and visual sparse features respectively; The semantic correlation between the radar sparse features and the visual sparse features is calculated to obtain the cross-modal attention weights; Based on the cross-modal attention weights, the radar sparse features and visual sparse features are weighted and summed to obtain the cross-modal fusion features. The calculation formula is as follows: Where F fused For cross-modal fusion features, A cross For cross-modal attention weights, V vision V represents the value of the visual sparse feature. radar This represents the value of the radar sparse feature.

5. The method as described in claim 1, characterized in that, In step S4, the shared encoder is a three-layer fully connected network, wherein the weights of the first two layers are shared in the radar feature processing branch and the visual feature processing branch, which is used to converge the radar sparse features and the visual sparse features to the same distribution pattern; the third layer is a dimension compression layer, which is used to compress high-dimensional features into the same latent space.

6. The method as described in claim 1, characterized in that, In step S4, the decoder includes a radar decoder and a visual decoder, and the radar decoder and the visual decoder use independent parameters; the radar decoder is used to reconstruct the spatial structure features of the radar point cloud, and the visual decoder is used to reconstruct the texture detail features of the depth vision.

7. The method as described in claim 1, characterized in that, In step S6, the preset threshold is determined in the following way: under normal scenarios without adversarial attacks, multiple sets of the multimodal data are collected, the reconstruction error corresponding to each set of data is calculated, and the maximum value of all reconstruction errors is taken as the preset threshold.

8. The method according to any one of claims 1-7, characterized in that, It also includes a lightweight optimization step, which includes: Sparse pruning is performed on the shared encoder and decoder, removing the 30% of neurons with the lowest weight magnitudes; The model weights are quantized to 8-bit, converting the 32-bit floating-point weights into 8-bit integer weights. The cross-attention mechanism is optimized using low-rank decomposition and window attention to reduce computational complexity.

9. An adversarial attack detection system based on a cross-modal attention mechanism, characterized in that, include: The acquisition module is used to acquire multimodal data, including radar point cloud data and depth vision data; The feature extraction module is used to extract and reconstruct sparse features from the radar point cloud data and the depth visual data respectively, to obtain radar sparse features and visual sparse features. The feature fusion module is used to fuse the radar sparse features and visual sparse features based on a cross-modal attention mechanism to obtain cross-modal fused features; The cross-modal attention mechanism includes quality-aware gating and cross-attention mechanism. The quality-aware gating is used to dynamically allocate initial weights based on data quality, and the cross-attention mechanism is used to fine-tune the fusion weights based on the semantic relevance between modalities. The feature reconstruction module is used to input the cross-modal fusion features into an autoencoder SLAE with a shared latent space for consistency learning. The autoencoder SLAE includes a shared encoder and a decoder. The shared encoder maps the cross-modal fusion features to the same latent space, and the decoder is used to reconstruct the mapped features to obtain radar reconstruction features and visual reconstruction features. The calculation module is used to calculate the first Euclidean distance between the radar sparse features and the radar reconstructed features, and the second Euclidean distance between the visual sparse features and the visual reconstructed features, and to use the sum of the first Euclidean distance and the second Euclidean distance as the reconstruction error. The judgment module is used to determine that there is an adversarial attack if the reconstruction error is greater than a preset threshold. Otherwise, it is determined that there is no counterattack.

10. A computer device, characterized in that, It includes a processor, a memory, and a communication bus, wherein the communication bus is used to establish a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the anti-attack detection method as described in any one of claims 1-8.

Images