An attack countermeasure processing method and device, a storage medium and an electronic device

Abstract

This specification discloses an adversarial attack handling method, apparatus, storage medium, and electronic device. The method includes: monitoring multimodal input data of a target multimodal large model and extracting multimodal input features; calculating initial modal trust features for each modal channel based on feature distribution anomalies; performing causal reasoning verification and counterfactual reasoning verification based on the multimodal input features to obtain causal anomaly path detection results and attack detection results; jointly modulating the initial modal trust features using the causal and attack detection results to generate security weight vectors for each modal channel; integrating the initial modal trust features, various detection results, and security weight vectors to perform security fusion evaluation on the input features to obtain input security decision results; and controlling whether input data flows to the target large model accordingly. Using this specification, accurate identification and defense against unknown adversarial attacks on multimodal inputs are achieved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium, and electronic device for handling adversarial attacks. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, multimodal large modeling (MLLM) (such as models based on CLIP, BLIP, etc.) has been widely used in transactional scenarios such as intelligent customer service, autonomous driving, and content compliance review. However, with the continuous increase in the depth of multimodal feature fusion, the input security attack surface of multimodal large model application systems has also expanded exponentially, leading to increasingly severe cross-modal adversarial attacks during the inference phase of multimodal large models. Cross-modal adversarial attacks have become a serious security threat. Attackers can exploit the inconsistencies, dominance biases, and cross-modal alignment vulnerabilities in the multimodal input data used as input to the large model to mislead the decision-making process of the multimodal large model by perturbing a certain modality.

[0003] Currently, mainstream defense solutions in the industry often suffer from insurmountable engineering limitations. For example, consistency detection mechanisms based on static similarity thresholds are coarse-grained anomaly detection methods, which are prone to misjudging rare but legitimate semantic combinations as attacks, leading to a high false positive rate. While purification techniques based on latent space iterative optimization (such as denoising diffusion processes) offer better defense, they come with extremely high computational complexity and inference latency, making them unsuitable for the millisecond-level real-time online transaction requirements of industry, and they easily erase normal high-frequency semantic details. Furthermore, traditional multimodal adversarial training is not only computationally expensive but also prone to catastrophic forgetting of the model's generalization ability and is almost ineffective against unknown zero-day attacks.

[0004] Therefore, there is an urgent need in this field for a multimodal adversarial attack defense method that can achieve low latency and high precision without affecting the original performance of the main model. Summary of the Invention

[0005] This specification provides an embodiment of a method, apparatus, storage medium, and electronic device for combating attacks. The technical solution is as follows: Firstly, embodiments of this specification provide a method for handling counter-attacks, the method comprising: Monitor the multimodal input data for the target multimodal large model input, extract the multimodal input features of the multimodal input data, and calculate the initial modal confidence features of each modal channel based on the feature distribution anomaly. Based on the multimodal input features, causal reasoning verification for causal decision paths is performed to obtain causal anomaly path detection results, and counterfactual reasoning verification for attack types is performed based on the multimodal input features to obtain attack detection results. Based on the causal anomaly path detection results and the attack detection results, the initial modal trust features are jointly modulated to obtain the security weight vector of each modal channel of the multimodal input data; Based on the initial modal trust features, the causal anomaly path detection results, the attack detection results, and the security weight vector, the multimodal input features are subjected to input security fusion evaluation processing to obtain input security decision results. Based on the input security decision results, the multimodal input data is controlled to flow to the target multimodal large model.

[0006] Secondly, embodiments of this specification provide an anti-attack processing device, the device comprising: The input monitoring module is used to monitor multimodal input data for the target multimodal large model, extract multimodal input features from the multimodal input data, and calculate the initial modal confidence features of each modal channel based on feature distribution anomalies. The attack detection module is used to perform causal reasoning verification for causal decision paths based on the multimodal input features to obtain causal abnormal path detection results, and to perform counterfactual reasoning verification for attack types based on the multimodal input features to obtain attack detection results. The attack detection module is used to perform joint modulation processing on the initial modal trust feature based on the causal abnormal path detection result and the attack detection result to obtain the security weight vector of each modal channel of the multimodal input data. The input decision module is used to perform input security fusion evaluation processing on the multimodal input features based on the initial modal trust features, the causal abnormal path detection results, the attack detection results, and the security weight vector to obtain the input security decision results, and to control whether the multimodal input data is transferred to the target multimodal large model based on the input security decision results.

[0007] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0008] Fourthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0009] Fifthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0010] Sixthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0011] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: In one or more embodiments of this specification, initial modal trust features are obtained by extracting multimodal input features and measuring feature distribution anomalies. Logical rationality and decision-making continuity are verified using a dual-engine approach combining causal reasoning and counterfactual reasoning. Anomaly and attack detection results are used to modulate the initial trust to dynamically reconstruct the security weight vectors of each modal channel. Finally, the features are subjected to security fusion evaluation, and adaptive hierarchical data flow control is executed based on the evaluation results. This solves the input security problem of large multimodal models in artificial intelligence applications facing cross-modal inconsistency attacks, modality dominance bias, and unknown covert adversarial attacks. It upgrades from overall consistency judgment to multi-dimensional, three-dimensional anti-attack defense including modal-level trust, causal logic, and counterfactual inference, enabling it to cope with unknown input threats and improving the ability to identify input attacks. The entire detection process does not intrude on the original large multimodal model, minimizing the impact on the original system performance while improving security. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a scenario for an anti-attack processing system provided in the embodiments of this specification; Figure 2 This is a flowchart illustrating an anti-attack processing method provided in the embodiments of this specification; Figure 3 This is a schematic diagram of a process for determining initial modality confidence features provided in the embodiments of this specification; Figure 4 This is a schematic diagram of a causal reasoning verification process provided in the embodiments of this specification; Figure 5This is a schematic diagram of a counterfactual reasoning verification process provided in the embodiments of this specification; Figure 6 This is a schematic diagram of an input security decision processing method provided in the embodiments of this specification; Figure 7 This is a schematic diagram of the structure of an anti-attack processing device provided in the embodiments of this specification; Figure 8 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0014] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0015] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0016] Please see Figure 1 This is a schematic diagram of a scenario for an anti-attack processing system provided in this specification. Figure 1 As shown, the anti-attack processing system may include at least a client cluster and a service platform 100.

[0017] The client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0018] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, electronic device, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0019] The service platform 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet server device, or a workstation, mainframe, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0020] In one or more embodiments of this specification, the service platform 100 can establish a communication connection with at least one client in the client cluster, and complete the data interaction during the adversarial attack processing based on the communication connection, such as online transaction data interaction. The client can receive multimodal input data from the user to the target multimodal large model maintained by the service platform 100. The service platform 100 or the client can execute the adversarial attack processing method of one or more embodiments of this specification.

[0021] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster for interactive communication via a network. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, Universal Serial Bus (USB), or Controller Area Network (CAN). In one or more embodiments of the specification, technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0022] The embodiments of the anti-attack processing system provided in this specification and the anti-attack processing methods in one or more embodiments belong to the same concept. The execution subject corresponding to the anti-attack processing methods involved in one or more embodiments in the specification can be an electronic device, which can be the aforementioned service platform 100 or a client.

[0023] The present specification will now be described in detail with reference to specific embodiments.

[0024] In one embodiment, such as Figure 1 As shown, a method for handling adversarial attacks is proposed. This method can be implemented using a computer program and can run on an adversarial attack processing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The adversarial attack processing device can be an electronic device.

[0025] The scenarios in which the counter-attack handling methods shown in this specification are applied are not limited; This can be applied to scenarios involving high concurrency and transaction continuity assurance in the cloud (such as intelligent content compliance review, intelligent customer service, and search engines): In a cloud environment with abundant computing power, a target multimodal large model needs to concurrently process massive amounts of query and interaction requests for combined data such as images, text, audio, and video. For example, in a content review platform, an attacker might submit an image hiding highly dangerous contraband and maliciously alter the text prompt to: "This is a beautiful everyday photo," attempting to launch a cross-modal inconsistency attack.

[0026] This technology can be applied in scenarios where edge computing power is limited and high security is required (such as financial-grade identity verification and autonomous driving). In environments with edge device deployment or extremely high security standards, the tolerance for false positives and false negatives in security audits is low, and millisecond-level real-time online defense is required. For example, in the alignment space of cameras and radar relied upon by autonomous driving, or in the joint reasoning of facial images and voiceprints in financial identity verification, attackers often attempt to inject covert adversarial noise into high-frequency facial textures or road sign vision.

[0027] The adversarial attack handling method described in this specification can be fully adapted to multimodal and complex application scenarios, from large-scale content filtering in the cloud to autonomous driving and financial verification on the edge, by flexibly configuring the depth of feature extraction and the backend interception route.

[0028] Specifically, the counter-attack handling method includes: S102: Monitor the multimodal input data for the target multimodal large model input, extract the multimodal input features of the multimodal input data, and calculate the initial modal confidence features of each modal channel based on the feature distribution anomaly. Multimodal input data refers to a heterogeneous set of raw data received by a front-end transaction system (such as application software on a client), containing two or more different information carriers (such as images, text, audio, etc.), and intended to be input into a target multimodal large model for joint reasoning.

[0029] Multimodal input features refer to the deep semantic vectors or tensors obtained by mapping the aforementioned multimodal input data from the original physical space (such as pixel matrices or character sequences) to a high-dimensional latent space through a heterogeneous feature encoding network.

[0030] Feature distribution anomaly refers to the degree to which the probability distribution or density distribution of the extracted multimodal input features in the latent space deviates from the pre-constructed benign and normal historical probability distribution. This degree of anomaly is usually used to characterize whether the data has been subjected to underlying adversarial perturbations or belongs to unknown malicious samples.

[0031] Initial modal confidence features refer to the quantitative characteristics that initially determine the security confidence baseline of each modal channel based on shallow anomaly calculations such as feature distribution drift, feature quality, and internal consistency of single-modal bottom-level data.

[0032] Target multimodal large model refers to a multimodal large model (MLLM) that receives input from various heterogeneous information carriers (such as images, text, audio, etc.) and performs joint feature mapping and target transaction prediction in a high-dimensional latent space.

[0033] In a schematic manner, a bypass mirror gateway monitors and intercepts multimodal input data targeting a large multimodal model in real time. In parallel, a heterogeneous modal encoder, independent of the target large multimodal model, extracts high-dimensional deep semantic representations of each modal channel in the latent space as multimodal input features. The projection density distribution (features) of each channel's multimodal input features in the high-dimensional space is calculated in real time, and a pre-stored baseline benign historical probability distribution (features) is retrieved. The KL divergence value between the projection density distribution and the baseline benign historical probability distribution is calculated to accurately quantify the degree of feature distribution anomalies. Simultaneously, the clarity and integrity features of each multimodal input feature are determined as feature quality scores, and the matching coefficient (highest value) between the multimodal input features and known attack patterns is calculated. The feature quality score, the degree of feature distribution anomaly, and the matching coefficient are weighted and fused to output the initial modal confidence feature for each modal channel.

[0034] S104: Based on the multimodal input features, perform causal reasoning verification for causal decision paths to obtain causal abnormal path detection results, and perform counterfactual reasoning verification for attack types based on the multimodal input features to obtain attack detection results; The causal decision path refers to the logical link in a structural causal model that starts from each independent modal input feature or intermediate implicit representation node, passes through a series of directed topological connections, and finally transmits and influences the prediction decision result node of the large model.

[0035] Causal reasoning verification refers to the defensive deduction process that measures the average causal effect of a single modality on the final decision by using pre-set intervention methods (such as blocking the feature flow of a specific variable), and then compares it with the benchmark causal graph to identify false causal paths that violate objective common sense.

[0036] Counterfactual reasoning verification refers to setting hypothetical conditions in the latent space (such as keeping other input data modes frozen and applying controlled perturbations or masks only to the target mode), constructing counterfactual samples in real time, and observing continuous changes in the decision surface of the large model in order to detect the defense mechanisms against unknown adversarial attacks.

[0037] The causal anomaly path detection result refers to the qualitative label output by the causal reasoning verification module, which is used to indicate whether there is false causal logic in the current multimodal input data during the reasoning stage, which is forged by adversarial examples and excessively dominates the final decision.

[0038] Attack detection results refer to the structured quantitative information output by the counterfactual reasoning verification module. These typically include an attack strength scalar calculated based on the decision boundary flip slope, and attack type labels (such as transfer attack, hidden attack, and dominant attack) based on multi-channel response difference classification.

[0039] As an illustration, after acquiring multimodal input features, the causal verification stream and the counterfactual detection stream are simultaneously started for parallel cross-validation; In the causal verification flow: the input feature variables of each modality, the intermediate implicit representation variables, and the final decision state of the large model are mapped to network nodes in the causal topology model of the directed acyclic graph. The feature flow input values ​​of non-target modal channels are forcibly blocked by the interference quantizer, and the average causal effect scalar value of the target test modality on the final decision state is isolated and calculated. When it is determined that the average causal effect scalar value suddenly exceeds the threshold set by the normal historical benchmark, the electronic device system determines that there is a false causal path forged by the adversarial sample in the current input, and outputs the causal abnormal path detection result of the active state. Meanwhile, in the counterfactual detection stream, under the condition that the features of other modal channels are kept frozen in the latent space, a small amount of Gaussian noise scalar is injected into the deep embedding vector of the target modality in the multimodal input features to generate multiple sets of counterfactual sample feature groups. Then, the magnitude of the predicted activation value change generated by the surrogate decision network (a local surrogate model after local parameter lightweighting of the target multimodal large model) when processing the original multimodal input features and the counterfactual sample feature groups is captured to verify whether the magnitude of the predicted activation value change violates the Lipshitz continuity, that is, to verify whether the magnitude of the predicted activation value change of the surrogate decision network remains within a bounded smooth linear scale range under extremely small input perturbations. If the predicted activation value changes in a non-linear, abrupt manner, meaning that the injection causes a significant difference in the output prediction value of the local agent decision network on the decision surface (e.g., the change in the output prediction value exceeds the difference threshold), then the topological slope is calculated. This involves extracting the ratio between the difference in the output prediction value (output change difference) and the input micro-Gaussian noise scalar (input perturbation) to characterize the gradient change rate, and quantifying this topological slope as an attack strength scalar. Subsequently, based on the feedback response differences of the counterfactual sample feature group across different modal channels, the target attack event is classified into transfer attacks, concealment attacks, dominant attacks, and non-attack events to generate attack type labels. The attack strength scalar and attack type label are packaged and output as the attack detection result.

[0040] S106: Based on the causal abnormal path detection results and the attack detection results, the initial modal trust features are jointly modulated to obtain the security weight vector of each modal channel of the multimodal input data; Joint modulation processing refers to the process of using the initial feature quantity of a certain dimension of the initial modal trust feature as the base, introducing the detection signals of the causal abnormal path detection result and the attack detection result as the penalty or incentive factor for the specific value, and adjusting the feature values ​​of each channel based on the factor.

[0041] In a schematic manner, the initial modal trust features of each modal channel are extracted as the base of the benchmark weights. Simultaneously, the Boolean values ​​of the causal anomaly path detection results and the attack strength scalars in the attack detection results are compared with a preset hard security threshold. If the causal anomaly path detection results show a false causal relationship or the attack strength scalar exceeds the hard security threshold, a preset absolute zero mask is generated for the attacked modal channel. The preset absolute zero mask is then multiplied by the initial modal trust features corresponding to that modality to clear the weight of the suspicious modality to zero. For the other modal channels that have not triggered an alarm signal, the values ​​of their initial modal trust features are retained. Subsequently, through linear summation and scaling operations, a set of discrete security weight vectors composed of absolute zero values ​​and original trust ratio values ​​are output, thereby completing the secure reconstruction of the weights with extremely low computational overhead and extremely high execution efficiency.

[0042] In one feasible implementation, the joint modulation processing of the initial modal trust features based on the causal anomaly path detection results and the attack detection results is specifically performed to obtain the security weight vector of each modal channel of the multimodal input data. This can be done in the following way: Step A2: Analyze the causal anomaly path detection results and the attack detection results to determine the modal trust type corresponding to each modal channel; Modal trust type refers to the security status classification label assigned to each modal channel after comprehensively analyzing qualitative causal logic anomaly signals and quantitative adversarial attack signals. It is usually configured with different levels of classification labels such as high trust modality / secure modality, suspicious modality / contaminated modality, and potential victim modality (for transfer attack scenarios).

[0043] Step A4: Adjust the weights of each modal channel in the initial modal trust feature based on the modal trust type to obtain the security weight vector of each modal channel of the multimodal input data.

[0044] Weight adjustment refers to the nonlinear numerical calculation action that, based on the modal trust type assigned to each modal channel, calls the penalty or incentive operator corresponding to the modal trust type to scale, exponentially decay, or hard-zero the original initial modal trust feature of that channel.

[0045] Indicatively, step A2 reads the characteristics of each channel of the multimodal input data and parses the causal anomaly path detection results and attack detection results in the input buffer. Cross-logic judgment is used to determine the modal trust type corresponding to each modal channel. Specifically, if the causal anomaly path detection result for a certain modality is found to be active or the type label in the attack detection result indicates a dominant attack, then that modality is marked as an extremely dangerous suspected modal type. If a transfer attack label is parsed, both the disturbance source and the affected modal are marked as potential victim modal types, while modal channels that have not triggered any abnormal signals are marked as high-trust modal types. Step A4 involves dynamically adjusting the weights of each modal channel in the initial modal trust features based on the aforementioned modal trust types. A gain amplification function is matched to high-trust modal types to increase their weight ratio; a linear decay function is matched to potentially victimized modal types to perform proportional weight reduction; and a steep exponential penalty operator is matched to suspicious modal types to forcibly reduce their initial modal trust feature values ​​to near zero. Finally, the adjusted values ​​of each channel are normalized in the feature space to obtain a security weight vector for each modal channel of the multimodal input data with continuous floating-point precision. By employing the aforementioned mechanism of qualitative classification followed by quantitative calculation, complex attack signals are transformed into clear classification labels. This not only makes the joint modulation logic clearer and greatly enhances the anti-evasion capability but also builds a weight isolation wall, ensuring that input attack-type features are completely isolated before entering the final decision network.

[0046] S108: Based on the initial modal trust features, the causal abnormal path detection results, the attack detection results, and the security weight vector, the multimodal input features are subjected to input security fusion evaluation processing to obtain input security decision results. Based on the input security decision results, the multimodal input data is controlled to flow to the target multimodal large model.

[0047] Input security fusion evaluation processing refers to the process of performing a multi-dimensional comprehensive security posture assessment on multimodal input features after integration, in order to quantify the overall decision confidence level.

[0048] Input security decision results refer to the structured control instructions generated after the final integration evaluation is completed, which are used to instruct the underlying routing gateway to perform specific hierarchical response actions (such as full access, downgraded flow, physical circuit breaking).

[0049] Flow control refers to physical intervention actions implemented on the original multimodal input data packets based on the input security decision results, including full pass-through, discarding specific carrier channel packets, and directly blocking communication links.

[0050] Indicatively, a modal trust score is calculated based on initial modal trust features and security weight vectors, and the causal anomaly path detection results and attack detection results are simultaneously analyzed to determine the severity of the current attack. Based on the attack severity and modal trust score, a graded input security decision result is determined (which may include the attack severity and the modal trust level corresponding to each modality). In response to a moderate attack severity, the input security decision result is configured as a flexible degradation instruction to exclude suspicious modalities and use other modal decisions. Finally, based on the input security decision result, state transition control processing is executed to intercept and exclude suspicious modal data streams in the multimodal input data, and control the remaining healthy modal data streams to flow to the target multimodal large model. This achieves accurate security interception while maximizing the continuous operation of high-concurrency transactions.

[0051] In the embodiments of this specification, initial modal trust features are obtained by extracting multimodal input features and measuring feature distribution anomalies. The logical rationality and decision-making continuity are verified by combining causal reasoning and counterfactual reasoning dual engines. Anomaly and attack detection results are used to modulate the initial trust to dynamically reconstruct the security weight vectors of each modal channel. Finally, the features are subjected to security fusion evaluation, and adaptive hierarchical data flow control is executed based on the evaluation results. This solves the input security problem of large multimodal models in artificial intelligence applications facing cross-modal inconsistency attacks, modality dominance bias, and unknown covert adversarial attacks. It upgrades from overall consistency judgment to multi-dimensional, three-dimensional anti-attack defense including modal-level trust, causal logic, and counterfactual inference, which can cope with unknown input threats and improve the ability to identify input attacks. The entire detection process does not intrude on the original large multimodal model, minimizing the impact on the original system performance while improving security.

[0052] Optional, please see Figure 3 , Figure 3 This is a flowchart illustrating an initial modal confidence feature determination method proposed in this specification. Specifically, the method for calculating the initial modal confidence feature of each modal channel by measuring the multimodal input features based on feature distribution anomalies can be referenced as follows: S202: For the multimodal input data, the heterogeneous modal encoder is invoked in parallel to extract the deep semantic feature representation of each modal channel; Heterogeneous modal encoders refer to specific deep neural network architectures used to independently map raw input data from one or more modalities (such as pixel arrays of images, discrete character sequences of text, and continuous waveforms of audio) to a unified high-dimensional latent space and output corresponding deep semantic vector representations (e.g., feature encoders that can be locally replicated from the feature extraction backbone network of a target multimodal large language model). For example, image modalities use visual encoders to extract features; text modalities use text encoders to extract semantic features; and audio modalities use audio encoders to extract acoustic features. S204: Based on the deep semantic feature representation, perform multimodal anomaly analysis from the feature distribution dimension, modal internal consistency dimension, and historical anomaly pattern matching dimension to generate feature quality evaluation index, anomaly degree evaluation index, modal historical reputation evaluation index, and cross-modal consistency evaluation index. Multimodal anomaly analysis refers to a process that does not rely solely on a single boundary detection, but rather integrates multiple dimensions such as the statistical divergence of feature distribution, the signal-to-noise ratio of high-frequency activation features inherent in the modality, and an external historical security fingerprint database to conduct comprehensive security detection of deep semantic features.

[0053] Feature quality evaluation metrics characterize the semantic clarity and completeness of features; anomaly evaluation metrics characterize the drift of the current feature from the benign probability distribution; modal history reputation evaluation metrics characterize the security score baseline of a specific modal channel in past defense cycles; cross-modal consistency evaluation metrics characterize the logical rationality of the alignment of semantic vectors of each modality.

[0054] S206: The initial modal trust feature is obtained by normalizing the feature quality evaluation index, anomaly degree evaluation index, modal history reputation evaluation index and cross-modal consistency evaluation index.

[0055] Indicatively, after receiving multimodal input data, step S202 calls the pre-deployed heterogeneous modal encoder in parallel to extract the deep semantic feature representations of each modal channel in the latent space; step S204 calculates the Euclidean distance between the deep semantic feature representations and the local cache standard distribution boundary to generate an anomaly evaluation index. Frequency domain analysis or high-pass filtering is performed on the deep semantic feature representation to separate high-frequency activation features from low-frequency core semantic features. The ratio of the energy value of the high-frequency activation features to the total feature energy value is calculated as a signal-to-noise ratio penalty. A feature quality evaluation index is generated by weighting the sparsity and information entropy of the deep semantic features in the latent space. This index can accurately quantify the severity of how low-level adversarial noise (usually manifested as abnormal high-frequency perturbations) masks the true semantics.

[0056] The system reads the historical attack frequency and successful interception count of a specific modality channel within a preset time window from the security defense log, using this as a time decay variable. Based on the initial inherent reputation score of this modality, a preset exponential decay function (i.e., the more frequent the recent attacks, the more severe the decay) is called to perform dynamic decay calculations and generate historical reputation evaluation indicators for the modality. This transforms prior security knowledge in the time dimension into numerical parameters.

[0057] The deep semantic feature representations of different modal channels (such as visual and text) are mapped to a multimodal aligned shared grid space of a unified dimension. The cosine similarity between the feature vectors of each modality in the shared space is calculated. The cosine similarity value is then thresholded and smoothed as a cross-modal consistency evaluation index. When executing step S206, the initial modal trust features are generated by normalizing the extracted feature quality evaluation index, anomaly degree evaluation index, modal history reputation evaluation index, and cross-modal consistency evaluation index using basic linear scaling rules, thereby completing the preliminary trust assessment with low time complexity and computational cost.

[0058] Optional, please see Figure 4 , Figure 4 This is a flowchart illustrating a causal reasoning verification process proposed in this specification. Specifically, to obtain causal anomaly path detection results by performing causal reasoning verification based on the multimodal input features for causal decision paths, the following method can be used: S302: Based on the multimodal input features, construct a causal graph between each modal feature and the security decision result, wherein the graph nodes in the causal graph are configured as each modal feature, intermediate representation and security decision result, and the graph edges in the causal graph are configured as intermodal causal relationships and feature decision influence relationships; Causal graphs refer to topological network models that use the data structure of directed acyclic graphs (DAGs) to represent the objective dependence of variables on driving mechanisms during the forward propagation and feature fusion of multimodal data within a neural network.

[0059] Intermediate representation refers to the hidden layer state vector or latent space fusion feature tensor generated after the multimodal input features are processed by a self-attention mechanism or a cross-modal cross-attention mechanism before flowing to the final classification head used for security decision-making.

[0060] Intermodal causal relationships refer to the directed edges in a causal graph that connect different modal nodes or intermediate representation nodes. They are used to characterize how the input state of a certain modal feature interferes with or restricts the representation of another modal feature at the fusion level.

[0061] Feature-decision influence relationship refers to the directed edges in the causal graph that run directly from feature nodes at each level to the final security decision result node. These edges represent the direct driving weights of features at a specific level on the final classification probability output of the security decision.

[0062] To illustrate, after extracting multimodal input features, in order to prevent unaudited potentially malicious data from directly flowing and contaminating the downstream target multimodal large model, a local proxy fusion network based on the target multimodal large model and deployed locally in a physically isolated pre-security detection environment is invoked to perform security pre-inference, thereby completing the graph node entity mapping. Specifically, the underlying semantic feature vectors of the multimodal input features in their respective single-modal channels are defined as the modal feature nodes of the causal graph. At the same time, the deep network fusion structure of the local proxy fusion network during the forward pre-inference process is monitored in real time. The cross-modal joint attention matrix and latent space feature splicing tensor output in real time from the deep network fusion structure (e.g., cross-modal cross-attention layer or multi-layer perceptron fusion block) are extracted and defined as the intermediate representation node of the causal graph. The pre-inference security confidence score output by the final classification head of the local proxy fusion network is defined as the security decision result node. After completing the physical mapping of all nodes, a causal discovery algorithm (such as PC algorithm or FCI algorithm) based on Conditional Mutual Information (CMI) is initiated. In the high-dimensional feature space composed of multimodal input features and their derived intermediate representations, the conditional independence between the aforementioned types of nodes is calculated pairwise. If a significant distributional association that breaks statistical independence exists between any two nodes, an initial connection graph edge is generated between the two nodes. Furthermore, to determine the absolute direction of causal driving, the attention weight gradient flow generated by the local agent fusion network in the deep network fusion structure during pre-deduction is extracted. The natural physical transmission of this attention weight gradient flow is then utilized. The system performs directed configuration on the edges of the generated initial connection graph. Specifically, the system configures the graph edges representing the information interaction and intervention of different modal feature flows at intermediate representation nodes as intermodal causal relationships, and configures the graph edges that directly point from any modal feature node or intermediate representation node to the final classification head output as feature decision influence relationships. Finally, the system uses a graph structure generator to dynamically assemble and generate a high-fidelity real-time directed acyclic causal topology graph as a causal graph. This causal graph can rigorously and losslessly pre-parse the complex nonlinear data dependencies and causal driving links inside the multimodal input data without calling the target multimodal large model.

[0063] S304: Perform causal effect reasoning analysis based on the causal graph to calculate the direct causal effect characteristics of each modal channel on the security decision result and the interactive causal effect characteristics between each modal channel; Causal effect reasoning analysis refers to the quantitative calculation process that, based on the established causal graph topology, introduces pre-set intervention methods (such as truncating specific graph edges or fixing specific node variables) to forcibly change the natural observation distribution of data flow, thereby accurately extracting the pure causal influence of the target node on the result node.

[0064] The direct causal effect characteristic refers to the expected change in probability of the final security decision result when a disturbance or intervention is applied to a specific modal channel while keeping the values ​​of all other modal channels and intermediate representation nodes rigidly fixed (i.e., freezing irrelevant paths).

[0065] The interactive causal effect characteristic refers to the nonlinear synergistic or antagonistic effect produced when multiple modal channels are input together. It is characterized as the difference (or residual) between the overall causal effect produced by multimodal joint intervention and the sum (or product) of the causal effects produced by each single modality's independent intervention.

[0066] In a schematic manner, causal effect pre-inference parsing is initiated for each modal feature node in the causal graph. First, the quantitative extraction of direct causal effect features is performed: the mathematical interference quantifier of causal inference is simulated with code-level logic in the local agent fusion network; specifically, using the backdoor adjustment criterion, the cross-modal cross-attention weight matrix between the current target modal feature node and other modal feature nodes is forcibly masked or cut off in computer memory, thereby physically blocking false promiscuous paths; then, under the premise of strictly keeping the activation values ​​of all other modal feature vectors frozen in the latent space (i.e. blocking their gradient updates and feature feedforward), a pre-set controlled Gaussian perturbation vector is injected into the latent space input of the target modal feature, and the expected value of the pre-simulated safe decision probability output of the final classification head of the local agent fusion network after undergoing the perturbation is captured; further, the expected probability value after the perturbation injection and the original expected probability value before the perturbation injection are extracted, the absolute change difference between the two is calculated, and this change difference is directly used as the direct causal effect feature for quantifying the independent driving force of the target modal channel.

[0067] Furthermore, joint intervention analysis is performed to extract interactive causal effect features: the above-mentioned controlled unit Gaussian perturbation is simultaneously applied to two or more related modal feature nodes (such as visual nodes and auditory nodes) in the causal graph, and the total variation of the expected overall pre-simulated security decision probability output by the local agent fusion network under the joint intervention state is recorded; then, nonlinear residual stripping calculation is performed to extract the total variation, and the algebraic sum of the direct causal effect features of each target modality participating in this joint intervention when it is independently intervened is subtracted, thereby stripping out the remaining nonlinear collaborative residual terms that cannot be explained by the linear superposition of a single modality; finally, the nonlinear collaborative residual terms are mapped to the [0,1] floating-point interval through a preset Softmax or Sigmoid normalization function, and the output is an interactive causal effect feature that accurately represents the collaborative or antagonistic driving force between each modal channel.

[0068] The entire feature quantification process is completed in a closed loop within the local proxy network sandbox, without any real data interaction with the downstream target multimodal large model. This ensures both the extremely high theoretical accuracy of causal effect quantification and eliminates the risk of spillover contamination from unknown adversarial samples.

[0069] S306: Based on the direct causal effect features and the interactive causal effect features, identify key causal paths from the causal graph, and based on the key causal paths, identify abnormal paths to obtain causal abnormal path detection results.

[0070] A critical causal path refers to one or more topological links in a causal graph with quantified weights that start from a certain modal feature node, pass through a series of directed edges, and finally reach the safe decision result node, and whose cumulative causal effect (or bottleneck connectivity strength) occupies an absolutely dominant position globally.

[0071] Anomaly path identification refers to the graph computation process that mathematically compares the key causal path extracted in real time with the preset benign benchmark logic in two dimensions: topological structure (node ​​connectivity) and weight distribution (the magnitude of the causal effect of the edge), in order to determine whether the current decision is hijacked by a certain edge feature or malicious noise.

[0072] The causal anomaly path detection result refers to the output structured state signal, which is used to determine whether there is a false causal driving link in the current multimodal input that violates objective physical common sense or transaction logic.

[0073] Indicatively, after obtaining the direct causal effect features and interactive causal effect features, a lightweight greedy algorithm is directly used to perform single-step reverse tracing starting from the decision result node. In each step of backtracking, only the in-degree edge with the largest causal effect coefficient value is selected for reverse jump. All candidate in-degree edges connected to the current node are read, and the values ​​of the direct causal effect features and interactive causal effect features corresponding to each candidate in-degree edge are extracted. The absolute values ​​of the two are linearly weighted and summed to calculate the path confidence score of the current jump. The in-degree edge with the largest path confidence score is selected as the only jump path for reverse tracing, and the other branches are discarded. This single-step reverse jump process is repeated until the bottom feature node of a certain modality is reached. This series of node sequence arrays spliced ​​together by local maximum confidence edges is directly stored and output as the key causal path. After locking down the critical causal path, the pre-set static rule matching engine is invoked to perform abnormal path identification. First, the source node identifier (i.e., the modal channel that initiates the dominant drive) is extracted from the node sequence array, and the direct causal effect feature value corresponding to the source node is extracted. The core transaction dominant modal whitelist and the corresponding security redundancy threshold are queried in the local configuration file. If the comparison finds that the source node identifier does not match the whitelist (for example, in the audiovisual dual-modal verification, the system finds that the source of the critical causal path is actually a marginal modal with pure background environmental noise), and the direct causal effect feature value corresponding to the source node exceeds the security redundancy threshold (i.e., the marginal modal not only exceeds the authority, but also has an absolutely overwhelming driving force on the final decision), it is determined that the decision causal logic of the current agent network has been broken by the local adversarial disturbance, thereby generating a discrete Boolean signal in an active state as the causal abnormal path detection result. The whole process only relies on extremely simple array traversal and Boolean logic operation, which improves the response speed of the end-side defense.

[0074] In one feasible implementation, the abnormal path identification based on the key causal path described in S306 to obtain the causal abnormal path detection result can be performed as follows: Step B2: Perform a topological comparison between the causal graph including the key causal path and a preset normal causal graph to detect the causal intensity change characteristics on the key causal path, and detect whether there are any unexpected causal paths belonging to the modal channel excessively influencing decision-making based on the causal intensity change characteristics. A normal causal graph refers to a causal graph template with a standard topological connection structure and a normal baseline effect weight range, which is extracted and solidified after pre-simulating the proxy fusion network using a large number of benign multimodal samples in an offline security sandbox.

[0075] Causal intensity variation characteristics refer to the quantitative indicators (such as residual norm) that represent the degree of distortion of driving force obtained by mathematically subtracting or calculating the baseline weight of the corresponding edge or path in the normal causal graph from the connectivity weight of the key causal path extracted in real time.

[0076] Step B4: Perform causal consistency verification on the security decision result and the key causal path for each information aggregation node to verify whether the topological mapping relationship between the security decision result and the key causal path conforms to the preset causal logic, and detect whether there are false causal relationships in the key causal path caused by adversarial attacks that cause causal connectivity anomalies and lead to abnormal prediction decision states of the information aggregation node, thereby obtaining the causal anomaly path detection result.

[0077] Information convergence nodes refer to hub nodes in causal graph topologies that receive feature inputs from two or more upstream modal branches, perform cross-modal feature concatenation, dot product, or nonlinear activation, and output the final probability of the secure decision result.

[0078] Causal consistency verification refers to the graph logic deduction process that checks whether the distribution of input conditions and output states at the information convergence node violates the preset objective physical common sense (for example, in multimodal fusion verification, the input of a single mode should not directly determine the final safe passage state on its own).

[0079] Indicatively, after identifying the key causal path, step B2 is executed for quantitative screening. A preset baseline adjacency weight matrix corresponding to a normal causal graph is loaded into local memory, and the current adjacency weight matrix of the real-time causal graph including the key causal path is extracted. A matrix subtraction operation is performed on the corresponding elements of these two matrices to extract the residual matrix. The absolute value of the weight difference corresponding to the key causal path in this residual matrix is ​​extracted as the causal intensity change feature. This causal intensity change feature is then compared with a preset tolerance threshold. If a sharp increase in the causal effect weight on the path is found (e.g., exceeding the sharp increase weight threshold), an unexpected causal path belonging to a specific modality channel that has exceeded its weight and excessively influenced the decision-making type is accurately detected at the graph computation level. Step B4 is then executed for qualitative logical screening, using a graph traversal algorithm to locate the end of the key causal path as information. The security decision result node of the aggregation node and each cross-modal fusion feature node are used, and the conditional probability distribution table at the node is extracted to perform causal consistency verification. That is, reverse deduction is performed to see if the predicted decision state of the information aggregation node will undergo a subversive reversal after the input contribution of the key causal path is removed (that is, the type of predicted decision state is different). If the verification finds that the security decision result node is an auxiliary role of a single abnormal modality branch in the topological mapping (that is, it violates the preset causal logic of multimodal joint decision-making), it is confirmed at the underlying physical logic that there is a causal connectivity anomaly caused by noise injected by covert adversarial attacks in the key causal path, and this anomaly successfully forges a false causal relationship that causes the predicted decision state of the information aggregation node to reverse abnormally. At this time, a qualitative causal anomaly path detection result is generated and output with this rigorous graph computation and matrix reasoning logic.

[0080] In this specification, steps B2 and B4 are used to construct a two-layer anomaly detection defense line that combines quantitative verification with qualitative verification. This transforms the black-boxed, attacked state of the large model into a white-boxed, traceable topological evidence, improving the system's robustness against complex and covert input attacks and providing a basis for final security interception.

[0081] Optional, please see Figure 5 , Figure 5 This is a flowchart illustrating a counterfactual reasoning verification process proposed in this specification. Specifically, to obtain attack detection results by performing counterfactual reasoning verification based on the multimodal input features targeting the attack type, the following method can be used: S402: In the latent space, noise is applied to each target mode to be tested in the multimodal input features to generate multiple counterfactual sample features corresponding to the target mode to be tested; Counterfactual sample features refer to a series of virtual high-dimensional representations that evolve by injecting minute perturbations or noise into a specific distribution, such that "if reality undergoes a slight shift, the features will change".

[0082] S404: Perform security decision consistency detection on the multimodal input features and the counterfactual sample features to obtain decision consistency score and decision inconsistency features; The decision consistency score refers to the similarity or alignment between the safe probability distributions output by the local agent model when processing original features and counterfactual features. It is usually quantified using KL divergence and Euclidean distance.

[0083] The decision inconsistency feature refers to the specific direction vectors and gradient anomaly distribution data of the deep activation state of the network that produce violent oscillations after noise injection, which is used to characterize the direction of the model decision surface in local regions.

[0084] S406: Based on the decision consistency score and decision inconsistency features, calculate the decision difference information between the counterfactual sample features and the multimodal input features, and perform attack type identification and attack strength assessment based on the decision difference information to obtain attack detection results.

[0085] Decision difference information refers to a structured set of information that integrates the above scores and features to describe the magnitude of changes in the model output in multiple dimensions. It serves as a reference for subsequent calculation of topological slope and identification of attack types.

[0086] Indicatively, S402 locks a feature extraction layer of a specific depth within the latent space of the local agent fusion network, and injects Gaussian noise or adversarial perturbation operators of different scales into the deep embedding vector of each target modality in the multimodal input features along the normal direction of the feature manifold distribution, thereby generating multiple counterfactual sample feature tensors corresponding to the target modality. In step S404, the multimodal input features and the generated counterfactual sample features are respectively input into the subsequent classification network of the local agent fusion network. The expected values ​​of the pre-simulated security decision probabilities output by the two are captured and compared. The system calculates the KL divergence between the two probability distributions and takes its reciprocal or smooths it through a negative exponential curve to obtain the decision consistency score. At the same time, the network node indexes that have undergone drastic activation flips and their gradient differences are extracted in the feature space and packaged into decision inconsistency features. S406, based on the decision consistency score and decision inconsistency features, extracts the absolute difference in the output prediction probability and combines it with the magnitude of the noise scalar applied to the input to calculate the decision difference information between the counterfactual sample features and the multimodal input features. During attack strength assessment, it verifies whether the magnitude of the change in this decision difference information disrupts continuity; that is, it verifies whether, under extremely small latent space perturbations, the change in the predicted output of the local agent fusion network exceeds the bounded smooth linear scaling constraint. If, in response to this small change magnitude, the model's predicted probability causes a nonlinear cliff-like reversal (e.g., exceeding the prediction probability reversal threshold), then... If the security confidence level instantly reverses across boundaries, the topological slope of the nonlinear reversal (i.e., the difference in output probability divided by the gradient change rate of the input perturbation) is extracted, and the topological slope is quantified as an attack strength scalar. When performing attack type identification, based on the feedback response difference characteristics of the above counterfactual sample feature group in different modal channels (e.g., perturbing mode A causes the feature collapse of mode B, and a very small perturbation directly dominates the global decision), the target attack event is accurately classified into transfer attack, hidden attack, dominant attack, and non-input attack. The calculated attack strength scalar and the classified attack type label are structured and encapsulated, and the output is the attack detection result.

[0087] In this specification, S402-S406 actively probe the model decision boundary through latent space counterfactual noise injection and decision consistency verification. This mechanism innovatively utilizes continuity and topological slope to transform black-box vulnerabilities into precise quantitative measurements. It can accurately classify attack types and quantify attack intensity without relying on known attack characteristics, thereby improving the system's sensitivity to detecting unknown and covert adversarial attacks and its physical attribution capability.

[0088] Optionally, the attack type identification and attack strength assessment based on the decision difference information described in S406 can be performed to obtain the attack detection results, and the following methods can be used as a reference: Step C2: Based on the decision difference information, conduct adversarial attack evaluation to obtain the success rate and severity of the adversarial attack; The success rate refers to the percentage of samples that, among multiple counterfactual sample features generated for the target modality under test, successfully cause the security decision output by the local agent network to undergo a category flip (or the confidence level to fall below the security baseline). It is used to measure the breadth of the attack domain coverage.

[0089] The degree of harm refers to the severity to which counterfactual samples cause the model's decision to collapse. It can be understood as the magnitude of the change in the predicted activation value when the decision boundary is breached, that is, the peak or mean of the "topological slope" produced when the (Lipschitz) continuity is broken.

[0090] Step C4: Quantify the success rate and severity into an attack strength scalar; Attack strength scalar refers to a standardized continuous floating-point value that represents the current threat level of a system by weighting and fusing multi-dimensional attack evaluation metrics (breadth success rate and depth harm level).

[0091] Step C6: Based on the feedback response difference information of the counterfactual sample features in different modal channels, generate an attack type label for the current input attack detection event; Attack type labeling refers to the structured classification of unknown attack methods (such as transfer attack, concealment attack, dominant attack, non-input attack, etc.) based on the differences in nonlinear transmission and response caused by counterfactual perturbations in different modal channels.

[0092] Step C8: Obtain the attack detection result based on the attack strength scalar and the attack type label.

[0093] Indicatively, step C2 extracts the decision difference information output by the previous step, counts the frequency of samples in the batch counterfactual sample features that cause the pre-simulated security decision to undergo a nonlinear cliff-like reversal, calculates the success rate of the adversarial attack based on this frequency ratio, extracts the maximum gradient change rate (i.e., topological slope) between the output prediction difference and the input perturbation in the decision difference information, and uses it as an indicator to measure the depth of the destruction of Lipshitz continuity, thereby obtaining the degree of harm of the adversarial attack. Step C4 calls a preset non-linear activation function (such as the preset Sigmoid or Softmax function), takes the extracted success rate (representing the prevalence of the attack) and the degree of harm (representing the lethality of the attack) as input features, performs dynamic weighted fusion and interval mapping, and precisely quantizes them into a continuous floating-point number in the interval [0, 1], which is the attack strength scalar. Step C6 extracts the feedback response difference information between different modal channels triggered during the flow of the counterfactual sample features and performs mode matching. Specifically, if it is detected that injecting a perturbation into modality A causes the feature distribution of modality B to collapse, a transfer attack label is generated; if it is detected that the feature response of the target modality is deliberately suppressed after injecting a perturbation to cover up malicious signals from other modalities, a hidden attack label is generated; if it is detected that the response of a single perturbation modality completely overwhelms the collaborative decision-making of the other healthy modalities, a dominant attack label is generated; if it is detected that the feedback response difference information is of the normal difference type, a non-input attack label is generated. Step C8 encapsulates the attack strength scalar and attack type label using a structured protocol to obtain the final attack detection result.

[0094] Optional, please see Figure 6 , Figure 6This is a flowchart illustrating an input security decision-making process proposed in this specification. Specifically, it involves performing input security fusion evaluation on the multimodal input features based on initial modal trust features, causal anomaly path detection results, attack detection results, and security weight vectors to obtain the input security decision result. The following method can be used as a reference: S502: Perform multimodal secure fusion on the multimodal input features based on the security weight vector to obtain the target multimodal input features; The target multimodal input feature refers to the pure fused feature tensor generated after weighting and fusing the original modal feature vectors in the latent space using a security weight vector (such as element-wise dot product or cascaded projection), which has cut off or significantly attenuated the malicious adversarial perturbation components at the feature level.

[0095] In a schematic example, in a physically isolated pre-security detection environment, a security weight vector is used as a gating matrix to perform channel-wise tensor dot product and spatial projection on the multimodal input features, thereby eliminating or attenuating high-frequency adversarial disturbances in the suspicious modal features and obtaining the target multimodal input features after security reconstruction. Optionally, the multimodal secure fusion based on the security weight vector described in S502 can be performed in the following manner: Step D2: Determine the fusion method based on the initial modality trust features; Step D4: If the fusion method is a weighted fusion method, then perform multimodal weighted fusion on the multimodal input features based on the security weight vector; In the weighted fusion method, no modality is discarded. Instead, the activation values ​​of each modality feature vector are scaled continuously with floating-point precision using a safe weight vector to finely suppress minor perturbations.

[0096] Step D6: If the fusion method is a selective fusion method, then perform selective fusion to exclude low-trust modality types based on the security weight vector for the multimodal input features.

[0097] In the selective fusion method, an absolute zero mask is invoked to directly block and discard severely damaged modal features in physical memory or latent space, allowing only the remaining healthy modal features to participate in the final assembly.

[0098] Schematic, step D2 is executed to determine the control route for feature assembly. The initial modal confidence features calculated for each modal channel can be extracted, and the mean and range of these initial modal confidence features on the global dimension can be calculated. If it is determined that the initial confidence of all modal channels is within the safe confidence interval and the variance is preset to a small threshold, then the fusion method is determined to be a weighted fusion method. If it is determined that the initial modal confidence feature of a certain modality is greater than the preset cliff threshold or the preset red line threshold, then a cliff drop or crossing of the preset high-risk penalty red line occurs, and the fusion method is determined to be a selective fusion method. The corresponding fusion operator branch is triggered according to the determination result. If step D4 is executed, i.e. the current determination is a weighted fusion method, then the continuous floating-point values ​​in the security weight vector are extracted, and the values ​​are used as attention multipliers to perform element-wise multiplicative scaling operations on the deep tensor matrix of the multimodal input features in the latent space. Then, smooth multimodal weighted fusion is performed through a cross-modal cross-attention network or a channel splicing layer, thereby effectively attenuating suspicious noise while maximizing the preservation of global semantics. If step D6 is executed, i.e., the current determination is selective fusion, then the circuit breaker mask (such as absolute zero value) in the security weight vector is directly extracted. This mask is directly applied to the multimodal input features marked as low-trust modality types to completely erase the data activation stream of that channel in the tensor dimension. Then, the remaining uncontaminated healthy modality feature matrix is ​​pushed into the fusion pool to perform selective fusion that excludes low-trust modality types. This rigorous dynamic diversion mechanism ensures that the finally reconstructed multimodal features have a certain semantic purity.

[0099] S504: Based on the initial modal trust features and the security weight vector, calculate the overall trust score for the target multimodal input features; The overall trust score is a quantitative indicator used to characterize the overall security and reliability of the current input batch of data, which is calculated by comprehensively considering the original feature quality and historical reputation of each single-modal channel, as well as the actual fusion weight after security reconstruction.

[0100] In a schematic manner, the initial modal trust features of each single-modal channel are extracted and the aforementioned security weight vector is used. The weighted inner product operator or cosine similarity weighting formula is called to calculate the joint projection length of the two in the high-dimensional space. Based on this length, the overall trust score is used to quantify the overall reliability of the current data. S506: Based on the overall trust score, generate a decision confidence adjustment strategy by classifying the anti-input attack strategy confidence level; and, based on the causal abnormal path detection results and the attack detection results, assess the current attack severity and generate a graded attack response strategy based on the attack severity. The decision confidence adjustment strategy refers to a sequence of control operators generated based on the overall confidence level, used to dynamically scale, penalize, or calibrate the confidence level of the large model output at the back end of the model.

[0101] Tiered attack response strategy refers to a set of routing instructions generated based on the severity of physical threats assessed by a causal and counterfactual engine. These instructions are used at the network routing or bus communication layer to determine what transactional intervention actions (such as full access, partial blocking, downgrade replacement, physical circuit breaking) should be taken on the current data flow.

[0102] Indicatively, a dual-path strategy generation engine is launched in parallel: The first-path strategy engine inputs the overall trust score into a preset nonlinear grading function, executes the anti-input attack strategy confidence grading, and if the score is in the medium-high range, it is configured to not adjust or fine-tune the final model output probability. If the score falls below the safety threshold, a dynamic penalty factor is generated to proportionally reduce the final prediction confidence of the downstream large model, thereby generating a decision confidence adjustment strategy; The second-path strategy engine parses the input causal abnormal path detection results (qualitative graph topology) and attack detection results (quantitative topology slope and attack type), comprehensively evaluates the current attack severity according to preset evaluation rules, and generates a graded attack response strategy based on the attack severity according to preset graded response rules. S508: Aggregate the decision confidence adjustment strategy with the hierarchical attack response strategy to obtain the input security decision result.

[0103] Indicatively, the dynamic penalty factor in the decision confidence adjustment strategy and the flow routing instructions in the hierarchical attack response strategy are concatenated using multidimensional tensors or aggregated using formatted messages, and the resulting output is the input security decision result.

[0104] In this specification, S502 to S508 are executed to construct a defense closed loop from feature cleanup to physical isolation. This maximizes the parsing of residual semantics of healthy modalities while accurately blocking covert attacks, taking into account both stringent security levels and the continuity of smooth transaction degradation.

[0105] Optionally, the specific implementation of S108, which controls whether the multimodal input data is transferred to the target multimodal large model based on the input security decision result, can be carried out in the following manner: Step E2: Analyze the input security decision results to determine the severity of the current input attack defense; Step E4: If the attack severity is classified as a mild attack, then control the full flow of the multimodal input data to the target multimodal large model and asynchronously output mild attack alarm information; Full transmission refers to transmitting the original messages or reconstructed features of multimodal input data to the backend main transaction model—the target multimodal large model—without performing any channel elimination or blocking, in a transparent proxy manner.

[0106] Step E6: If the attack severity is a moderate attack type, then perform a flexible degradation flow to filter out suspicious modal data in the multimodal input data, and control the flow of reference modal data other than suspicious modal data to the target multimodal large model; Flexible degradation routing refers to a high-availability routing strategy for local modal contamination. It discards, truncates, and emptys suspicious modal data payloads that are identified as high-risk, allowing only the remaining trusted modal data to pass through, thereby ensuring that the core transaction system does not crash while sacrificing some perception accuracy.

[0107] Step E8: If the attack severity is a severe attack type, then execute a hard physical circuit breaker to prevent the multimodal input data from flowing to the target multimodal large model and trigger manual review.

[0108] Hard physical circuit breakers refer to a low-level security defense mechanism that, when faced with a severe complex attack that is sufficient to breach the decision boundary, directly cuts off the data transmission link with the target large model at the physical memory or gateway protocol layer to prevent poisoned data from contaminating the real transaction large model.

[0109] Indicatively, step E2 is executed through the routing control gateway to parse the sequence of input security decision results in the input buffer, extract the security alarm flag and threat score to determine the severity of the current input attack defense; then, the corresponding control logic branch is triggered based on the dynamic routing distribution engine. If step E4 is triggered, i.e. the attack severity is determined to be a mild attack type (e.g., only slight background white noise exists), then the transparent pass-through protocol is invoked to control the full flow of the multimodal input data to the target multimodal large model to ensure zero transaction latency, and the asynchronous message queue is simultaneously woken up in the background to asynchronously output mild attack alarm information to the security monitoring center. If step E6 is triggered, i.e. the attack severity is determined to be a moderate attack type (such as targeted adversarial disturbance in a specific channel), then the payload of the multimodal data packet is parsed at the network protocol layer, and a flexible degradation flow is performed to directly discard or filter out suspicious modal data in the multimodal input data in memory, and the data packet is re-encapsulated. The reference modal data other than the suspicious modal data is controlled to flow to the target multimodal large model to drive the main model to perform degradation inference under the incomplete modality. If step E8 is triggered, i.e. the attack severity is determined to be a severe attack type (such as a complete collapse of the decision surface and the existence of false causal dominance), then the underlying network interface is directly invoked to execute a hard physical circuit breaker (such as resetting the TCP connection or issuing a firewall drop instruction) to completely prevent the multimodal input data flow from reaching the target multimodal large model. At the same time, an anti-tampering attack snapshot file is generated and a manual review workflow is triggered, thereby completing the physical-level blocking of the adversarial attack with an extremely rigorous tiered network routing strategy.

[0110] The following will combine Figure 7 This specification provides a detailed description of the anti-attack processing device provided in the embodiments. It should be noted that... Figure 7 The anti-attack processing device shown is used to execute this specification. Figures 1-6 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-6 The example shown.

[0111] Please see Figure 7 This diagram illustrates the structure of an anti-attack processing device according to an embodiment of this specification. The anti-attack processing device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the anti-attack processing device 1 includes an input monitoring module 11, an attack detection module 12, and an input decision module 13, specifically used for: The input monitoring module 11 is used to monitor the multimodal input data for the target multimodal large model input, extract the multimodal input features of the multimodal input data, and calculate the initial modal confidence features of each modal channel based on the feature distribution anomaly. The attack detection module 12 is used to perform causal reasoning verification for causal decision paths based on the multimodal input features to obtain causal abnormal path detection results, and to perform counterfactual reasoning verification for attack types based on the multimodal input features to obtain attack detection results. The attack detection module 12 is used to perform joint modulation processing on the initial modal trust feature based on the causal abnormal path detection result and the attack detection result to obtain the security weight vector of each modal channel of the multimodal input data. The input decision module 13 is used to perform input security fusion evaluation processing on the multimodal input features based on the initial modal trust features, the causal abnormal path detection results, the attack detection results, and the security weight vector to obtain the input security decision results, and to control whether the multimodal input data is transferred to the target multimodal large model based on the input security decision results.

[0112] Optionally, the step of measuring the multimodal input features based on feature distribution anomalies to calculate the initial modal confidence features of each modal channel includes: For the multimodal input data, a heterogeneous modal encoder is invoked in parallel to extract deep semantic feature representations for each modal channel; Based on the deep semantic feature representation, multimodal anomaly analysis is performed from the dimensions of feature distribution, modality internal consistency, and historical anomaly pattern matching to generate feature quality evaluation index, anomaly degree evaluation index, modality historical reputation evaluation index, and cross-modal consistency evaluation index. The initial modal trust features are obtained by normalizing the feature quality evaluation index, anomaly degree evaluation index, modal history reputation evaluation index, and cross-modal consistency evaluation index.

[0113] Optionally, the step of performing causal inference verification for causal decision paths based on the multimodal input features to obtain causal anomaly path detection results includes: Based on the multimodal input features, a causal graph is constructed between each modal feature and the security decision result. The graph nodes in the causal graph are configured as each modal feature, intermediate representation, and security decision result, and the graph edges in the causal graph are configured as intermodal causal relationships and feature-decision influence relationships. Based on the causal graph, causal effect inference analysis is performed to calculate the direct causal effect characteristics of each modal channel on the security decision result and the interactive causal effect characteristics between each modal channel; Based on the direct causal effect features and the interactive causal effect features, key causal paths are identified from the causal graph, and abnormal paths are identified based on the key causal paths to obtain causal abnormal path detection results.

[0114] Optionally, the step of identifying causal anomaly paths based on the key causal path to obtain causal anomaly path detection results includes: The causal graph including the key causal path is topologically compared with a preset normal causal graph to detect the causal intensity change characteristics on the key causal path, and based on the causal intensity change characteristics, it is detected whether there are any unexpected causal paths belonging to the modal channel excessively influencing decision-making type. A causal consistency verification is performed on the security decision result and the key causal path for each information aggregation node to verify whether the topological mapping relationship between the security decision result and the key causal path conforms to the preset causal logic, and to detect whether there are false causal relationships in the key causal path caused by adversarial attacks that result in abnormal causal connectivity and cause the prediction decision state of the information aggregation node to be abnormal, thereby obtaining the causal abnormal path detection result.

[0115] Optionally, the step of obtaining the attack detection result by performing counterfactual reasoning verification based on the multimodal input features to target the attack type includes: Within the latent space, noise is applied to each target modality in the multimodal input features to generate multiple counterfactual sample features corresponding to the target modality; The multimodal input features and the counterfactual sample features are subjected to security decision consistency detection to obtain decision consistency scores and decision inconsistency features; Based on the decision consistency score and decision inconsistency features, the decision difference information between the counterfactual sample features and the multimodal input features is calculated. Based on the decision difference information, attack type identification and attack strength assessment are performed to obtain attack detection results.

[0116] Optionally, the step of performing attack type identification and attack strength assessment based on the decision difference information to obtain attack detection results includes: Based on the aforementioned decision difference information, an adversarial attack evaluation is conducted to obtain the success rate and severity of the adversarial attack. The success rate and severity are quantified into an attack intensity scalar; Based on the feedback response difference information of the counterfactual sample features across different modal channels, an attack type label is generated for the current input attack detection event; The attack detection result is obtained based on the attack strength scalar and the attack type label.

[0117] Optionally, the joint modulation processing of the initial modal trust features based on the causal anomaly path detection results and the attack detection results to obtain the security weight vector of each modal channel of the multimodal input data includes: The causal anomaly path detection results and the attack detection results are analyzed to determine the modal trust type corresponding to each modal channel; Based on the modal trust type, the weights of each modal channel in the initial modal trust feature are adjusted to obtain the security weight vector of each modal channel of the multimodal input data.

[0118] Optionally, the step of performing input security fusion evaluation processing on the multimodal input features based on the initial modal trust features, the causal anomaly path detection results, the attack detection results, and the security weight vector to obtain the input security decision results includes: Multimodal secure fusion is performed on the multimodal input features based on the security weight vector to obtain the target multimodal input features; Based on the initial modal trust features and the security weight vector, calculate the overall trust score for the target multimodal input features; Based on the overall trust score, a confidence level is generated for the anti-input attack strategy to produce a decision confidence adjustment strategy. Based on the causal abnormal path detection results and the attack detection results, the current attack severity is assessed and a graded attack response strategy is generated based on the attack severity. The decision confidence adjustment strategy and the hierarchical attack response strategy are aggregated to obtain the input security decision result.

[0119] Optionally, performing multimodal secure fusion on the multimodal input features based on the security weight vector includes: The fusion method is determined based on the initial modal trust characteristics; If the fusion method is a weighted fusion method, then multimodal weighted fusion is performed on the multimodal input features based on the security weight vector; If the fusion method is a selective fusion method, then selective fusion is performed on the multimodal input features based on the security weight vector to exclude low-trust modality types.

[0120] Optionally, controlling whether the multimodal input data is transferred to the target multimodal large model based on the input security decision result includes: The input security decision results are analyzed to determine the severity of the current input attack defense. If the attack severity is classified as a mild attack, then the multimodal input data is fully transferred to the target multimodal large model and a mild attack alarm message is output asynchronously. If the attack severity is a moderate attack type, a flexible degradation flow is performed to filter out suspicious modal data in the multimodal input data, and the reference modal data other than the suspicious modal data is controlled to flow to the target multimodal large model; If the attack severity is classified as a severe attack, a hard physical circuit breaker is executed to prevent the multimodal input data from flowing to the target multimodal large model and to trigger manual review.

[0121] It should be noted that the anti-attack processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the anti-attack processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the anti-attack processing device and the anti-attack processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0122] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0123] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-6 The specific execution process of the anti-attack processing method described in the illustrated embodiment can be found in [reference needed]. Figures 1-6 The specific details of the illustrated embodiments will not be elaborated here.

[0124] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-6 The specific execution process of the anti-attack processing method described in the illustrated embodiment can be found in [reference needed]. Figures 1-6 The specific details of the illustrated embodiments will not be elaborated here.

[0125] Please refer to Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 can be connected to each other via the bus 1050.

[0126] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or a combination of central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0127] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0128] The input device 1030 is used to receive input instructions or data, and the input device 1030 includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and the output device 1040 includes, but is not limited to, display devices and speakers.

[0129] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (Wi-Fi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0130] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0131] exist Figure 8 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the anti-attack processing method as described in the various method embodiments of this specification.

[0132] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0133] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the features, data, and information involved in this specification were all obtained under full authorization.

[0134] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A method for handling adversarial attacks, characterized in that, The method includes: Monitor the multimodal input data for the target multimodal large model input, extract the multimodal input features of the multimodal input data, and calculate the initial modal confidence features of each modal channel based on the feature distribution anomaly. Based on the multimodal input features, causal reasoning verification for causal decision paths is performed to obtain causal anomaly path detection results, and counterfactual reasoning verification for attack types is performed based on the multimodal input features to obtain attack detection results. Based on the causal anomaly path detection results and the attack detection results, the initial modal trust features are jointly modulated to obtain the security weight vector of each modal channel of the multimodal input data; Based on the initial modal trust features, the causal anomaly path detection results, the attack detection results, and the security weight vector, the multimodal input features are subjected to input security fusion evaluation processing to obtain input security decision results. Based on the input security decision results, the multimodal input data is controlled to flow to the target multimodal large model.

2. The method according to claim 1, characterized in that, The step of measuring the multimodal input features based on feature distribution anomalies to calculate the initial modal confidence features of each modal channel includes: For the multimodal input data, a heterogeneous modal encoder is invoked in parallel to extract deep semantic feature representations for each modal channel; Based on the deep semantic feature representation, multimodal anomaly analysis is performed from the dimensions of feature distribution, modality internal consistency, and historical anomaly pattern matching to generate feature quality evaluation index, anomaly degree evaluation index, modality historical reputation evaluation index, and cross-modal consistency evaluation index. The initial modal trust features are obtained by normalizing the feature quality evaluation index, anomaly degree evaluation index, modal history reputation evaluation index, and cross-modal consistency evaluation index.

3. The method according to claim 1, characterized in that, The causal inference verification based on the multimodal input features to obtain the causal abnormal path detection result includes: Based on the multimodal input features, a causal graph is constructed between each modal feature and the security decision result. The graph nodes in the causal graph are configured as each modal feature, intermediate representation, and security decision result, and the graph edges in the causal graph are configured as intermodal causal relationships and feature-decision influence relationships. Based on the causal graph, causal effect inference analysis is performed to calculate the direct causal effect characteristics of each modal channel on the security decision result and the interactive causal effect characteristics between each modal channel; Based on the direct causal effect features and the interactive causal effect features, key causal paths are identified from the causal graph, and abnormal paths are identified based on the key causal paths to obtain causal abnormal path detection results.

4. The method according to claim 3, characterized in that, The process of identifying causal anomaly paths based on the key causal paths to obtain causal anomaly path detection results includes: The causal graph including the key causal path is topologically compared with a preset normal causal graph to detect the causal intensity change characteristics on the key causal path, and based on the causal intensity change characteristics, it is detected whether there are any unexpected causal paths belonging to the modal channel excessively influencing decision-making type. A causal consistency verification is performed on the security decision result and the key causal path for each information aggregation node to verify whether the topological mapping relationship between the security decision result and the key causal path conforms to the preset causal logic, and to detect whether there are false causal relationships in the key causal path caused by adversarial attacks that result in abnormal causal connectivity and cause the prediction decision state of the information aggregation node to be abnormal, thereby obtaining the causal abnormal path detection result.

5. The method according to claim 1, characterized in that, The attack detection result is obtained by performing counterfactual reasoning verification based on the multimodal input features to target the attack type, including: Within the latent space, noise is applied to each target modality in the multimodal input features to generate multiple counterfactual sample features corresponding to the target modality; The multimodal input features and the counterfactual sample features are subjected to security decision consistency detection to obtain decision consistency scores and decision inconsistency features; Based on the decision consistency score and decision inconsistency features, the decision difference information between the counterfactual sample features and the multimodal input features is calculated. Based on the decision difference information, attack type identification and attack strength assessment are performed to obtain attack detection results.

6. The method according to claim 5, characterized in that, The attack type identification and attack strength assessment based on the decision difference information, to obtain the attack detection result, includes: Based on the aforementioned decision difference information, an adversarial attack evaluation is conducted to obtain the success rate and severity of the adversarial attack. The success rate and severity are quantified into an attack intensity scalar; Based on the feedback response difference information of the counterfactual sample features across different modal channels, an attack type label is generated for the current input attack detection event; The attack detection result is obtained based on the attack strength scalar and the attack type label.

7. The method according to claim 1, characterized in that, The initial modal trust feature is jointly modulated based on the causal anomaly path detection result and the attack detection result to obtain the security weight vector of each modal channel of the multimodal input data, including: The causal anomaly path detection results and the attack detection results are analyzed to determine the modal trust type corresponding to each modal channel; Based on the modal trust type, the weights of each modal channel in the initial modal trust feature are adjusted to obtain the security weight vector of each modal channel of the multimodal input data.

8. The method according to claim 1, characterized in that, The input security decision result is obtained by performing input security fusion evaluation on the multimodal input features based on the initial modal trust features, the causal abnormal path detection results, the attack detection results, and the security weight vector, including: Multimodal secure fusion is performed on the multimodal input features based on the security weight vector to obtain the target multimodal input features; Based on the initial modal trust features and the security weight vector, calculate the overall trust score for the target multimodal input features; Based on the overall trust score, a confidence level is generated for the anti-input attack strategy to produce a decision confidence adjustment strategy. Based on the causal abnormal path detection results and the attack detection results, the current attack severity is assessed and a graded attack response strategy is generated based on the attack severity. The decision confidence adjustment strategy and the hierarchical attack response strategy are aggregated to obtain the input security decision result.

9. The method according to claim 8, characterized in that, The step of performing multimodal secure fusion on the multimodal input features based on the security weight vector includes: The fusion method is determined based on the initial modal trust characteristics; If the fusion method is a weighted fusion method, then multimodal weighted fusion is performed on the multimodal input features based on the security weight vector; If the fusion method is a selective fusion method, then selective fusion is performed on the multimodal input features based on the security weight vector to exclude low-trust modality types.

10. The method according to claim 1 or 8, characterized in that, The step of controlling whether the multimodal input data is transferred to the target multimodal large model based on the input security decision result includes: The input security decision results are analyzed to determine the severity of the current input attack defense. If the attack severity is classified as a mild attack, then the multimodal input data is fully transferred to the target multimodal large model and a mild attack alarm message is output asynchronously. If the attack severity is a moderate attack type, a flexible degradation flow is performed to filter out suspicious modal data in the multimodal input data, and the reference modal data other than the suspicious modal data is controlled to flow to the target multimodal large model; If the attack severity is classified as a severe attack, a hard physical circuit breaker is executed to prevent the multimodal input data from flowing to the target multimodal large model and to trigger manual review.

11. An anti-attack processing device, characterized in that, The device includes: The input monitoring module is used to monitor multimodal input data for the target multimodal large model, extract multimodal input features from the multimodal input data, and calculate the initial modal confidence features of each modal channel based on feature distribution anomalies. The attack detection module is used to perform causal reasoning verification for causal decision paths based on the multimodal input features to obtain causal abnormal path detection results, and to perform counterfactual reasoning verification for attack types based on the multimodal input features to obtain attack detection results. The attack detection module is used to perform joint modulation processing on the initial modal trust feature based on the causal abnormal path detection result and the attack detection result to obtain the security weight vector of each modal channel of the multimodal input data. The input decision module is used to perform input security fusion evaluation processing on the multimodal input features based on the initial modal trust features, the causal abnormal path detection results, the attack detection results, and the security weight vector to obtain the input security decision results, and to control whether the multimodal input data is transferred to the target multimodal large model based on the input security decision results.

12. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded by a processor and executed according to any one of claims 1 to 10.

14. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 10.

Images