Multi-mode poisoning attack defense method and system

Abstract

The embodiment of the invention provides a multi-mode poisoning attack defense method and system, and the method comprises the steps: obtaining a preprocessing data set; determining fusion features corresponding to the preprocessed data set; performing feature reconstruction on the fused features to determine reconstructed features; performing quality evaluation on the reconstructed features, and determining the confidence coefficient of the fusion features; acquiring a historical attack sample; inputting the historical attack sample and the preprocessed data set into a generative adversarial network, and determining attack mode matching confidence; determining a security confidence coefficient parameter according to the fusion feature confidence coefficient and the attack mode matching confidence coefficient; and triggering a defense action according to the security confidence parameter. According to the scheme of the invention, the safety confidence is determined according to the fusion feature confidence and the attack pattern matching confidence, so that the balance of comprehensive defense and dynamic adaptability is realized, and the safety of a banking business system is improved.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a multimodal poisoning attack defense method and system. Background Art

[0002] At present, with the widespread application of artificial intelligence, various industries have connected artificial intelligence to customer service systems to improve customer service efficiency. However, attackers' poisoning attack data may exist in data of multiple modalities such as voice, text, images, and user behavior. Existing technologies that use artificial intelligence to defend against poisoning attacks can often only target single-modal data sets. In actual work or usage scenarios, the usage data of each participant contains multiple modalities, and covert attacks are constructed through the correlation between multimodal data to implement voice poisoning, text and image poisoning, and behavior poisoning. Existing single-modal defense methods are difficult to deal with hidden instructions and cross-modal attack chains in multimodal data. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a multimodal poisoning attack defense method and system, which can solve the problem that single-modal defense methods are difficult to deal with hidden instructions and cross-modal attack chains in multimodal data.

[0004] In view of this, an embodiment of the first aspect of the present invention provides a multi-modal poisoning attack defense method.

[0005] An embodiment of the second aspect of the present invention provides a multi-modal poisoning attack defense system.

[0006] In order to achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention provides a multimodal poisoning attack defense method, including: obtaining a preprocessed data set; determining a fusion feature corresponding to the preprocessed data set; performing feature reconstruction on the fusion feature to determine the reconstructed feature; performing quality assessment on the reconstructed feature to determine the fusion feature confidence; obtaining historical attack samples; inputting the historical attack samples and the preprocessed data set into a generative adversarial network to determine the attack pattern matching confidence; determining a security confidence parameter based on the fusion feature confidence and the attack pattern matching confidence; and triggering a defense action based on the security confidence parameter.

[0007] According to the multimodal poisoning attack defense method provided by the present invention, feature fusion is performed on the preprocessed data set including multimodal data, and feature reconstruction is performed on the fused features generated after the fusion, and the reconstructed features corresponding to the preprocessed data set are determined. Combined with the federated knowledge collaboration and dynamic response mechanism, a full-process defense system from multimodal data preprocessing to attack interception and triggering of defense actions is constructed. The multimodal poisoning attack defense method also combines a generative adversarial network, wherein the generator in the generative adversarial network outputs poisoning attack samples, and the discriminator in the generative adversarial network processes multimodal features through a dual-branch network, and fuses the two branch information of the user branch and the attack branch through a cross-branch attention mechanism to determine the cross-modal attention parameters of the user features and the attack sample features. The attack pattern matching confidence is determined based on the cross-modal attention parameters. The smaller the value corresponding to the attack pattern matching confidence, the more similar the user data features and the poisoning attack sample are. Finally, the security confidence parameter is determined based on the attack pattern matching confidence and the fusion feature confidence to trigger the corresponding defense action of the system.

[0008] In the above technical solution, obtaining the preprocessed data set includes: collecting the user's original data set, which includes voice data, image data, text data and behavior data; determining a denoised data set corresponding to the user's original data set; determining multiple modal features in the denoised data set, which include voice modality corresponding to voice data, image modality corresponding to image data, text modality corresponding to text data and behavior modality corresponding to behavior data; mapping the voice modality, image modality, text modality and behavior modality to the same latitude, and performing time alignment to determine the preprocessed data set.

[0009] In this solution, the user's original data set in the business scenario is collected. The original data set includes the user's voice data, image data, text data, and behavior data. Multiple data in the original data set correspond to the same business process. By denoising the user's original data set, abnormal signals or privacy information in multiple data are removed, and a denoised data set is generated, thereby improving data quality, reducing the interference of noise on subsequent feature extraction and model training, and improving operational efficiency. Multiple modal features in the denoised data set are determined, retaining the key information required for attack detection, and multiple modalities are mapped to the same dimension and time-series aligned to eliminate dimensional differences. This allows different modal features to be fused and compared in the same spatial dimension, improving the accuracy and real-time performance of attack detection.

[0010] In any of the above technical solutions, optionally, determining a denoised data set corresponding to the user's original data set includes: determining sensitive features in the user's original data set and marking multiple sensitive features; stripping off the marked sensitive features in the voice data, image data, text data and behavior data respectively to determine the denoised data set.

[0011] In this solution, a denoised dataset is determined before extracting features from different modalities. Many data within the denoised dataset have already been stripped of sensitive features. Because poisoning attacks can target sensitive user information, sensitive features are stripped from the user's original dataset during the preprocessing phase, eliminating tampering opportunities for attackers. This protects user privacy while ensuring the effectiveness of the defense system. By retaining high-level features after desensitization, data accuracy is ensured during subsequent multimodal fusion and attack detection.

[0012] In any of the above technical solutions, optionally, determining the fusion features corresponding to the preprocessed data set includes: obtaining a learning parameter matrix; determining a feature matrix corresponding to the preprocessed data set based on the learning parameter matrix; and performing a multi-head attention mechanism calculation on the feature matrix to determine the fusion features.

[0013] In this scheme, the dependencies between multiple modalities are captured through a multi-head attention mechanism, and the features of each modality are converted into a query matrix, a key matrix, and a value matrix. The cross-modal attention weights are calculated, and the residual connection and layer normalization are combined to fuse the features of multiple modalities and determine the fused features.

[0014] In any of the above technical solutions, optionally, a quality assessment is performed on the reconstructed features to determine the confidence of the fused features, including: performing a reconstruction error analysis on the reconstructed features to determine the reconstruction error analysis parameters; determining the reconstruction quality parameters based on the reconstruction error analysis parameters; determining the mutual information corresponding to the modal features and the fused features; determining the modal consistency parameters based on the mutual information; and determining the confidence of the fused features based on the modal consistency parameters and the reconstruction quality parameters.

[0015] In this solution, the generator in a generative adversarial network is used to reconstruct the fused features. The reconstruction error analysis parameter between the fused and reconstructed features is calculated and converted into a quality score, namely the reconstruction quality parameter, through a functional model. By determining the reconstruction error analysis parameter and the reconstruction quality parameter, anomaly detection is performed. Normal data, due to real user operations and conforming to a natural distribution, has a small reconstruction error. However, attack data, due to its secondary generation, has a reconstruction error greater than the average. Using the reconstruction error analysis parameter and the reconstruction quality parameter, synthetic or tampered parameters can be identified, improving the robustness of user anomaly detection.

[0016] In any of the above technical solutions, optionally, determining the attack pattern matching confidence includes: receiving historical attack samples through a generator in a generative adversarial network; generating poisoned attack samples based on the historical attack samples; processing modal features through a discriminator in the generative adversarial network using a dual-branch network architecture, the dual-branch network architecture including a first branch corresponding to a preprocessed data set and a second branch corresponding to the poisoned attack sample; determining a user feature query matrix based on the first branch; determining a poisoned attack sample feature key-value matrix and a poisoned attack sample eigenvalue matrix based on the second branch; determining a cross-modal attention parameter based on the user feature query matrix, the poisoned attack sample feature key-value matrix and the poisoned attack sample eigenvalue matrix; and determining the attack pattern matching confidence based on the cross-modal attention parameter.

[0017] In this scheme, by combining generative adversarial networks and cross-modal attention mechanisms, multimodal poisoning attacks are detected to determine the matching confidence between the user's original dataset and the poisoning attack samples, namely the attack pattern matching confidence.

[0018] In any of the above technical solutions, optionally, triggering a defense action according to a security confidence parameter includes: obtaining a defense response strategy;

[0019] Determine the risk level based on security confidence parameters; determine the defense action based on the risk level and defense response strategy.

[0020] This solution dynamically responds to the user's input raw data set based on a defense response strategy. The security confidence parameter determines the corresponding interval, and the confidence interval determines the risk level of the user's raw data set. Defense actions are then taken based on the risk level and the parameters preset in the defense response strategy. The defense response strategy includes the risk level, confidence interval, action triggering conditions, and defense actions. The security confidence value is used to determine the risk level of the user's raw data, and defense actions are then taken based on the risk level.

[0021] In any of the above technical solutions, optionally, the multimodal poisoning attack defense method also includes: determining a six-stage dialogue process based on a hidden Markov model; determining an abnormal state during the processing of the preprocessed data set according to the six-stage dialogue process; and triggering a defense action when an abnormal state is detected.

[0022] In this solution, a Deep Structured Semantic Model (DSSM) uses a hidden Markov model (HMM) to construct a six-stage conversation process. The six-stage conversation process includes identity authentication, service processing, confirmation, execution, and termination. The DSSM calculates the probability of abnormal state transitions in real time. When an illegal transition is detected, a circuit breaker mechanism is triggered and the attack path is traced back.

[0023] The second aspect of an embodiment of the present invention proposes a multimodal poisoning attack defense system, which includes: a dynamic response module for obtaining a preprocessed data set; a feature fusion module for determining a fusion feature corresponding to the preprocessed data set; reconstructing the fusion feature to determine the reconstructed feature; performing quality assessment on the reconstructed feature to determine the confidence of the fusion feature; a federal defense module for obtaining historical attack samples; inputting the historical attack samples and the preprocessed data set into a generative adversarial network to determine the attack pattern matching confidence; determining a security confidence parameter based on the fusion feature confidence and the attack pattern matching confidence; and a security defense module for triggering a defense action based on the security confidence parameter.

[0024] According to the multimodal poisoning attack defense system provided by the present invention, a multimodal poisoning attack defense method is implemented, multimodal features in user data are fused and reconstructed, and the confidence of the fused features is determined; multimodal poisoning attack samples and user multimodal features are analyzed in combination with a generative adversarial network including a dual-branch network architecture to determine the attack pattern matching confidence; finally, the security confidence is determined based on the fused feature confidence and the attack pattern matching confidence, thereby achieving a balance between comprehensive defense and dynamic adaptability and improving the accuracy and robustness of multimodal data poisoning defense.

[0025] In any of the above technical solutions, optionally, the multimodal poisoning attack defense system also includes: a perception defense module, used to determine a six-stage dialogue process based on a hidden Markov model; determine an abnormal state during the processing of the preprocessed data set according to the six-stage dialogue process; and trigger the defense action when the abnormal state is detected.

[0026] Additional aspects and advantages of the technical solutions of the present invention will become apparent in the following description or will be understood through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a multi-modal poisoning attack defense method according to an embodiment of the present application is shown;

[0028] Figure 2A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0029] Figure 3 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0030] Figure 4 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0031] Figure 5 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0032] Figure 6 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0033] Figure 7 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0034] Figure 8 A partial flow chart of a multimodal poisoning attack defense method according to an embodiment of the present application is shown;

[0035] Figure 9 A schematic structural block diagram of a multimodal poisoning attack defense system according to an embodiment of the present application is shown.

[0036] in, Figure 9 The corresponding relationship between the reference numerals and component names is as follows:

[0037] 900: Multimodal poisoning attack defense system; 902: Dynamic response module; 904: Feature fusion module; 906: Federation defense module; 908: Security defense module; 910: Perception defense module. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the embodiments of the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0040] The following is combined with Figures 1 to 9 , the multimodal poisoning attack defense method and system provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0041] This embodiment provides a multi-modal poisoning attack defense method, such as Figure 1 As shown, the multimodal poisoning attack defense method includes:

[0042] Step S100: obtaining a preprocessed data set;

[0043] Step S102: determining fusion features corresponding to the preprocessed data set;

[0044] Step S104: reconstructing the fused features to determine the reconstructed features;

[0045] Step S106: performing quality assessment on the reconstructed features and determining the confidence of the fused features;

[0046] Step S108: Obtain historical attack samples;

[0047] Step S110: inputting historical attack samples and pre-processed datasets into the generative adversarial network to determine attack pattern matching confidence;

[0048] Step S112: Determine a security confidence parameter based on the fusion feature confidence and the attack pattern matching confidence;

[0049] Step S114: triggering a defense action according to the security confidence parameter.

[0050] According to the multimodal poisoning attack defense method provided by the present invention, feature fusion is performed on the preprocessed data set including multimodal data, and feature reconstruction is performed on the fused features generated after the fusion, and the reconstructed features corresponding to the preprocessed data set are determined. Combined with the federated knowledge collaboration and dynamic response mechanism, a full-process defense system from multimodal data preprocessing to attack interception and triggering of defense actions is constructed. The multimodal poisoning attack defense method also combines a generative adversarial network, wherein the generator in the generative adversarial network outputs poisoning attack samples, and the discriminator in the generative adversarial network processes multimodal features through a dual-branch network, and fuses the two branch information of the user branch and the attack branch through a cross-branch attention mechanism to determine the cross-modal attention parameters of the user features and the attack sample features. The attack pattern matching confidence is determined based on the cross-modal attention parameters. The smaller the value corresponding to the attack pattern matching confidence, the more similar the user data features and the poisoning attack sample are. Finally, the security confidence parameter is determined based on the attack pattern matching confidence and the fusion feature confidence to trigger the corresponding defense action of the system.

[0051] It can be understood that the multimodal poisoning attack defense method provided by the present invention fuses and reconstructs multimodal features in user data to determine the confidence of the fused features; combines a generative adversarial network including a dual-branch network architecture to analyze multimodal poisoning attack samples and user multimodal features to determine the attack pattern matching confidence; and finally determines the security confidence based on the fused feature confidence and the attack pattern matching confidence. This method achieves a balance between comprehensive defense and dynamic adaptability, improving the accuracy and robustness of multimodal data poisoning defense. The method of judging attack patterns based solely on the similarity between single-modal features and historical attack samples has limitations and cannot detect cross-modal data. The present invention addresses this problem by fusing and reconstructing the user's multimodal data, performing consistency analysis and quality assessment on the reconstructed features, determining the fused feature confidence, and finally determining the final security confidence based on the fused feature confidence and attack pattern matching confidence. Through multiple analyses of user multimodal data, the problem of attack pattern matching failing to cover unknown attacks and complex forgery scenarios is solved, improving the system's generalization and robustness.

[0052] Moreover, since security confidence is obtained by combining two confidence levels, it takes into account both the rapid interception of known attacks and the active discovery of unknown threats. At the same time, it adapts to different business scenarios through dynamic weights to improve the security of multimodal poisoning attack defense.

[0053] Furthermore, since the fused feature confidence is obtained by fusing and reconstructing multimodal features, it can adapt to a variety of usage scenarios, improving the generalization ability and applicability of the multimodal poisoning attack defense method.

[0054] Specifically, during the data collection phase, the system collects and preprocesses users' raw multimodal data through a data sandbox mechanism to improve robustness during data processing and protect users' information security and privacy. During the feature fusion process of the preprocessed dataset, the system introduces a multi-head attention mechanism to transform the user's modal features into query, key, and value matrices, dynamically calculate cross-modal association weights, and generate fused features through residual connections and layer normalization. This improves the complementarity and consistency of multimodal data and suppresses noise interference.

[0055] For example, the raw data corresponding to the preprocessed dataset in the multimodal poisoning attack defense method comes from multiple data interfaces, including bank ATMs, teller counters, smart terminals, mobile banking software, and online banking application programming interfaces (APIs). For example, the raw data includes user operation video streams, touch screen interaction behaviors, and user voice commands at the bank ATM port; and document scan information, user facial feature data, and user signature data during the teller terminal user identity authentication process.

[0056] Specifically, the formula used to determine the security confidence parameter based on the fusion feature confidence and the attack pattern matching confidence is as follows:

[0057] C=μ×C fusion +(1-μ)×C pattern ;

[0058] Among them, C is the safety confidence parameter, μ is the first balance parameter, C fusion is the confidence of the fusion feature, C pattern Confidence of attack pattern matching.

[0059] Optionally, the default value of the first balance parameter is 0.6. The system is set with a risk threshold and detects the risk parameter in real time through a multimodal poisoning attack defense system. When the risk parameter is higher than the risk threshold, the business scenario is determined to be a high-risk scenario, and the first balance parameter is automatically adjusted to 0.4. Under this balance parameter setting, it tends to match attack patterns and is used to find poisoned data in the business scenario; when the risk parameter is lower than the risk threshold, the business scenario is determined to be a low-risk scenario, and the first balance parameter is automatically adjusted to 0.7. Under this balance parameter setting, it tends to fuse feature analysis and is used to protect user data security.

[0060] In some embodiments, optionally, as Figure 2 As shown, step S100: obtaining a pre-processed data set, including:

[0061] Step S1002: Collecting user original data sets, which include voice data, image data, text data, and behavior data;

[0062] Step S1004: determining a denoised data set corresponding to the user's original data set;

[0063] Step S1006: determining multiple modal features in the denoised dataset, where the modal features include a speech modality corresponding to the speech data, an image modality corresponding to the image data, a text modality corresponding to the text data, and a behavioral modality corresponding to the behavioral data;

[0064] Step S1008: Map the speech modality, image modality, text modality, and behavioral modality to the same dimension, perform time alignment, and determine the preprocessed data set.

[0065] In this embodiment, the user's original data set in the business scenario is collected. The original data set includes the user's voice data, image data, text data, and behavior data. Multiple data in the original data set correspond to the same business process. By denoising the user's original data set, abnormal signals or privacy information in multiple data are removed, and a denoised data set is generated, thereby improving data quality, reducing the interference of noise on subsequent feature extraction and model training, and improving operational efficiency. Multiple modal features in the denoised data set are determined, retaining the key information required for attack detection, and multiple modalities are mapped to the same dimension and time-series aligned to eliminate dimensional differences. This allows different modal features to be fused and compared in the same spatial dimension, improving the accuracy and real-time performance of attack detection.

[0066] Specifically, after obtaining the denoised dataset, various data in the denoised dataset need to be extracted using the corresponding neural network architecture.

[0067] The speech modality extraction method includes: determining the 13-dimensional Mel-Frequency Cepstral Coefficients (MFCC) of the speech data in the denoising dataset to characterize the vocal tract characteristics of the speech; extracting the 128-dimensional voiceprint vector from the MFCC using the convolutional neural network model ResNet-50, and using the 128-dimensional voiceprint vector as the speech modality;

[0068] The image modality extraction method includes: extracting 2048-dimensional convolutional layer features of image data through a residual network (ResNet); calculating the liveness probability in the 2048-dimensional convolutional layer features through three-dimensional (3D) structured light analysis; and determining the convolutional layer features whose liveness probability exceeds a corresponding preset threshold as image modalities;

[0069] The text modality extraction method includes: obtaining a 768-dimensional semantic vector through a Bidirectional Encoder Representations from Transformers (BERT) pre-trained model; outputting a sentiment polarity score based on a robustly optimized BERT pretraining approach (RoBERTa) model; and determining that a text modality is one whose sentiment polarity score exceeds the corresponding preset semantic vector.

[0070] The method of extracting behavioral modes includes: obtaining the user click heat map; converting the user click heat map into a 128-dimensional action pattern vector through Long Short-Term Memory (LSTM) encoding, and using the 128-dimensional action pattern vector as the action mode.

[0071] Exemplarily, the user click heat map is a 100×100 pixel heat map.

[0072] Furthermore, after determining the speech modality, text modality, image modality, and behavioral modality, the method further includes: performing feature normalization processing on the speech modality, text modality, image modality, and behavioral modality to determine a standardized feature set. The formula for the standardized feature set is:

[0073] F' = {f'1,f'2,f'3,f'4};

[0074] Among them, F' is the standardized feature set, f'1 is the standardized speech modality feature, f'2 is the standardized text modality feature, f'3 is the standardized image modality feature, and f'4 is the standardized behavior modality feature.

[0075] f' i =LayerNorm(FC(f i ));

[0076] Among them, f' i is any standardized feature, LayerNorm is layer normalization, FC is the fully connected layer, f i For any mode.

[0077] Through standardization, multiple modal features are mapped to the same dimension, and the feature scale differences are eliminated through layer normalization. By processing a single modality through a fully connected layer, the dimensionality reduction / increase conversion of single modal features is completed.

[0078] Furthermore, multimodal timing alignment uses the Dynamic Time Warping (DTW) algorithm to synchronize the timestamps between multiple modalities and calculate the alignment cost matrix:

[0079] D(i,j)=max(D(i-1,j),D(i,j-1),D(i-1,j-1));

[0080] Where i represents the modal index of the first time series, j represents the modal index of the second time series, D(i, j) is the minimum accumulated alignment cost from the starting point (1, 1) to the current point (i, j), D(i-1, j) is the cumulative cost from (i-1, j) to (i, j), D(i, j-1) is the cumulative cost from (i, j-1) to (i, j), and D(i-1, j-1) is the cumulative cost from (i-1, j-1) to (i, j).

[0081] By aligning the cost matrix D, the timestamps between the standardized modalities are synchronized. Since multiple data in the original data set correspond to the same business process, the timestamps are aligned to further align the standardized modalities, thereby improving the data accuracy of the user data set.

[0082] For example, the voice data is the voice prompt received by the user when the mobile online banking API payment is completed, the image data is the reminder image or user face image received by the user when the online banking API payment is completed, the text data is the relevant text information displayed on the mobile terminal when the user completes the online banking API payment, and the behavioral data is the screen touch instructions received by the mobile terminal during the online banking API payment operation.

[0083] In some embodiments, optionally, as Figure 3 As shown, step S1004: determining a denoised data set corresponding to the user's original data set, including:

[0084] Step S10042: Determine sensitive features in the user's original data set and mark multiple sensitive features;

[0085] Step S10044: stripping off the marked sensitive features in the voice data, image data, text data and behavior data respectively to determine the denoising data set.

[0086] In this embodiment, a denoised dataset is determined before extracting features from different modalities. Many data within the denoised dataset have been stripped of sensitive features. Because poisoning attacks can target sensitive user information, sensitive features are stripped from the user's original dataset during the preprocessing phase, eliminating tampering opportunities for attackers. This protects user privacy while ensuring the effectiveness of the defense system. By retaining high-order features after desensitization, data accuracy is ensured during subsequent multimodal fusion and attack detection.

[0087] Furthermore, by stripping sensitive features to determine the denoised dataset, the limitations of the user's original dataset are eliminated, and the adaptability of the multimodal poisoning attack defense method to different business scenarios is improved.

[0088] Specifically, the sensitive features in the user's original data set are first marked. The sensitive features in voice data include: user identity features, conversation environment features, and user voice emotion features; the sensitive features in image data include: facial key points, facial expression features, and skin color features; the sensitive features in text data include: personal identity information, account information, and password information; the sensitive features in behavioral data include: precise location information, device unique identification, and input mode features.

[0089] After the sensitive features are marked, the marked features are stripped. Each sensitive feature has a corresponding blurring or replacement method, which converts the sensitive feature into fuzzy data through blurring or replaces it with a universal parameter through replacement.

[0090] Exemplarily, stripping sensitive features in the preprocessing stage includes:

[0091] Voice data stripping: Using spectrum perturbation technology, random noise is added to the emotional characteristics of the user's voice;

[0092] Image data stripping: Perform random pixel shifting on facial key points in the face data to determine the key point offset layer; perform Gaussian blur processing on the key point offset layer to determine the data stripping layer, and use the data stripping layer as the new face image data;

[0093] Text data stripping: Obtain universal markers; Use entity replacement technology to replace sensitive features in text data with universal markers;

[0094] Behavioral data stripping: Determine the information granularity parameters of sensitive features in behavioral data; reduce the information granularity parameters. For example: obtain precise location information and determine the granularity parameters of the user's precise location information; reduce the intensity parameters of the user's precise location information to convert the user's precise location coordinates into regional range data.

[0095] It can be understood that by stripping away the user's sensitive features in the user's original data set, the user's information security is ensured and the user's data security and privacy in banking-related business scenarios are improved.

[0096] In some embodiments, optionally, as Figure 4 As shown, step S102: determining the fusion features corresponding to the preprocessed data set, including:

[0097] Step S1022: Obtaining a learning parameter matrix;

[0098] Step S1024: determining a feature matrix corresponding to the preprocessed data set according to the learning parameter matrix;

[0099] Step S1026: Perform multi-head attention mechanism calculation on the feature matrix to determine the fusion feature.

[0100] In this embodiment, the dependencies between multiple modalities are captured through a multi-head attention mechanism, the features of each modality are converted into a query matrix, a key matrix, and a value matrix, and the cross-modal attention weights are calculated. The residual connection and layer normalization are combined to fuse the multiple modal features and determine the fused features.

[0101] Understandably, by integrating heterogeneous data such as speech, image, text, and behavioral features, we can further eliminate the noise interference of a single modality. By fusing features from multiple modalities, we can improve feature expression capabilities, imbuing the fused features with complex meanings and addressing the problem of a single modality being unable to adequately represent complex patterns. Furthermore, by fusing features from multiple modalities, we can prevent modality substitution attacks, thereby improving the robustness of subsequent attack detection.

[0102] Specifically, three sets of weight matrices are obtained through training and learning, that is, the learning parameter matrix includes: query weight matrix, key weight matrix and value weight matrix. The weight matrix maps multiple modal features to different semantic spaces, and the initial value of the weight matrix is ​​randomly initialized during the training process. The standardized features are converted into query matrix, key matrix and value matrix through the weight matrix. The query matrix, key matrix and value matrix are divided into multiple sub-matrices, that is, multiple attention heads, where each head independently calculates the attention weight. The attention weight calculates the correlation strength between different modal features through the softmax function, for example, the consistency weight between speech emotion and text instructions. Multiple sub-matrices are combined with the attention weight to generate subspace fusion features, and a unified 256-dimensional fusion feature is generated by splicing the outputs of all heads and passing through the fully connected layer.

[0103] In the process of fusion feature generation, residual connections and layer normalization are combined to prevent the gradient disappearance between modalities.

[0104] The formula for converting each modal feature of the user into a feature matrix is ​​as follows:

[0105] Q=F′×W Q ;

[0106] K=F′×W K ;

[0107] V=F′×W V ;

[0108] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, F′ is the standardized feature set, and W Q 、W K and W V Both are learning parameter matrices, WQ is the query weight matrix, W K is the key weight matrix, W V is the value weight matrix.

[0109] The calculation formula based on the attention mechanism is as follows:

[0110]

[0111] Among them, d k is the key dimension, Q is the query matrix, K is the key matrix, V is the value matrix, K T is the transpose of the key matrix, and Attention(Q,K,V) is the attention output matrix.

[0112]

[0113] MultiHead(Q,K,V)=Concat(head1,...,head h )×W o ;

[0114] Where j is the modal index of the second time series, Q is the query matrix, K is the key matrix, V is the value matrix, and j is the modal order. is the query weight projection matrix of the j-th head, is the key weight projection matrix of the j-th head, is the weighted projection matrix of the jth head, Concat is the connection function, head j is the jth head, h is the number of attention heads, W o is the output control matrix, and MultiHead(Q,K,V) is the multi-head attention output matrix.

[0115] Optionally, a learning parameter matrix is ​​determined according to different banking business scenarios, and corresponding weights are assigned to business scenarios by sensing different business scenarios, and the weights are used as initial values ​​of the learning parameter matrix.

[0116] The formula for determining the weight ω of business scenario perception is as follows:

[0117] ω={ω1,ω2,ω3,ω4};

[0118] Among them, ω1 is the speech modality weight, ω2 is the image modality weight, ω3 is the text modality weight, and ω4 is the behavior modality weight.

[0119] For example, in the remittance business scenario, ω1 = 0.3, ω2 = 0.2, ω3 = 0.1, ω4 = 0.4; in the account opening business scenario, ω1 = 0.3, ω2 = 0.2, ω3 = 0.4, ω4 = 0.1; in the complaint handling business scenario, ω1 = 0.3, ω2 = 0.5, ω3 = 0.1, ω4 = 0.1; in the information query business scenario, ω1 = 0.2, ω2 = 0.4, ω3 = 0.1, ω4 = 0.3.

[0120] The formula for generating fusion features is as follows:

[0121] X fused =LayerNorm(MultiHead(Q,K,V)+Residual));

[0122] Among them, X fused is the fusion feature, LayerNorm is the layer normalization, Residual is the residual connection, and MultiHead(Q,K,V) is the multi-head attention output matrix.

[0123] In some embodiments, optionally, as Figure 5 As shown, step S106: performing quality assessment on the reconstructed features and determining the confidence of the fused features, including:

[0124] Step S1062: performing reconstruction error analysis on the reconstructed features to determine reconstruction error analysis parameters;

[0125] Step S1064: determining a reconstruction quality parameter according to the reconstruction error analysis parameter;

[0126] Step S1066: determining the mutual information corresponding to the modal features and the fusion features;

[0127] Step S1068: Determine the modal consistency parameter based on the mutual information;

[0128] Step S1070: Determine the fusion feature confidence level according to the modal consistency parameter and the reconstruction quality parameter.

[0129] In this embodiment, the generator in the generative adversarial network is used to reconstruct the fused features and calculate the reconstruction error analysis parameter between the fused and reconstructed features. The reconstruction error analysis parameter is converted into a quality score, namely the reconstruction quality parameter, through a function model. By determining the reconstruction error analysis parameter and the reconstruction quality parameter, anomaly detection is performed. Normal data, due to real user operations, conforms to the natural distribution, so the corresponding reconstruction error of normal data is small. However, the reconstruction error value of attack data is larger than the average value because it is secondary generated data. The reconstruction error analysis parameter and the reconstruction quality parameter are used to identify synthetic or tampered parameters, thereby improving the robustness of user anomaly detection.

[0130] Furthermore, the mutual information between multiple modal features and fusion features is determined, and the modal consistency parameter is determined based on the mutual information to detect logical contradictions in cross-modal detection of multimodal features.

[0131] It can be understood that through the dual quality evaluation of modal consistency parameters and reconstruction quality parameters, deep verification of multimodal data is achieved, the confidence of fusion features is determined, and while focusing on data authenticity, synthetic or tampered attack data in the user's original data set is identified, and the logical consistency between multimodal data is ensured through mutual information, thereby improving the data accuracy of subsequent attack detection.

[0132] Specifically, the fusion features are reconstructed based on the generated adversarial network. The formula is as follows:

[0133] X recon =G(X fused );

[0134] Among them, X recon is the reconstruction feature, G is the generator in the generative adversarial network, X fused For fusion features.

[0135] The formula for reconstruction error analysis of fusion features is as follows:

[0136]

[0137] Among them, E recon is the reconstruction error analysis parameter, G is the generator in the generative adversarial network, X fused For fusion features.

[0138] The formula for the reconstruction quality score is as follows:

[0139] C recon =sigmoid(β×(θ recon -E rec0n ));

[0140] Among them, sigmoid is the S-type activation function, β is the sensitivity parameter, C recon To reconstruct the quality parameter, E recon is the reconstruction error analysis parameter, θ recon is the reconstruction error threshold.

[0141] The formula for determining the mutual information corresponding to the modal features and the fusion features is as follows:

[0142] MI i =MI(f i ,X fused )=H(f i )+H(Xfused )-H(f i ,X fused );

[0143] Among them, i is the modal index of the first time series, MI i is the mutual information of the i-th normalized feature, X fused

[0144] is the fusion feature, f i is any standardized feature, H(f i ) is the entropy of any standardized feature, H(X fused ) is a fusion feature

[0145] The entropy of the characteristic, H(f i ,X fused ) is the joint entropy of any standardized feature and fusion feature, MI(f i ,X fused ) represents the mutual information between any standardized feature and the fusion feature.

[0146] The formula for determining the modal consistency parameters is as follows:

[0147]

[0148] Among them, i is the modal index of the first time series, w i is the weight of the i-th mode, n is the total number of modes, C modal is the modal consistency parameter, MI i is the mutual information of the i-th normalized feature.

[0149] The calculation formula for multimodal feature confidence is as follows:

[0150] C fusion =λ×C modal +(1-λ)×C recon ;

[0151] Among them, C recon To reconstruct the quality parameter, C modal is the modal consistency parameter, C fusion is the multimodal feature confidence, and λ is the second balancing parameter.

[0152] Optionally, the second balance parameter is dynamically adjusted according to the business scenario. The default value of the second balance parameter is 0.6. The system is set with a risk threshold and detects the risk parameter in real time through the multimodal poisoning attack defense system. When the risk parameter is higher than the risk threshold, the business scenario is determined to be a high-risk scenario, and the second balance parameter is automatically adjusted to 0.5. Under this balance parameter setting, the reconstruction quality and modal consistency are equally important; when the risk parameter is lower than the risk threshold, the business scenario is determined to be a low-risk scenario, and the second balance parameter is automatically adjusted to 0.7. Under this balance parameter setting, modal consistency is more important.

[0153] In some embodiments, optionally, as Figure 6 As shown, step S110: inputting historical attack samples and pre-processed datasets into the generative adversarial network to determine the attack pattern matching confidence, including:

[0154] Step S1102: Receive historical attack samples through the generator in the generative adversarial network;

[0155] Step S1104: Generate poisoning attack samples based on historical attack samples;

[0156] Step S1106: Processing modal features using a dual-branch network architecture through a discriminator in a generative adversarial network, where the dual-branch network architecture includes a first branch corresponding to the preprocessed dataset and a second branch corresponding to the poisoning attack sample;

[0157] Step S1108: determining a user feature query matrix according to the first branch;

[0158] Step S1110: determining the poisoning attack sample feature key matrix and the poisoning attack sample feature value matrix according to the second branch;

[0159] Step S1112: Determine a cross-modal attention parameter based on the user feature query matrix, the poisoning attack sample feature key matrix, and the poisoning attack sample feature value matrix;

[0160] Step S1114: Determine the attack pattern matching confidence based on the cross-modal attention parameters.

[0161] In this embodiment, by combining the generative adversarial network and the cross-modal attention mechanism, multimodal poisoning attacks are detected to determine the matching confidence between the user's original dataset and the poisoning attack sample, that is, the attack pattern matching confidence.

[0162] As can be understood, the dual-branch architecture compares user characteristics with poisoning attack samples, and combines it with an attention mechanism to focus on key features, reducing redundant computation. By integrating text, voice, image, and behavioral features through cross-modal attention, it identifies combined attacks from at least one modality, improving the accuracy and reliability of poisoning attack defenses.

[0163] For example, a lightweight generative adversarial network model is deployed on servers at multiple bank branches. Historical attack samples from these branches are collected and fed into a generator as training data. Simulated poisoning attack adversarial samples are then generated using a multi-layer convolutional neural network. Poisoning attack adversarial samples include voice poisoning adversarial samples, image poisoning adversarial samples, text poisoning adversarial samples, and behavioral poisoning adversarial samples. For example, voice poisoning adversarial samples are speech data with added background noise or voice distortion; image poisoning adversarial samples are image data with Gaussian noise or local blurring; and text poisoning adversarial samples are text data with synonym replacement or semantic flipping.

[0164] Furthermore, the discriminator in the generative adversarial network includes a two-branch network architecture, wherein the first branch corresponds to the preprocessed dataset and serves as the user branch to process the user's real-time multimodal data. The first branch includes a modality-specific encoder, which completes the operation of mapping multiple modalities in the preprocessed dataset to the same dimension through the modality-specific encoder. The second branch corresponds to the poisoning attack samples and uses the same encoder architecture as the first branch to process multiple modalities, learn the common feature distribution of the poisoning attack adversarial samples generated by the generator, and map the common feature distribution to the same dimension.

[0165] Furthermore, the fusion layer in the generative adversarial network combines the cross-branch attention mechanism to determine the cross-modal attention output matrix.

[0166] Specifically, the two branch information is fused through the cross-branch attention mechanism, and the formula for calculating the cross-modal attention parameters of the first branch and the second branch is:

[0167]

[0168] Among them, q is the user feature query matrix, k is the poisoning attack sample feature key value matrix, v is the poisoning attack sample feature value matrix, d k is the key latitude, k T is the transpose of the key matrix, Attention(q,k,v) is the cross-modal attention output matrix, and softmax is the normalization function.

[0169] Exemplarily, the poisoned attack sample feature key-value matrix represents the mapping of the poisoned attack sample, which is used to perform similarity calculation with the user query feature Q. The system matches potential attack patterns through the poisoned attack sample feature key-value matrix. The poisoned attack sample feature key-value matrix determines the part of the user input feature corresponding to the attack sample; the poisoned attack sample feature value matrix contains the actual content or semantic information of the poisoned attack sample.

[0170] The formula for determining the attack pattern matching confidence based on the cross-modal attention parameters is as follows:

[0171] C pattern =1-softmax(Attention(q,k,v)×1 d );

[0172] Among them, Attention(q,k,v) is the cross-modal attention output matrix, C pattern is the attack pattern matching confidence, 1 d is a vector whose elements are all 1, and softmax is a normalization function.

[0173] It can be understood that by multiplying a vector with all elements equal to 1 with the cross-modal attention output matrix, the multi-dimensional weights are compressed into a scalar value for easy normalization.

[0174] In some embodiments, optionally, as Figure 7 As shown, step S114: triggering a defense action according to the security confidence parameter includes:

[0175] Step S1142: Obtaining a defense response strategy;

[0176] Step S1144: determining the risk level according to the security confidence parameter;

[0177] Step S1146: Determine a defense action based on the risk level and the defense response strategy.

[0178] In this embodiment, a defense response strategy dynamically responds to the user's input raw data set. The security confidence parameter determines the corresponding interval, and the confidence interval determines the risk level of the user's raw data set. Defense actions are then taken based on the risk level and parameters preset in the defense response strategy. The defense response strategy includes a risk level, a confidence interval, triggering conditions, and defense actions. The security confidence value is used to determine the risk level of the user's raw data, and defense actions are then taken based on the risk level.

[0179] It can be understood that by presetting different risk levels and matching the risk levels with confidence intervals, defensive actions are triggered to avoid manual decision-making delays and subjectivity, ensure real-time interception when the original data set input by the user includes poisoning attack samples, and improve the timeliness of attack interception in the business environment, thereby protecting user information security and preventing poisoning attacks from further spreading in the business environment.

[0180] In some embodiments, optionally, as Figure 8 As shown, the multimodal poisoning attack defense method also includes:

[0181] Step S116: determining a six-stage dialogue process based on a hidden Markov model;

[0182] Step S118: determining abnormal conditions during the processing of the pre-processed data set according to the six-stage dialogue process;

[0183] Step S120: When an abnormal state is detected, a defensive action is triggered.

[0184] In this embodiment, a dynamic session state machine (DSSM) uses a hidden Markov model (HMM) to construct a six-stage session flow. The six-stage process includes identity authentication, service processing, confirmation, execution, and termination. The DSSM calculates the probability of abnormal state transitions in real time. When an illegal jump is detected, a circuit breaker mechanism is triggered and the attack path is traced back.

[0185] As you can understand, during a user's banking process, the system monitors the status of user data transfers in real time and calculates the probability of abnormal transfers. If it detects a jump or operation that falls outside the six-stage conversation process, it automatically determines it as the highest risk level, triggers a circuit breaker on the user's business system, and terminates all current transactions. The system then automatically traces the attack path associated with the abnormal jump or operation. By monitoring the entire user transaction process in real time, the accuracy of attack interception in the business environment and the reliability of the banking system are improved.

[0186] The defense response strategy is shown in Table 1:

[0187] Table 1

[0188]

[0189] As shown in Table 1, C is the safety confidence parameter. The risk levels include: safe, low risk, medium risk, high risk and extremely high risk. The confidence interval corresponding to the safe level is C>0.9; the confidence interval corresponding to the low risk level is 0.7<C≤0.9; the confidence interval corresponding to the medium risk is 0.5<C≤0.7; the confidence interval corresponding to the high risk level is 0.3<C≤0.5; and the confidence interval corresponding to the extremely high risk level is C≤0.3. At the security level, the system processes user information normally and no defense action is triggered, that is, it follows the standard processing procedure; at the low risk level, the system enhances monitoring, triggers defense actions in the low risk state, and enhances customer background audits; at the medium risk level, the system switches to manual takeover, triggers defense actions in the medium risk state, suspends the user's current operation and verifies the identity; at the high risk level, the system blocks the user's device, triggers defense actions in the high risk state, terminates the session between the system and the user, and marks the terminal as a high-risk terminal; at the extremely high risk level, the system circuit breaker triggers defense actions in the extremely high risk state, terminates all transactions and triggers a system alarm.

[0190] In one specific embodiment, the multimodal poisoning attack defense system optionally includes a multimodal context-aware defense module (i.e., a perceptual defense module). This module constructs a six-stage conversation process (identity authentication → service processing → confirmation → execution → termination) based on a hidden Markov model and calculates the probability of abnormal state transitions in real time. When an illegal jump is detected (e.g., a direct transfer without identity verification), a circuit breaker mechanism is triggered and the attack path is retraced.

[0191] A dynamic time warping algorithm is used to align the time axes of speech, text, and images, and a cross-modal consistency score is defined.

[0192] The multimodal poisoning attack defense system deploys a lightweight GAN model on bank branch servers through a lightweight federal defense module. It outputs poisoning attack adversarial samples based on a localized desensitized data generator. The discriminator processes multimodal features through a dual-branch network, outputs a joint confidence score, and dynamically enhances the robustness of the detection model.

[0193] Among them, the function of the generator G is to receive the historical attack samples of the localized desensitization of the bank branch server as input, and generate simulated poisoning attack adversarial samples through a multi-layer convolutional neural network.

[0194] The generator G includes: encoding layer: 3 convolution layers (the convolution kernel sizes are 5×5, 3×3, and 3×3 respectively) to extract attack features; conversion layer: 2 fully connected layers (1024 nodes and 512 nodes) to achieve feature mapping; decoding layer: 3 deconvolution layers to reconstruct and generate high-quality adversarial samples.

[0195] The types of poisoned adversarial samples generated include: voice poisoning: adding background noise (signal-to-noise ratio <5dB) or voice-changing interference; image poisoning: injecting Gaussian noise with a peak signal-to-noise ratio (PSNR) of <20dB, or partially blurring the image; text poisoning: synonym replacement (replacement rate 5%-15%) or semantic reversal.

[0196] The discriminator D uses a dual-branch network architecture to process multimodal features.

[0197] The federal defense module uses secure multi-party computing (MPC) to aggregate the poisoning attack adversarial sample features generated by various bank branches.

[0198] Each bank branch maintains and updates a local poisoning attack sample library to form a local knowledge graph library (the knowledge in the knowledge graph mainly includes attack feature information such as deep fake images, device fingerprints, IP addresses, and black market rhetoric; the knowledge graph relationships mainly include cross-institutional money laundering gang associations, high-frequency abnormal transaction patterns and other behaviors). The aggregation (Krum) algorithm for defending against malicious client attacks selects attack features that are consistent with the majority of nodes, and homomorphically encrypts the attack features of each branch and updates the entity library and association relationships in the global attack sample knowledge graph.

[0199] The raw data used for training in the multimodal poisoning attack defense system comes from various data entry points such as bank ATMs, counters, smart terminals, mobile banking apps, and online banking APIs. Specifically, it includes user ATM operation video streams, touch screen interaction behaviors, and voice commands at the bank ATM port; multimodal data of the user identity authentication process at the counter terminal, including document scans, facial features, and signature features; and user online interaction data on the mobile banking app and online banking, including click streams, dwell time, input features, user consultation content, and question type distribution.

[0200] By integrating the multimodal poisoning attack defense system with various banking business processes, more accurate attack detection and defense can be achieved. Business processes include:

[0201] Account opening service: connects with the account real-name authentication process, extracts multimodal data from the authentication process, and performs defense detection;

[0202] Transfer and remittance business: Connect with the risk control system to focus on monitoring multi-modal anomalies in scenarios such as high-frequency small-value transfers, inter-bank fund collection, and large-value transfers. For example, when a "system circuit breaker" is triggered, the bank's anti-money laundering system is simultaneously notified for linked anti-money laundering risk analysis.

[0203] Loan application business: connect with the fraud detection system to analyze abnormal behavior during the loan application process;

[0204] Complaint handling: Connect with the customer satisfaction evaluation system to prevent poisoning attacks during the complaint process; identify specific misleading tactics targeting bank customers and detect and defend against poisoning attacks using social engineering.

[0205] The calculation of attack confidence is the core of the multimodal data poisoning attack defense system. Its design comprehensively considers multimodal data characteristics, cross-modal consistency, and attack pattern matching. The following is the specific defense implementation process:

[0206] Step 1: Data collection and sensitive feature stripping:

[0207] 1.1: Banking Data Collection: The raw data used for training comes from various data entry points such as bank automated teller machines (ATMs), teller counters, smart terminals, mobile banking apps, and online banking APIs. Specifically, this includes user ATM operation video streams, touchscreen interaction behaviors, and voice commands at bank ATM ports; multimodal data from the user identity authentication process at teller terminals, including document scans, facial features, and signature features; and online user interaction data from mobile banking apps and online banking, including clickstreams, dwell time, input features, user inquiry content, and question type distribution.

[0208] 1.2: Sensitive feature labeling: First, sensitive modalities are identified at the raw data level. Poisoning attacks often attempt to obtain sensitive user information. Removing sensitive features during the preprocessing phase protects user privacy while maintaining the effectiveness of the defense system. This provides cleaned, high-quality data for subsequent analysis and reduces data noise.

[0209] Sensitive modal markers at the raw data level: Voiceprint data: speaker identity characteristics, speaking environment characteristics, and emotional characteristics; facial data: facial key points (eyes, nose, mouth), facial expression characteristics, and skin color characteristics; text data: personal identity information (name, phone number, ID number), account information, and password information; behavioral data: precise location information, unique device identifier, and input mode characteristics.

[0210] 1.3: Sensitive feature stripping: Stripping in the raw data preprocessing stage: Voice data: Spectral perturbation technology (adding ±2dB random noise) blurs the speaker's features; Face data: Key point perturbation (±2 pixel random offset) and Gaussian blur (σ=1.5) processing; Text data: Entity replacement technology, using universal tags to replace sensitive information; Behavioral data: The granularity of location information is reduced (accurate to the area rather than the exact coordinates).

[0211] Step 2: Multimodal Data Feature Extraction and Standardization: Features from different modalities require specialized neural network architectures for extraction. For example, speech requires a time series model, and images require a convolutional network. The original features of each modality have inconsistent dimensions and must be mapped to a unified dimension for subsequent fusion and comparison. Using specialized models, we extract the most valuable and discriminative features, converting heterogeneous features from different modalities into a common feature space. This eliminates dimensional differences between features from different modalities and ensures more accurate subsequent calculations.

[0212] 2.1: Modal Feature Extraction: Speech modality f1: Extracts 13-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) to characterize the vocal tract characteristics of speech; extracts a 128-dimensional voiceprint vector using ResNet-50. Text modality f2: Obtains a 768-dimensional semantic vector using the BERT pre-trained model; outputs a sentiment polarity score (range [0, 1]) based on the RoBERTa model. Image modality f3: Extracts 2048-dimensional convolutional layer features using ResNet; calculates the liveness probability (range [0, 1]) using 3D structured light analysis. Behavioral modality f4: Utilizes a 100×100 pixel heat map of user clicks to reflect areas of high user click density; and generates a 128-dimensional behavioral pattern vector encoded using LSTM.

[0213] 2.2: Feature normalization: All features are uniformly mapped to the same dimension d = 256. LayerNorm is a layer normalization operation to eliminate feature scale differences. FC is a fully connected layer that performs feature dimensionality reduction and dimensionality increase.

[0214] The formula for normalizing the feature set is:

[0215] F' = {f'1,f'2,f'3,f'4};

[0216] Among them, F' is the standardized feature set, f'1 is the standardized speech modality feature, f'2 is the standardized text modality feature, f'3 is the standardized image modality feature, and f'4 is the standardized behavior modality feature.

[0217] f' i =LayerNorm(FC(f i ));

[0218] Among them, f' i is any standardized feature, LayerNorm is layer normalization, FC is the fully connected layer, f i For any mode.

[0219] Step 3: Multimodal feature fusion: Traditional simple concatenation or average fusion methods ignore the interactive relationship between modalities. The multi-head attention mechanism can capture the complex dependencies between modalities. In different banking business scenarios, the importance of each modality varies, and the weights need to be dynamically adjusted.

[0220] 3.1: Multimodal timing alignment: Use the dynamic time warping algorithm to synchronize multimodal timestamps and calculate the alignment cost matrix:

[0221] D(i,j)=max(D(i-1,j),D(i,j-1),D(i-1,j-1));

[0222] Where i represents the modal index of the first time series, j represents the modal index of the second time series, D(i, j) is the minimum accumulated alignment cost from the starting point (1, 1) to the current point (i, j), D(i-1, j) is the cumulative cost from (i-1, j) to (i, j), D(i, j-1) is the cumulative cost from (i, j-1) to (i, j), and D(i-1, j-1) is the cumulative cost from (i-1, j-1) to (i, j).

[0223] 3.2: Multi-head attention mechanism for multimodal feature fusion: The formula for converting the user's modal features into a feature matrix is ​​as follows:

[0224] Q=F′×W Q ;

[0225] K=F′×W K ;

[0226] V=F′×W V ;

[0227] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, F′ is the standardized feature set, and W Q 、W K and W V Both are learning parameter matrices, W Q is the query weight matrix, W K is the key weight matrix, W V is the value weight matrix.

[0228] Q, K, and V all come from different transformations of all multimodal features. Q 、W K and W V It is a learnable parameter matrix, which is the weight parameter learned from the training data rather than an artificially fixed value. It is used to transform the input features into a suitable feature space and help the model capture the key patterns in the data. The initial value is randomly initialized during the training process.

[0229] After the attention mechanism is introduced, the calculation formula is as follows:

[0230]

[0231] Among them, d kis the key dimension, Q is the query matrix, K is the key matrix, V is the value matrix, K T is the transpose of the key matrix, Attention(Q,K,V) is the attention output matrix, and softmax is the normalization function.

[0232] Introducing multi-head attention mechanism calculation for multimodal feature fusion:

[0233]

[0234] MultiHead(Q,K,V)=Concat(head1,...,head h )×W o ;

[0235] Among them, Q is the query matrix, K is the key matrix, and V is the value matrix. is the query weight projection matrix of the j-th head, is the key weight projection matrix of the j-th head, is the weighted projection matrix of the jth head, Concat is the connection function, head j is the jth head, h is the number of attention heads, W o is the output control matrix, and MultiHead(Q,K,V) is the multi-head attention output matrix.

[0236] 3.3: Dynamic weight allocation: Set the weight ω of business scenario perception. The formula for determining the weight ω of business scenario perception is as follows:

[0237] ω={ω1,ω2,ω3,ω4};

[0238] Among them, ω1 is the speech modality weight, ω2 is the image modality weight, ω3 is the text modality weight, and ω4 is the behavior modality weight.

[0239] For example, in the remittance business scenario, ω1 = 0.3, ω2 = 0.2, ω3 = 0.1, ω4 = 0.4; in the account opening business scenario, ω1 = 0.3, ω2 = 0.2, ω3 = 0.4, ω4 = 0.1; in the complaint handling business scenario, ω1 = 0.3, ω2 = 0.5, ω3 = 0.1, ω4 = 0.1; in the information query business scenario, ω1 = 0.2, ω2 = 0.4, ω3 = 0.1, ω4 = 0.3.

[0240] 3.4: Fusion feature generation: The formula for fusion feature generation is as follows:

[0241] X fused =LayerNorm(MultiHead(Q,K,V)+Residual);

[0242] Among them, X fused is the fusion feature, LayerNorm is the layer normalization, Residual is the residual connection, and MultiHead(Q,K,V) is the multi-head attention output matrix.

[0243] Step 4: Restructure the quality assessment process to detect internal consistency of the data and discover unnatural or artificially synthesized anomalies. This serves as the first line of defense for anomaly detection and is also effective against unknown attack patterns. It also assesses feature quality and provides a reliability reference for subsequent defense decisions.

[0244] 4.1: Feature reconstruction: Reconstruct the fusion features based on the generative adversarial network. The formula is as follows:

[0245] X recon =G(X fused );

[0246] Among them, X recon is the reconstruction feature, G is the generator in the generative adversarial network, X fused To fuse features, normal data and abnormal data show obvious differences when reconstructed, and normal data is easier to be accurately reconstructed.

[0247] 4.2: Reconstruction error analysis: The formula for reconstruction error analysis of fusion features is as follows:

[0248]

[0249] Among them, E recon is the reconstruction error analysis parameter, G is the generator in the generative adversarial network, X fused For fusion features.

[0250] 4.3: Reconstruction quality score: The formula for reconstruction quality score is as follows:

[0251] C recon =sigmoid(β×(θ recon -E recon ));

[0252] Among them, sigmoid is the S-type activation function, β is the sensitivity parameter, C recon To reconstruct the quality parameter, E recon is the reconstruction error analysis parameter, θ rrcon is the reconstruction error threshold.

[0253] Exemplarily, sigmoid(x)=1 / (1+e^(-x)), where e is a natural constant and x is an activation object.

[0254] For example, β is a sensitivity parameter with a default value of 10; the default value of the reconstruction error threshold is 0.15; C recon The larger the value of ∈[0,1], the higher the reconstruction quality.

[0255] Step 5: Modal mutual information evaluation:

[0256] 5.1: Mutual Information Calculation: Calculate the mutual information between the modal features of the data and the fusion features. The formula for determining the mutual information corresponding to the modal features and the fusion features is as follows:

[0257] MI i =MI(f i ,X fused )=H(f i )+H(X fused )-H(f i ,X fused );

[0258] Where i is the modal index of the first time series, MI i is the mutual information of the i-th normalized feature, X fused for

[0259] Fusion features, f i is any standardized feature, H(f i ) is the entropy of any standardized feature, H(X fused ) is the entropy of the fusion feature, H(f i ,X fused ) is the joint entropy of any standardized feature and fusion feature, MI(f i ,X fused ) represents the mutual information between any standardized feature and the fusion feature.

[0260] 5.2: Modal consistency score. The modal consistency score is defined as the weighted average of the mutual information of each mode. The formula for determining the modal consistency parameter is as follows:

[0261]

[0262] Where i is the modal index of the first time series, w i is the weight of the i-th mode, n is the total number of modes, C modal is the modal consistency parameter, MI i is the mutual information of the i-th normalized feature, C modal The larger the ∈[0,1] value is, the higher the consistency between modalities is, which verifies whether each modality comes from the same interaction scenario, prevents modality replacement attacks, and highlights the contribution of important modalities through the weight mechanism to improve detection accuracy.

[0263] Step 6: Calculate the confidence of multimodal features. The calculation formula of multimodal feature confidence is as follows:

[0264] C fusion =λ×C modal +(1-λ)×C recon ;

[0265] Among them, C recon To reconstruct the quality parameter, C modal is the modal consistency parameter, C fusion is the confidence of multimodal features, λ is the second balance parameter, and λ can be dynamically adjusted according to the business scenario: high-risk scenario: λ = 0.5 (reconstruction quality and modal consistency are equally important); low-risk scenario: λ = 0.7 (modal consistency is more important); C fusion The larger the value of ∈[0,1], the higher the credibility of the fusion feature.

[0266] Step 7: Calculate the attack pattern matching confidence. Aiming at the accurate recognition capability of known attack patterns, a dual-branch structure is used to achieve accurate comparison between user features and attack samples.

[0267] 7.1: Construction of a dual-branch network in the adversarial network. The dual-branch network includes a user branch (first branch) and an attack branch (second branch).

[0268] User branch: Processes real-time multimodal user data, including modality-specific encoders (speech encoder, text encoder, image encoder, and behavior encoder). Each modality is processed by an independent encoder. Speech uses the CNN+LSTM architecture, text uses RoBERTa, images use ResNet-50, and behavior uses LSTM. All features are uniformly mapped to a 256-dimensional space.

[0269] Attack branch: Receives simulated attack samples generated by generator G, processes samples in the local knowledge graph library, uses the same encoder architecture as branch 1 to process each modality, learns the common feature distribution of poisoning attacks, and performs the same 256-dimensional mapping on the attack sample features.

[0270] 7.2: Cross-branch attention calculation,The cross-branch attention mechanism fuses the information of two branches and calculates the cross-modal attention score of user features and attack sample features.

[0271] The formula for calculating the cross-modal attention parameters of the first branch and the second branch is:

[0272]

[0273] Among them, q is the user feature query matrix, k is the poisoning attack sample feature key value matrix, v is the poisoning attack sample feature value matrix, d k is the key latitude, kT is the transpose of the key matrix, Attention(q,k,v) is the cross-modal attention output matrix, and softmax is the normalization function.

[0274] k represents the mapping representation of the attack sample features, used to calculate similarity with the query matrix Q. The system uses it to "index" or "find" potential matching patterns. It determines which parts of the attack sample the user input features should focus on. k comes from the feature representation of the simulated attack sample generated by the GAN generator. v contains the actual content or semantic information of the attack sample. After the system determines the similarity between the user features and the attack features, it extracts relevant information from v. The final output of the attention mechanism is a weighted combination of v, with weights determined by the similarity between q and k. v provides detailed information about the potential attack, which is used for the final defense decision. The system first calculates the similarity between the user features q and the attack features k, converts these similarities into attention weights through the Softmax function, and then uses these weights to perform a weighted combination of v to produce the final feature representation.

[0275] 7.3: Attack pattern matching confidence. The formula for determining the attack pattern matching confidence based on the cross-modal attention parameter is as follows:

[0276] C pattern =1-softmax(Attention(q,k,v)×1 d );

[0277] Among them, Attention(q,k,v) is the cross-modal attention output matrix, C pattern is the attack pattern matching confidence, softmax is the normalization function, 1 d is a vector with all elements set to 1, C pattern Smaller values ​​indicate greater similarity to the attack pattern.

[0278] Step 8: Final confidence calculation:

[0279] 8.1: Final confidence score. The formula used to determine the security confidence parameter based on the fusion feature confidence and attack pattern matching confidence is as follows:

[0280] C=μ×C fusion +(1-μ)×C pattern ;

[0281] Among them, C is the safety confidence parameter, μ is the first balance parameter, C fusion is the confidence of the fusion feature, C patternConfigure the confidence level for the attack mode. μ can be dynamically adjusted according to the business scenario: High-risk scenario: μ = 0.4 (tending to attack mode matching); Low-risk scenario: μ = 0.7 (tending to fusion feature analysis); The larger the value of C ∈ [0, 1], the higher the interactive security level.

[0282] As shown in Table 2, the division thresholds for high-risk scenarios are based on: transfer transactions, sensitive information, and logging in with a new device for the first time. Among them, transfer transactions include: large transfers with amounts exceeding the 80% quantile of the user's historical transfer amount; transactions with amounts above 50,000 yuan, etc.; Sensitive information change operations include: modifying passwords, binding mobile phone numbers, changing security questions, etc.; Logging in with a new device for the first time includes: new IP address, new device identifier, remote login, etc. The dynamic weight corresponding to the high-risk scenario is 0.4, which focuses on the attack mode confidence level.

[0283] The division thresholds for low-risk scenarios are based on: non-fund-sensitive operations, business consultations, and balance query operations. Non-fund-sensitive operations include: account queries and quota applications; Business consultations include: product information, interest rate queries, branch navigation; Balance query operations include information acquisition behaviors that do not involve fund changes. The dynamic weight corresponding to the low-risk scenario is 0.7, which focuses on fusion feature analysis.

[0284] On the basis of the fixed threshold setting, there is also a threshold dynamic adjustment mechanism:

[0285] Quarterly retrospective analysis: Randomly adjust the threshold based on false positives / missed reports, with a ±0.2 floating range;

[0286] Threat intelligence-driven: Temporarily adjust the threshold according to industry threat intelligence, with a ±0.2 floating range;

[0287] A / B test optimization: Continuously optimize the threshold setting through online experiments.

[0288] Table 2

[0289]

[0290] 8.2: Risk level division. The risk levels based on confidence include: Extremely high risk: C ≤ 0.3 (confirmed as an attack behavior); High risk: 0.3 < C ≤ 0.5 (strongly suspected of an attack); Medium risk: 0.5 < C ≤ 0.7 (abnormal); Low risk: 0.7 < C ≤ 0.9 (slightly abnormal); Safe: C > 0.9 (normal interaction).

[0291] 8.3: Set up a defense response strategy. The corresponding response strategies for risk levels include: Extremely high risk: system circuit breaker, termination of all user transactions and triggering of alarms; High risk: device blocking, termination of sessions, marking the terminal as a high-risk device; Medium risk: manual takeover, suspension of current operations and identity verification; Low risk: enhanced monitoring and enhanced customer background audits; Safe: normal processing and standard processing procedures.

[0292] Determine a hierarchical dynamic response strategy and implement a five-level risk level response strategy (normal processing, enhanced monitoring, manual takeover, device blocking, and system fuse) based on the attack confidence level. The strategy response delay is ≤200ms.

[0293] Step 9: Dynamically adjust and optimize the process:

[0294] 9.1: Dynamic parameter adjustment: Dynamically adjust parameters according to business scenarios, including:

[0295] The balance parameters λ and μ are adjusted in real time according to the scenario;

[0296] The modal weight ω is dynamically allocated according to the business type;

[0297] The risk threshold θ is optimized in real time based on threat intelligence.

[0298] 9.2: Model Update Mechanism: Incremental learning updates the defense model, and new attack samples are collected daily to update the attack signature library; incremental model training is performed weekly to update model parameters; full retraining is performed monthly to optimize overall defense capabilities.

[0299] The multimodal poisoning attack defense method for the bank's intelligent customer service system builds a three-dimensional defense system through the above closely connected processes. It can not only accurately identify known attack patterns, but also remain vigilant against unknown threats. At the same time, it customizes defense according to the characteristics of the bank's business scenarios, balancing security and user experience.

[0300] For example, taking the transfer attack of forged voiceprints and tampered documents as an example, the attack process is simulated: the attacker uses AI synthesized voice (voiceprint matching degree 92%) to initiate a large transfer request; at the same time, uploads the PS modified ID card image (the validity period is tampered).

[0301] Defense process:

[0302] (1) Feature extraction: Extract speech MFCC features, detect background noise spectrum anomalies (sudden increase in noise energy above 5kHz), and PS traces of ID images (PSNR < 20dB).

[0303] (2) Feature fusion: DTW alignment found that the speech and text emotions were contradictory (similarity < 0.5);

[0304] (3) Pattern matching: the federated knowledge base matches similar attack patterns (isomorphism 0.85);

[0305] (4) Confidence calculation: C fusion =0.45; C pattern =0.18; C=0.4×0.45+0.6×0.18=0.288.

[0306] Table 3

[0307]

[0308] As shown in Table 3, the confidence interval C≤0.3. The attack ultimately triggered a system response of extremely high risk level, causing the system to trip, terminate all transactions, and trigger an alarm. The system was notified synchronously through federated learning, achieving industry-level risk joint defense.

[0309] like Figure 9 As shown, the present invention provides a multimodal poisoning attack defense system 900, which includes: a dynamic response module 902, used to obtain a preprocessed data set; a feature fusion module 904, used to determine the fusion features corresponding to the preprocessed data set; reconstruct the fusion features to determine the reconstructed features; perform quality assessment on the reconstructed features to determine the confidence of the fusion features; a federal defense module 906, used to obtain historical attack samples; input the historical attack samples and the preprocessed data set into a generative adversarial network to determine the attack pattern matching confidence; determine the security confidence parameter based on the fusion feature confidence and the attack pattern matching confidence; and a security defense module 908, used to trigger a defense action based on the security confidence parameter.

[0310] According to the multimodal poisoning attack defense system 900 provided by the present invention, a multimodal poisoning attack defense method is implemented, the multimodal features in the user data are fused and reconstructed, and the confidence of the fused features is determined; the multimodal poisoning attack samples and the user's multimodal features are analyzed in combination with a generative adversarial network including a dual-branch network architecture to determine the attack pattern matching confidence; finally, the security confidence is determined based on the fused feature confidence and the attack pattern matching confidence, thereby achieving a balance between comprehensive defense and dynamic adaptability and improving the accuracy and robustness of multimodal data poisoning defense.

[0311] In some embodiments, optionally, as Figure 9 As shown, the multimodal poisoning attack defense system 900 also includes: a perception defense module 910, which is used to determine a six-stage dialogue process based on a hidden Markov model; determine the abnormal state during the processing of the preprocessed data set according to the six-stage dialogue process; and trigger the defense action when the abnormal state is detected.

[0312] In this embodiment, the perception and defense module 910 includes a DSSM. The dynamic session state machine (DSSM) constructs a six-stage session flow based on the HMM. The six-stage session flow includes: identity authentication, service processing, confirmation, execution, and termination. The DSSM calculates the probability of abnormal state transitions in real time. When an illegal jump is detected, the circuit breaker mechanism is triggered and the attack path is traced back.

[0313] In the present invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "plurality" refers to two or more, unless expressly limited otherwise. Terms such as "installed," "connected," "connected," and "fixed" should be interpreted broadly. For example, "connected" can mean a fixed connection, a detachable connection, or an integral connection; "connected" can mean a direct connection or an indirect connection through an intermediary. Those skilled in the art will understand the specific meanings of these terms in the present invention based on specific circumstances.

[0314] In the description of the present invention, it should be understood that the directions or positional relationships indicated by terms such as "up", "down", "left", "right", "front" and "back" are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0315] Throughout this specification, terms such as "one embodiment," "some embodiments," and "specific embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0316] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-modal poisoning attack defense method, characterized in that: include: Get the preprocessed dataset; determining fusion features corresponding to the preprocessed data set; Performing feature reconstruction on the fused features to determine reconstructed features; Performing quality assessment on the reconstructed features to determine the confidence of the fused features; Obtain historical attack samples; Inputting the historical attack samples and the preprocessed dataset into a generative adversarial network to determine attack pattern matching confidence; Determining a security confidence parameter according to the fusion feature confidence and the attack pattern matching confidence; A defense action is triggered according to the security confidence parameter.

2. The multimodal poisoning attack defense method according to claim 1, characterized in that: The obtaining of the preprocessing data set comprises: Collecting user original data sets, wherein the user original data sets include voice data, image data, text data and behavior data; Determining a denoised data set corresponding to the user's original data set; Determining a plurality of modal features in the denoised dataset, the modal features comprising a speech modality corresponding to the speech data, an image modality corresponding to the image data, a text modality corresponding to the text data, and a behavioral modality corresponding to the behavioral data; The speech modality, the image modality, the text modality, and the behavior modality are mapped to the same dimension, and time-series alignment is performed to determine a preprocessed data set.

3. The multimodal poisoning attack defense method according to claim 2, characterized in that: The determining of a denoised data set corresponding to the user's original data set includes: Determining sensitive features in the user's original data set and marking a plurality of the sensitive features; The marked sensitive features in the voice data, the image data, the text data, and the behavior data are respectively stripped to determine a denoising data set.

4. The multimodal poisoning attack defense method according to claim 1, characterized in that: The determining of the fusion features corresponding to the preprocessed data set includes: Get the learning parameter matrix; Determining a feature matrix corresponding to the preprocessed data set according to the learning parameter matrix; A multi-head attention mechanism is performed on the feature matrix to determine the fusion features.

5. The multi-modal poisoning attack defense method according to claim 2, characterized in that: The performing quality assessment on the reconstructed features and determining the confidence of the fused features includes: Performing reconstruction error analysis on the reconstruction features to determine reconstruction error analysis parameters; determining a reconstruction quality parameter according to the reconstruction error analysis parameter; determining mutual information corresponding to the modality feature and the fusion feature; determining a modal consistency parameter based on the mutual information; The confidence of the fusion feature is determined according to the modality consistency parameter and the reconstruction quality parameter.

6. The multi-modal poisoning attack defense method according to claim 1, characterized in that: Determining the attack pattern matching confidence level includes: Receiving historical attack samples through the generator in the generative adversarial network; generating a poisoning attack sample based on the historical attack sample; Processing modal features using a dual-branch network architecture through a discriminator in the generative adversarial network, wherein the dual-branch network architecture includes a first branch corresponding to the preprocessed dataset and a second branch corresponding to the poisoning attack sample; Determine a user feature query matrix according to the first branch; Determine the poisoning attack sample feature key matrix and the poisoning attack sample feature value matrix according to the second branch; Determining a cross-modal attention parameter according to the user feature query matrix, the poisoning attack sample feature key matrix, and the poisoning attack sample feature value matrix; The attack pattern matching confidence is determined according to the cross-modal attention parameter.

7. The multi-modal poisoning attack defense method according to claim 1, characterized in that: The triggering of a defense action according to the security confidence parameter includes: Obtain defense response strategies; determining a risk level based on the safety confidence parameter; A defense action is determined according to the risk level and the defense response strategy.

8. The multimodal poisoning attack defense method according to any one of claims 1 to 7, characterized in that: Also includes: Determine the six-stage dialogue process based on the Hidden Markov Model; Determining an abnormal state during processing of the preprocessed data set according to the six-stage dialogue process; When the abnormal state is detected, the defensive action is triggered.

9. A multi-modal poisoning attack defense system, characterized in that: include: Dynamic response module, used to obtain preprocessed data sets; A feature fusion module, configured to determine fusion features corresponding to the preprocessed data set; Performing feature reconstruction on the fused features to determine reconstructed features; Performing quality assessment on the reconstructed features to determine the confidence of the fused features; A federated defense module is configured to obtain historical attack samples; input the historical attack samples and the preprocessed dataset into a generative adversarial network to determine attack pattern matching confidence; Determining a security confidence parameter according to the fusion feature confidence and the attack pattern matching confidence; A security defense module is used to trigger a defense action according to the security confidence parameter.

10. The multi-modal poisoning attack defense system according to claim 9, characterized in that: Also includes: A perception and defense module, which is used to determine the six-stage dialogue process based on the hidden Markov model; Determining an abnormal state during processing of the preprocessed data set according to the six-stage dialogue process; When the abnormal state is detected, the defensive action is triggered.

Images