Aigc model robustness enhancement method and system based on adversarial sample defense

Abstract

The application provides an AIGC model robustness enhancement method and system based on an adversarial sample defense, relates to the technical field of artificial intelligence security, and first acquires an original reasoning path topology graph of an initial generative artificial intelligence model on a clean sample set, then generates an adversarial sample test set and constructs a perturbation reasoning path topology graph set. By comparing and analyzing a key network layer node and an activation offset direction vector, a dynamic path deflection unit is constructed and embedded into the initial model. The adversarial sample test set is used for joint optimization training of an intermediate state model, a target generative artificial intelligence model is generated, the defense capability of the model against adversarial samples is effectively enhanced, and the robustness is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and more specifically, to a method and system for enhancing the robustness of AIGC models based on adversarial example defense. Background Technology

[0002] With the rapid development of generative artificial intelligence (AIGC) technology, AIGC models have demonstrated strong application potential in many fields, such as image generation, text creation, and speech synthesis. However, these models also face serious security challenges, among which adversarial attacks are one of the most prominent issues.

[0003] Adversarial examples refer to carefully designed, subtle perturbations added to normal input samples, causing AIGC models to produce incorrect outputs on seemingly harmless samples. Most existing AIGC models are based on deep learning architectures, whose decision-making processes rely on complex interactions and activation patterns between neurons. However, these models often do not adequately consider the existence of adversarial examples during training, leading to abnormal changes in neuronal activation patterns when faced with adversarial examples. This disrupts the model's original inference path, resulting in incorrect or unreasonable outputs.

[0004] Currently, although there are some studies on model robustness, most of them focus on improving model training methods or increasing data diversity, lacking in-depth analysis of changes in neuronal activation patterns when models face adversarial examples, and targeted defense strategies based on this. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for enhancing the robustness of AIGC models based on adversarial example defense, the method comprising:

[0006] Obtain the original neuron activation pattern sequence generated when the initial generative artificial intelligence model propagates forward layer by layer on a preset clean sample set, and construct the original inference path topology map of the initial generative artificial intelligence model for normal input based on the original neuron activation pattern sequence.

[0007] The clean sample set is perturbed by a preset target perturbation generator to generate an adversarial sample test set covering multiple attack types. The adversarial sample test set is then input into the initial generative artificial intelligence model for forward propagation calculation. The perturbation neuron activation pattern sequence generated by the initial generative artificial intelligence model during layer-by-layer forward propagation is obtained. Based on the perturbation neuron activation pattern sequence, a perturbation inference path topology graph set for the initial generative artificial intelligence model for the perturbed input is constructed.

[0008] The original inference path topology is compared with each perturbation inference path topology in the set of perturbation inference path topology by performing a layer-by-layer node activation difference analysis to identify key network layer nodes in the initial generative artificial intelligence model that have significant shifts in activation mode under perturbation input, as well as the activation shift direction vectors corresponding to the key network layer nodes. The significant shift is determined by comparing the total cross-attack type shift of the network layer nodes with a preset shift threshold.

[0009] Based on the key network layer node and the activation offset direction vector, a dynamic path deflection unit oriented to a specific network layer node is constructed. The dynamic path deflection unit includes an activation direction modulation sublayer and a residual path selection sublayer that can be embedded in the output of the key network layer node.

[0010] The dynamic path deflection unit is embedded into the output of the corresponding key network layer node in the initial generative artificial intelligence model to construct an intermediate state robust enhancement model containing an activation direction modulation mechanism. The intermediate state robust enhancement model is jointly optimized and trained using the adversarial example test set. The internal modulation parameters of the dynamic path deflection unit and the original weight parameters of the initial generative artificial intelligence model are adjusted simultaneously to generate the target generative artificial intelligence model.

[0011] Furthermore, embodiments of the present invention also provide an AIGC model robustness enhancement system based on adversarial example defense, comprising:

[0012] A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the above-described AIGC model robustness enhancement method based on adversarial example defense by executing the machine-executable instructions.

[0013] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including machine-executable instructions stored in a computer-readable storage medium, a processor of an AIGC model robustness enhancement system based on adversarial example defense reading the machine-executable instructions from the computer-readable storage medium, the processor executing the machine-executable instructions, causing the AIGC model robustness enhancement system based on adversarial example defense to perform the aforementioned AIGC model robustness enhancement method based on adversarial example defense.

[0014] Based on the above, by obtaining the original neuron activation pattern sequence of the initial generative artificial intelligence model on a clean sample set, an original inference path topology map for normal input is constructed. A target perturbation generator is used to generate an adversarial sample test set covering multiple attack types, and the perturbed neuron activation pattern sequence of the model under perturbed input is obtained to construct a perturbed inference path topology map set. By comparing and analyzing the original inference path topology map and the perturbed inference path topology map set, the key network layer nodes and their activation offset direction vectors that have significantly shifted their activation patterns under perturbed input can be accurately identified. Based on this, a dynamic path deflection unit is constructed, which can flexibly adjust the inference path of the model when facing adversarial samples through activation direction modulation sublayer and residual path selection sublayer. By embedding dynamic path deflection units into the initial model to construct an intermediate robustness enhancement model, and using adversarial example test sets for joint optimization training, the internal modulation parameters of the dynamic path deflection units and the original weight parameters of the initial model are adjusted simultaneously. This enables the model to automatically adjust the inference path when facing adversarial examples, effectively resisting adversarial attacks, significantly improving the robustness of the target generative artificial intelligence model, and ensuring the stable operation and reliable output of the model in complex environments. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the execution flow of the AIGC model robustness enhancement method based on adversarial sample defense provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of exemplary hardware and software components of the AIGC model robustness enhancement system based on adversarial sample defense provided in an embodiment of the present invention. Detailed Implementation

[0017] Figure 1 This is a flowchart illustrating an AIGC model robustness enhancement method based on adversarial example defense provided in one embodiment of the present invention, which will be described in detail below.

[0018] Step S110: Obtain the original neuron activation pattern sequence generated by the initial generative artificial intelligence model during layer-by-layer forward propagation on a preset clean sample set, and construct the original inference path topology map of the initial generative artificial intelligence model for normal input based on the original neuron activation pattern sequence.

[0019] In this embodiment, an image generation model based on the Transformer architecture is selected as the initial generative AI model. This initial generative AI model includes an input layer, 12 encoder layers, 12 decoder layers, and an output layer, used to generate corresponding images from text descriptions. The preset clean sample set is a public dataset containing text-image pairs. All samples have been manually screened to ensure they do not contain privacy-sensitive information, and the samples cover various scene types such as natural scenery, portraits, animals, and architecture. Before subsequent processing, it is necessary to ensure that the text descriptions in the clean sample set conform to natural language specifications, and that the image data has undergone standardized preprocessing, such as normalizing pixel values ​​to the 0-1 range, to ensure the consistency of the model input.

[0020] Step S111: Obtain multiple clean sample units from the preset clean sample set. The clean sample units are original input text units or original image units without any added perturbation information.

[0021] Sample units were selected from the clean sample set mentioned above as processing objects. Each clean sample unit consists of a text description and a corresponding image. The text description is between 5 and 30 words in length, and the image resolution is 256×256 pixels, containing RGB color channels. The sample units underwent deduplication during selection to ensure no duplicate content. Random sampling was also used to ensure diversity in scene type, text length, and image complexity. For example, natural landscape samples included different sub-scenes such as mountains, rivers, and forests, while portrait samples covered people of different ages, genders, and postures to avoid bias in subsequent analysis results due to sample uniformity. Each clean sample unit was stored as a file in a specific format: the text portion was a UTF-8 encoded plain text file, and the image portion was a PNG file, facilitating subsequent model reading and processing.

[0022] Step S112: Input the multiple clean sample units into the initial generative artificial intelligence model in sequence, and start performing layer-by-layer forward propagation calculation after the input layer of the initial generative artificial intelligence model.

[0023] The text descriptions from the selected clean sample units are sequentially input into the input layer of the initial generative AI model. The input layer first performs word segmentation, using a BPE (byte-pair encoding)-based segmentation algorithm to divide the text into sub-word units. Each sub-word unit is converted into a corresponding word embedding vector through a pre-trained word embedding matrix, with a 512-dimensional word embedding vector. After passing through a positional encoding layer, sine and cosine positional encoding methods are used to add positional information to each word embedding vector. The formula for calculating the positional encoding is: for a word embedding vector at position pos, its i-th dimension positional encoding value is sin(pos / 10000^(2i / d_model)) when i is even, and cos(pos / 10000^(2i / d_model)) when i is odd, where d_model is the 512-dimensional word embedding vector. After obtaining an input feature matrix with dimensions of sequence length × 512, this input feature matrix enters the first encoder layer. Each encoder layer contains a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer uses eight attention heads, each with a dimension of 64. The attention output is obtained by calculating the scaled dot product attention of the query (Q), key (K), and value (V) matrices. Specifically, the input feature matrix is ​​multiplied by the weight matrices Q, K, and V respectively to obtain the Q, K, and V matrices. Then, the dot product of the transposes of Q and K is calculated and divided by the square root of the attention head dimension. This is then passed through a softmax function to obtain the attention weights. Finally, the attention weights are multiplied by the V matrix to obtain the output of a single attention head. The outputs of the eight attention heads are concatenated and subjected to a linear transformation to obtain the output of the multi-head self-attention sublayer. The feedforward neural network sublayer contains two linear transformation layers and a ReLU activation function. The first linear transformation maps the input features from 512 dimensions to 2048 dimensions. After ReLU activation, the second linear transformation maps them back to 512 dimensions. The encoder layer's output undergoes residual connection and layer normalization; that is, the sub-layer output is added to the input, and then layer normalization is performed. The normalization parameters are learnable scaling and translation parameters. The decoder layer has a similar structure to the encoder layer, but adds an encoder-decoder attention sub-layer. This sub-layer uses the encoder layer's output as the K and V matrices, and the decoder layer's multi-head self-attention sub-layer output as the Q matrix, to achieve attention to the encoder output.

[0024] Step S113: During the layer-by-layer forward propagation computation, an activation value capture hook function is set after each network layer of the initial generative artificial intelligence model to capture the original activation tensor output by the current network layer for the clean sample unit of the current input.

[0025] An activation capture hook function is set after each network layer of the model (including the encoder layer, decoder layer, and each sub-layer). The hook function is implemented using the register_forward_hook method in the PyTorch framework. When the model performs forward propagation on a clean sample unit, the hook function captures the original activation tensor of its output immediately after the current network layer has completed computation. For the multi-head self-attention sub-layers of the encoder and decoder layers, the shape of the output original activation tensor is batch size × sequence length × hidden layer dimension, where the batch size is set to 16 according to the hardware device's memory capacity, the sequence length is the number of sub-words after text segmentation, and the hidden layer dimension is 512; the output tensor shape of the feedforward neural network sub-layer is also batch size × sequence length × hidden layer dimension; the output tensor shape of the encoder-decoder attention sub-layer is also batch size × sequence length × hidden layer dimension. The hook function stores the captured raw activation tensors in a designated buffer. Each network layer corresponds to an independent buffer file, named in the format "network layer name_sample unit number_activation tensor.pt", for subsequent extraction and analysis of the activation tensors. Simultaneously, to avoid memory overflow, after capturing the activation tensors of a batch of sample units, the data in the buffer is promptly written to a disk file, and memory space is released.

[0026] Step S114: For each captured original activation tensor, record the activation value corresponding to each neuron in the original activation tensor to generate the original neuron activation vector of the current network layer under the input of the current clean sample unit.

[0027] For each captured raw activation tensor, it is expanded along the neuron dimension. For a raw activation tensor with the shape of batch size × sequence length × hidden layer dimension, where the batch size is 16, the sequence length is 30 (assuming the maximum number of subwords after text segmentation is 30), and the hidden layer dimension is 512, then each raw activation tensor contains 16 × 30 × 512 = 245760 neuron activation values. The activation values ​​of all neurons in the same network layer processing the same clean sample unit are arranged sequentially. Specifically, they are expanded along the batch dimension first, then the activation tensors in each batch are expanded along the sequence length dimension, and the hidden layer vectors at each sequence position are expanded sequentially along the dimensions, forming a one-dimensional vector. For example, for the first sample unit in a batch, the length of its raw activation vector is 30 × 512 = 15360. This raw activation vector is the raw neuron activation vector of the current network layer under the input of the current clean sample unit. The generated raw neuron activation vectors are stored as text files, with each value separated by a space. The network layer name and sample unit number are noted at the beginning of each line for later summarization and calculation.

[0028] Step S115: Summarize the original neuron activation vectors output by the same network layer for all clean sample units, calculate the average neuron activation vector of the network layer under all clean sample unit inputs, and use the average neuron activation vector as the original layer representation vector of the network layer.

[0029] We collect the raw neuron activation vectors output by the same network layer for all clean sample units. Assume we process N clean sample units, and the length of each raw neuron activation vector is L (L=15360 in the example above). Arrange these vectors in rows to form an N×L matrix, where each row represents the raw neuron activation vector corresponding to a clean sample unit. Calculate the average value for each column of this matrix (i.e., each neuron position). Specifically, for the j-th column (j from 0 to L-1), the average neuron activation value equals the sum of all N elements in that column divided by N. This yields an average neuron activation vector of length L, which is the raw layer representation vector of the network layer. The raw layer representation vector reflects the average activation state of the network layer under normal input conditions. It is stored as a binary file for later use when constructing the original inference path topology. During the calculation, to avoid numerical overflow, the activation value of each neuron is standardized, mapped to the range of 0-1. The standardization method is: subtract the minimum value of the neuron across all samples from each activation value, and then divide by the difference between the maximum and minimum values ​​of the neuron across all samples.

[0030] Step S116: Obtain the network structure connection information of the initial generative artificial intelligence model. The network structure connection information includes the sequential connection methods between each network layer in the model and the direction of data flow.

[0031] By parsing the structure file of the initial generative AI model (such as a PyTorch .pth model file), the torchsummary tool is used to obtain the network structure connection information. This network structure connection information is stored in the form of a directed graph, containing two parts: nodes and edges. Nodes represent the various network layers in the model, and each node contains information such as the name, type (e.g., input layer, encoder layer, decoder layer, output layer, etc.), input dimension, and output dimension of the network layer; edges represent the connection relationship between network layers, and each edge contains the source node (previous layer), the target node (next layer), and the data flow direction (from the source node to the target node). For example, the input layer node is named "input_layer", of type "Input", with an input dimension of "batch_size × seq_len" and an output dimension of "batch_size × seq_len × 512"; the first encoder layer node is named "encoder_layer_0", of type "EncoderLayer", with an input dimension of "batch_size × seq_len × 512" and an output dimension of "batch_size × seq_len × 512", its source node is "input_layer", and its target node is "encoder_layer_1". The network structure connection information also includes the connections between sub-layers within each network layer, such as the connections between the multi-head self-attention sub-layer and the feedforward neural network sub-layer in the encoder layer, ensuring a complete reflection of the model's internal structure.

[0032] Step S117: Based on the network structure connection information, using each network layer of the initial generative artificial intelligence model as a topology graph node, the original layer representation vector corresponding to each network layer as a node feature attribute, and the data flow direction between network layers as a directed connection edge, construct the original inference path topology graph of the initial generative artificial intelligence model for normal input.

[0033] Based on the acquired network structure connection information, the NetworkX library is used to construct the original inference path topology graph. The nodes of the topology graph are set as the network layers of the initial generative AI model, with each node's unique identifier being the name of the network layer. The node's feature attribute dictionary contains a "original layer representation vector" key, whose value is the original layer representation vector corresponding to that network layer. It also includes auxiliary information such as the network layer type, input dimension, and output dimension. The directed edges of the topology graph are set according to the data flow direction between network layers, pointing from the source network layer node to the target network layer node. The edge attributes contain dimensional information about the data flow, such as the shape of the input data and the shape of the output data. For example, the directed edge from the "input_layer" node to the "encoder_layer_0" node has "input_shape" as "batch_size × seq_len" and "output_shape" as "batch_size × seq_len × 512". During the construction process, it is ensured that the topology graph accurately reflects the model's forward propagation path, starting from the input layer, sequentially passing through each encoder layer, decoder layer, and finally reaching the output layer. The completed original inference path topology is stored in GraphML format for subsequent visualization and analysis.

[0034] Step S118: In the original inference path topology graph, each node carries the average activation mode information of the network layer under clean sample input, and the directed edges between nodes represent the computational dependency relationship of the output of the previous layer network as the input of the next layer network.

[0035] In the constructed original inference path topology, the "original layer representation vector" feature attribute of each node carries the average activation mode information of the corresponding network layer under clean sample input. This average activation mode information is reflected by the values ​​of each element in the original layer representation vector, with each element value corresponding to the average activation level of a neuron in the network layer. For example, the value of the k-th element in the original layer representation vector of the "encoder_layer_0" node is the average activation value of the k-th neuron in that encoder layer under all clean sample inputs. The directed edges between nodes clearly represent the computational dependency relationship where the output of the previous layer serves as the input of the next layer; that is, the computation of the next layer depends on the output of the previous layer. This computational dependency relationship is further clarified by the direction of the edges and the input / output shape information in the attributes. For example, the output of the "encoder_layer_0" node serves as the input of the "encoder_layer_1" node, and the directed edge between them indicates the above data transfer relationship. Furthermore, the "input_shape" and "output_shape" attributes of the edges ensure the matching of data dimensions. In this way, the original inference path topology diagram fully depicts the inference process of the model under normal input conditions and the average activation state of each network layer.

[0036] Step S120: Use a preset target perturbation generator to perform perturbation injection processing on the clean sample set to generate an adversarial sample test set covering multiple attack types.

[0037] The pre-defined target perturbation generator is a module integrating multiple attack algorithms, designed to inject perturbations into samples in a clean sample set, generating an adversarial sample test set capable of testing the robustness of the initial generative AI model. This generator supports multiple attack types and can simulate different perturbation injection methods to comprehensively evaluate the model's performance in the face of various malicious perturbations. Before performing perturbation injection, the samples in the clean sample set need to be preprocessed to ensure that the sample format and data range meet the input requirements of the target perturbation generator. For example, pixel values ​​of image data are converted to integers in the range of 0-255, and text data is converted to corresponding index sequences.

[0038] Step S121: Obtain the internal module configuration information of the preset target perturbation generator, which includes a gradient information attack module, a decision boundary attack module, a transformation-based attack module, and a score-based attack module.

[0039] The target perturbation generator comprises four main modules: a gradient information attack module, a decision boundary attack module, a transformation-based attack module, and a score-based attack module. Each module has independent configuration parameters and attack strategies, set through a configuration file. The gradient information attack module generates adversarial perturbations based on the model's gradient information; its configuration parameters include perturbation step size, maximum number of iterations, and loss function type. The decision boundary attack module generates adversarial examples by finding sample points near the model's decision boundary; its configuration parameters include binary search accuracy and initial perturbation range. The transformation-based attack module generates adversarial examples by performing spatial transformations or geometric distortions on samples; its configuration parameters include rotation angle range, scaling ratio range, and shearing factor. The score-based attack module generates adversarial examples by estimating the gradient direction of the model's output probability; its configuration parameters include sampling quantity and estimation step size. The internal module configuration information is stored in a JSON-formatted configuration file, containing the activation status, parameter values, and attack strength of each module. The target perturbation generator reads this configuration file and initializes each module upon startup.

[0040] Step S122: Each clean sample unit in the clean sample set is sequentially input into the gradient information attack module. The gradient information attack module calculates the gradient of the loss function of the initial generative artificial intelligence model relative to the clean sample unit, and adds perturbation to the clean sample unit in the gradient ascent direction to generate preliminary adversarial sample units based on gradient information.

[0041] Each clean sample unit in the clean sample set is sequentially input into the gradient information attack module. For clean sample units of image type, the gradient information attack module uses FGSM (Fast Gradient Sign Method) to generate adversarial perturbations. The specific process is as follows: First, the image sample is converted into a tensor in the model input format, and require_grad=True is set to enable gradient calculation; then, this tensor is input into the initial generative AI model to obtain the model's output; next, the loss function (such as cross-entropy loss) between the model output and the true label is calculated, and the derivative with respect to the input image tensor is taken to obtain the gradient of the loss function with respect to the input image; finally, the sign of the gradient is multiplied by a preset perturbation step size (such as 0.01) to obtain the adversarial perturbation, which is added to the original image tensor, and the result is restricted to the range of 0-1 by the clip function to generate the initial adversarial sample unit based on gradient information. For clean sample units of text type, a gradient-based word embedding perturbation method is used to calculate the gradient of the loss function with respect to the word embedding vector, and the word embedding vector is adjusted in the gradient ascent direction to generate adversarial text samples. The gradient information attack module also supports iterative attack algorithms, such as BIM (basic iterative method), which generates stronger adversarial examples by adding small step-size perturbations through multiple iterations. The number of iterations is set to 10, and the perturbation step size for each iteration is 0.001.

[0042] Step S123: Each clean sample unit in the clean sample set is sequentially input into the decision boundary attack module. The decision boundary attack module performs a binary search near the decision boundary of the initial generative artificial intelligence model to find the perturbation point that is closest to the clean sample unit and can cause the model to misclassify, thereby generating an initial adversarial sample unit based on the decision boundary.

[0043] Each clean sample unit in the clean sample set is sequentially input into the decision boundary attack module. The decision boundary attack module employs the C&W (Carlini & Wagner) attack algorithm, which finds the closest adversarial example to the clean sample unit by optimizing the L2 norm of the perturbation. Specifically, a target function is defined, which includes the cross-entropy loss of the model output and the L2 norm penalty term for the perturbation. Then, the Adam optimizer is used to minimize this target function, where the optimization variable is the perturbation vector. During optimization, the weights of the penalty term are adjusted through binary search to ensure that the generated adversarial example can successfully cause the model to misclassify while minimizing the L2 norm of the perturbation. For image samples, the dimension of the perturbation vector is the same as the number of pixels in the image; for example, a 256×256×3 image corresponds to a 196608-dimensional perturbation vector. For text samples, the perturbation vector acts on the word embedding space, and the perturbation is achieved by adjusting the word embedding vector. The binary search precision of the decision boundary attack module is set to 1e-4, and the maximum number of optimization iterations is set to 1000 to ensure that adversarial examples close to the decision boundary are found. The generated preliminary adversarial sample units based on decision boundaries have a small distance (L2 norm) from the original clean sample units, which makes them highly concealed.

[0044] Step S124: Each clean sample unit in the clean sample set is sequentially input into the transformation-based attack module. The transformation-based attack module performs spatial transformation or geometric distortion operations on the clean sample units to change the semantic structure of the samples while keeping human perception unchanged, thereby generating preliminary adversarial sample units based on transformation.

[0045] Each clean sample unit in the clean sample set is sequentially input into the transformation-based attack module. For image samples, the transformation-based attack module generates adversarial examples using various spatial transformation operations, including random rotation, scaling, shearing, translation, and flipping. Specific parameters are set as follows: rotation angle randomly selected between -15 and 15 degrees, scaling ratio randomly selected between 0.8 and 1.2, shearing factor randomly selected between -0.1 and 0.1, translation distance randomly selected between -5 and 5 pixels, and horizontal and vertical flipping probabilities of 0.5 each. These transformation operations are implemented using the OpenCV library, altering the local pixel distribution of the image while maintaining the overall visual effect, potentially leading to misclassification by the model. For text samples, perturbation is achieved using synonym replacement, word order adjustment, and insertion of irrelevant words. Synonym replacement uses the WordNet dictionary with a replacement ratio of 20%, word order adjustment uses adjacent word swapping, and the inserted irrelevant words are neutral words with low relevance to the text topic, ensuring that the perturbed text remains semantically coherent for human reading, but may alter the model's understanding. The initial adversarial sample units generated by the transformation-based attack module have strong transferability and can attack models with different structures.

[0046] Step S125: Each clean sample unit in the clean sample set is sequentially input into the score-based attack module. The score-based attack module estimates the gradient direction of the output probability of the initial generative artificial intelligence model by sampling, and adds perturbation to the clean sample unit in the estimated gradient direction to generate a preliminary adversarial sample unit based on score.

[0047] Each clean sample unit from the clean sample set is sequentially input into the score-based attack module. The score-based attack module is suitable for black-box models lacking gradient information, estimating the gradient direction of the model's output probability relative to the input samples through a sampling method. Specifically, for image samples, multiple random perturbation samples are first generated around the original sample, with a perturbation amplitude of a preset small value (e.g., 0.01), and the sampling quantity is set to 100. These perturbation samples are then input into the initial generative AI model to obtain the corresponding output probability scores. Next, a linear regression method is used to estimate the gradient direction based on the perturbation vector and the corresponding probability score changes. Finally, perturbations are added to the estimated gradient direction to generate preliminary score-based adversarial sample units. For text samples, a similar method is used, randomly sampling in the word embedding space to estimate the gradient direction. The sampling distribution of the score-based attack module adopts a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, ensuring the randomness and representativeness of the sampling perturbations. The generated adversarial sample units do not require the model's internal gradient information, making them suitable for scenarios where the model structure is unknown.

[0048] Step S126: Collect the preliminary adversarial sample units generated by the gradient information attack module, the decision boundary attack module, the transformation-based attack module, and the score-based attack module respectively, to form an original adversarial sample pool containing the generation results of multiple attack types.

[0049] Preliminary adversarial sample units generated by four attack modules are collected, each module corresponding to one attack type. Therefore, the original adversarial sample pool contains adversarial samples of four attack types. For each clean sample unit, the four attack modules generate one preliminary adversarial sample unit, so the number of samples in the original adversarial sample pool is four times the number of samples in the clean sample set. Each preliminary adversarial sample unit is stored in the same format as the original clean sample units, and the attack module type is indicated in the filename. For example, "sample_0001_fgsm.png" represents the adversarial sample image generated by the Gradient Information Attack Module (FGSM) for clean sample unit number 0001. The original adversarial sample pool is stored using a hierarchical directory structure. The root directory is divided into four subdirectories, each corresponding to one of the four attack types. Each subdirectory stores all preliminary adversarial sample units generated for that attack type, facilitating subsequent validity verification and categorized storage.

[0050] Step S127: Verify the attack effectiveness of the initial adversarial sample units in the original adversarial sample pool, filter out invalid adversarial sample units that cannot successfully attack the initial generative artificial intelligence model, and retain valid adversarial sample units that can cause the model to produce incorrect output.

[0051] The effectiveness of each initial adversarial sample unit in the original adversarial sample pool is verified. The verification process is as follows: the initial adversarial sample unit is input into the initial generative AI model to obtain the model's output; this output is compared with the ground truth label corresponding to the original clean sample unit. If the model's output does not match the ground truth label, the initial adversarial sample unit is considered a valid adversarial sample unit; otherwise, it is considered an invalid adversarial sample unit. For image generation models, the output verification uses image similarity evaluation methods, such as calculating the SSIM (Structural Similarity Index) between the generated image and the ground truth image. When the SSIM value is lower than a preset threshold (e.g., 0.5), it is considered an incorrect output. For text generation models, BLEU score or ROUGE score is used for evaluation; when the score is lower than a preset threshold, it is considered an incorrect output. The effectiveness verification results are recorded in a CSV log file, containing information such as sample number, attack type, and whether it is effective. After filtering out all invalid adversarial sample units, the number of remaining valid adversarial sample units is approximately 70%-80% of the original adversarial sample pool. The specific percentage depends on the effectiveness of the attack module and the robustness of the model.

[0052] Step S128: All the retained valid adversarial sample units are classified and stored according to the attack module type from which they originate, generating an adversarial sample test set covering multiple attack types.

[0053] The retained valid adversarial sample units are categorized and stored according to the attack module type from which they originate, into four subsets: gradient information attack, decision boundary attack, transformation-based attack, and score-based attack. Each subset contains all valid adversarial sample units for the corresponding attack type and is stored separately in the corresponding subdirectory of the adversarial sample test set. The root directory of the adversarial sample test set also contains a metadata file recording information such as the number of samples in each subset, attack type description, and average perturbation strength. The average perturbation strength is obtained by calculating the average L2 distance between each valid adversarial sample unit and the original clean sample units, and is used to evaluate the perturbation degree of different attack types. The generated adversarial sample test set can be used for subsequent robustness enhancement training and testing of the model, ensuring that the model can withstand various types of adversarial attacks.

[0054] Step S130: Input the adversarial example test set into the initial generative artificial intelligence model for forward propagation calculation, obtain the perturbation neuron activation pattern sequence generated by the initial generative artificial intelligence model during layer-by-layer forward propagation, and construct the perturbation inference path topology graph set of the initial generative artificial intelligence model for the perturbation input based on the perturbation neuron activation pattern sequence.

[0055] The constructed adversarial example test set is input into the initial generative AI model, and layer-by-layer forward propagation computation is performed to obtain the neuron activation patterns under perturbed input conditions. Similar to processing the clean sample set, an activation value capture hook function is set after each layer of the model to capture the activation tensor under perturbed conditions. By processing and analyzing the above activation tensor, perturbed inference path topology graphs are constructed for different attack types, forming a set of perturbed inference path topology graphs. Each topology graph in this set of perturbed inference path topology graphs corresponds to an attack type, reflecting the changes in the model's inference path and neuron activation patterns under that attack type.

[0056] Step S131: Select a subset of adversarial samples corresponding to the first attack type from the adversarial sample test set, and input multiple adversarial sample units from the subset of adversarial samples corresponding to the first attack type into the initial generative artificial intelligence model in sequence.

[0057] A subset of adversarial samples corresponding to the gradient information attack type is selected from the adversarial sample test set as the first attack type subset. This first attack type subset contains multiple valid adversarial sample units generated by the gradient information attack module. Each sample unit is image or text data and has been verified to cause the initial generative AI model to produce erroneous outputs. The adversarial sample units in this subset are input into the initial generative AI model in a random order. During the input process, the same batch size and preprocessing steps are maintained as when processing clean sample units to ensure the consistency of the model input. For example, image samples are also normalized, and text samples undergo word segmentation and word embedding transformation.

[0058] Step S132: After the input layer of the initial generative artificial intelligence model, perform layer-by-layer forward propagation computation, and after each network layer, capture the perturbation activation tensor of the adversarial example unit output of the current network layer for the current input through the activation value capture hook function.

[0059] Similar to the process for clean sample units, layer-by-layer forward propagation computation begins after the input layer of the initial generative AI model. For each input adversarial sample unit, after each network layer of the model (including the encoder layer, decoder layer, and sub-layers), the perturbed activation tensor output by the current network layer is captured using an activation value capture hook function. The implementation and storage strategy of the hook function are the same as when capturing the original activation tensor; that is, each network layer corresponds to an independent cache file, named in the format "network layer name_attack type_sample unit number_perturbed activation tensor.pt". The shape of the perturbed activation tensor is consistent with that of the original activation tensor, for example, batch size × sequence length × hidden layer dimension, but the element values ​​in the tensor reflect the neuron activation state of the model under perturbed input conditions.

[0060] Step S133: For each captured perturbation activation tensor, record the activation value corresponding to each neuron in the perturbation activation tensor to generate the perturbation neuron activation vector of the current network layer under the input of the current adversarial example unit.

[0061] For each captured perturbation activation tensor, it is expanded and recorded in the same manner as the original activation tensor. The activation values ​​of each neuron in the perturbation activation tensor are arranged in the order of batch dimension, sequence length dimension, and hidden layer dimension to form a one-dimensional perturbation neuron activation vector. For example, for a perturbation activation tensor with a batch size of 16, a sequence length of 30, and a hidden layer dimension of 512, the length of the perturbation neuron activation vector corresponding to each adversarial example unit is 30 × 512 = 15360. The generated perturbation neuron activation vectors are stored as text files in the same format as the original neuron activation vectors for subsequent summarization and comparison.

[0062] Step S134: Summarize the perturbation neuron activation vectors output by all adversarial sample units corresponding to the first attack type for the same network layer, calculate the average perturbation neuron activation vector of the network layer under the input of all adversarial sample units corresponding to the first attack type, and use the average perturbation neuron activation vector as the first perturbation layer representation vector of the network layer for the first attack type.

[0063] Collect the perturbation neuron activation vectors output by all adversarial sample units of the same network layer for the first attack type (gradient information attack). Assume this subset contains M adversarial sample units, and each vector has a length of L. Arrange these vectors in rows to form an M×L matrix. Calculate the average value for each column of the matrix to obtain the average perturbation neuron activation vector. The calculation method is the same as that for the original layer representation vector, i.e., summing the M elements of each column and dividing by M. Use this average perturbation neuron activation vector as the first perturbation layer representation vector for this network layer against the first attack type, store it as a binary file, and indicate the attack type in the filename, such as "encoder_layer_0_fgsm_perturbation_layer_vector.bin".

[0064] Step S135: Based on the network structure connection information, using each network layer of the initial generative artificial intelligence model as a topology graph node, the first perturbation layer representation vector corresponding to each network layer as a node feature attribute, and the data flow direction between network layers as a directed connection edge, construct the first perturbation inference path topology graph of the initial generative artificial intelligence model for the first type of attack.

[0065] Based on the network structure and connection information, a first perturbation inference path topology graph for the first attack type is constructed using the NetworkX library. The nodes of the topology graph are set as the various network layers of the model. The value of the "perturbation layer representation vector" key in the node's feature attributes is the first perturbation layer representation vector corresponding to that network layer. Other auxiliary information (such as network layer type, input / output dimensions) is the same as the original inference path topology graph. Directed connections between nodes are set according to the data flow direction between network layers, and the edge attributes are also the same as the original inference path topology graph. Figure 1 The completed first perturbation inference path topology graph is stored in GraphML format, with the filename containing attack type information, such as "fgsm_perturbation_inference_graph.graphml".

[0066] Step S136: Select adversarial sample subsets corresponding to the second attack type, the third attack type and so on up to the last attack type from the adversarial sample test set in sequence. For each adversarial sample subset corresponding to an attack type, repeat the above steps of forward propagation calculation, perturbation neuron activation vector recording, average perturbation neuron activation vector calculation and perturbation inference path topology graph construction.

[0067] From the adversarial example test set, select adversarial sample subsets corresponding to decision boundary attack type, transformation-based attack type, and score-based attack type sequentially as subsets for the second, third, and fourth attack types, respectively. For each subset of attack types, repeat steps S131 to S135: input the adversarial sample units in the subset into the model, capture the perturbation activation tensor, generate perturbation neuron activation vectors, calculate the average perturbation neuron activation vectors (referred to as the second, third, and fourth perturbation layer representation vectors, respectively), and construct the corresponding perturbation inference path topology graphs (referred to as the second, third, and fourth perturbation inference path topology graphs, respectively). The processing procedure and parameter settings for each attack type are consistent with those for the first attack type to ensure the comparability of results.

[0068] Step S137: Obtain the perturbation inference path topology map for each attack type, and summarize the perturbation inference path topology maps corresponding to all attack types to form the set of perturbation inference path topology maps for the perturbation input of the initial generative artificial intelligence model.

[0069] The four perturbation inference path topology graphs constructed for the four attack types are summarized to form a perturbation inference path topology graph set. This set of perturbation inference path topology graphs is stored in dictionary form, with the key being the attack type name (such as "fgsm", "cw", "transform", "score") and the value being the corresponding perturbation inference path topology graph object. The set also includes a metadata file recording the sample size for each attack type and statistical information (such as mean and standard deviation) of the average perturbation layer representation vector, for subsequent layer-by-layer node activation difference comparison analysis.

[0070] Step S140: Perform a layer-by-layer node activation difference comparison analysis between the original inference path topology graph and each perturbation inference path topology graph in the set of perturbation inference path topology graphs, and identify the key network layer nodes in the initial generative artificial intelligence model whose activation modes have significantly shifted under perturbation input, as well as the activation offset direction vectors corresponding to the key network layer nodes. The significant shift is determined by comparing the total cross-attack type offset of the network layer nodes with a preset offset threshold.

[0071] By comparing the node characteristic attributes of each topology graph in the original inference path topology graph set and the perturbed inference path topology graph set, the changes in the activation patterns of nodes in each network layer under different attack types are analyzed. Activation offsets and offset direction vectors are calculated to identify key network layer nodes where the activation patterns have significantly shifted.

[0072] Step S141: Extract the original layer representation vector corresponding to each network layer node from the original inference path topology graph as the first comparison benchmark.

[0073] Traverse all network layer nodes in the original inference path topology graph, and extract the "original layer representation vector" from the feature attributes of each node as the first comparison benchmark. The length of the original layer representation vector is L (e.g., 15360), reflecting the average activation mode of the network layer under clean sample input. Store the extracted original layer representation vectors in a dictionary, with the key being the network layer node name and the value being the corresponding original layer representation vector, for subsequent comparison with the perturbation layer representation vector.

[0074] Step S142: Sequentially obtain each perturbation inference path topology graph from the set of perturbation inference path topology graphs, and extract the perturbation layer representation vector corresponding to each network layer node from the currently obtained perturbation inference path topology graph as the second comparison object.

[0075] The perturbation inference path topology graphs for each attack type are sequentially obtained from the set of perturbation inference path topology graphs. All network layer nodes in each topology graph are traversed, and the "perturbation layer representation vector" from the node feature attributes is extracted as the second comparison object. Similar to the original layer representation vector, the length of the perturbation layer representation vector is also L, reflecting the average activation mode of the network layer under a specific attack type. The extracted perturbation layer representation vectors are categorized and stored according to attack type and network layer node name, for example, by constructing a three-dimensional dictionary where the first dimension is the attack type, the second dimension is the network layer node name, and the third dimension is the perturbation layer representation vector.

[0076] Step S143: For each network layer node, calculate the element-wise difference between the original layer representation vector and the perturbation layer representation vector to generate the activation offset vector of the network layer node under the current attack type.

[0077] For each network layer node and each attack type, the element-wise difference between the original layer representation vector and the perturbation layer representation vector corresponding to that attack type is calculated. Specifically, the calculation process is as follows: Let the original layer representation vector be V_original=[v1,v2,...,vL], and the perturbation layer representation vector be V_perturb=[u1,u2,...,uL], then the activation offset vector D=[u1-v1,u2-v2,...,uL-vL]. Each element in the activation offset vector D reflects the change in activation of the corresponding neuron under the perturbation input; positive values ​​indicate enhanced activation, and negative values ​​indicate weakened activation. The generated activation offset vectors are stored according to network layer nodes and attack types for subsequent analysis.

[0078] Step S144: Based on the value of each element in the activation offset vector, determine the activation change magnitude and direction of each neuron in the network layer node under the current attack type.

[0079] Analyze each element in the activation offset vector D. The absolute value of each element represents the magnitude of the change in neuron activation, and the sign indicates the direction of change (positive for enhancement, negative for reduction). For each network layer node, calculate the maximum, minimum, mean, and standard deviation of the elements in the activation offset vector to assess the overall activation change. Simultaneously, record the locations of neurons with large changes (e.g., neurons whose absolute values ​​are greater than the mean of all absolute values ​​plus twice the standard deviation), as these neurons may play a crucial role in the model's adversarial vulnerability.

[0080] Step S145: Sum the squares of all elements in the activation offset vector to obtain the overall activation offset of the network layer node under the current attack type.

[0081] The sum of squares of all elements in the activation offset vector D is calculated, i.e., the comprehensive activation offset S = sum(di^2 for di in D), where di is the i-th element of the activation offset vector D. The comprehensive activation offset S reflects the overall activation offset degree of the network layer node under the current attack type. The larger the value of S, the greater the impact of the disturbance on the activation mode of the network layer node. The calculated comprehensive activation offset is recorded according to the network layer node and the attack type.

[0082] Step S146: Sum the combined activation offsets of the same network layer node under different attack types to obtain the total cross-attack type offset of the network layer node.

[0083] For each network layer node, its comprehensive activation offsets under the four attack types are summed to obtain the total cross-attack type offset T = S1 + S2 + S3 + S4, where S1 to S4 are the comprehensive activation offsets of the node under the four attack types, respectively. The total cross-attack type offset T reflects the overall vulnerability of the network layer node under multiple attack types. The larger the T value, the more sensitive the node is to different types of adversarial attacks.

[0084] Step S147: Sort all network layer nodes according to the total offset across attack types, and select network layer nodes whose total offset across attack types exceeds a preset offset threshold as the set of candidate key network layer nodes whose activation mode has significantly shifted.

[0085] All network layer nodes are sorted from largest to smallest based on their total cross-attack type offset T. A preset offset threshold is determined by statistically analyzing the distribution of the total cross-attack type offsets of all network layer nodes; for example, the mean of all T values ​​plus twice the standard deviation can be used as the threshold. Network layer nodes whose total cross-attack type offset T exceeds this threshold are selected to form a candidate set of key network layer nodes. Nodes in the candidate set are typically layers in the model that are highly sensitive to adversarial perturbations, such as intermediate layers of the encoder or attention layers of the decoder.

[0086] Step S148: For each candidate key network layer node in the candidate key network layer node set, perform vector superposition on its activation offset vectors under different attack types to generate a synthetic comprehensive activation offset direction vector.

[0087] For each node in the candidate key network layer node set, its activation offset vectors D1, D2, D3, and D4 under the four attack types are vector-superimposed to obtain the comprehensive activation offset direction vector D_total = D1 + D2 + D3 + D4. The vector superposition process is element-wise addition, that is, the i-th element of D_total is equal to the sum of the i-th elements of D1, D2, D3, and D4. The comprehensive activation offset direction vector D_total reflects the overall activation offset trend of the node under multiple attack types.

[0088] Step S149: Normalize the comprehensive activation offset direction vector to obtain the activation offset direction vector corresponding to the key network layer node. The activation offset direction vector indicates the overall trend direction of the internal neuron activation mode of the key network layer node deviating from the original state when facing multiple attacks.

[0089] The comprehensive activation offset direction vector D_total is normalized by calculating its L2 norm, i.e., norm = sqrt(sum(di^2 for di in D_total)). Then, each element in D_total is divided by norm to obtain the normalized activation offset direction vector D_unit = D_total / norm. The normalized vector has a length of 1 and only represents the direction, without containing amplitude information. The activation offset direction vector D_unit indicates the overall trend direction of the internal neuron activation patterns deviating from the original state when facing various attacks in this key network layer node.

[0090] Step S150: Construct a dynamic path deflection unit for a specific network layer node based on the key network layer node and the activation offset direction vector. The dynamic path deflection unit includes an activation direction modulation sublayer and a residual path selection sublayer that can be embedded in the output of the key network layer node.

[0091] For the identified key network layer nodes, a dynamic path deflection unit is designed and constructed based on their activation offset direction vectors. This dynamic path deflection unit includes an activation direction modulation sublayer and a residual path selection sublayer, which can modulate the output of the key network layer nodes to resist activation mode shifts caused by adversarial perturbations.

[0092] Step S151: Obtain the activation offset direction vector corresponding to the key network layer node, and use the activation offset direction vector as the core modulation reference for constructing the activation direction modulation sublayer.

[0093] The activation offset direction vector D_unit is obtained from the attributes of key network layer nodes. This activation offset direction vector serves as the core modulation reference for constructing the activation direction modulation sublayer. The design goal of the activation direction modulation sublayer is to reduce or cancel the effects of perturbations in the activation offset direction; therefore, the direction of D_unit is a key reference for modulation.

[0094] Step S152: Construct the activation direction modulation sub-layer. The activation direction modulation sub-layer contains a learnable modulation intensity parameter vector and a fixed projection matrix. Each row of the projection matrix is ​​orthogonal to the activation offset direction vector, which is used to decompose the input feature vector into components along the offset direction and components perpendicular to the offset direction.

[0095] The structure of the activation direction modulation sublayer is as follows: First, a fixed projection matrix P is constructed, with the number of rows equal to the hidden layer dimension L and the number of columns L-1. Each row of the projection matrix P is orthogonal to the activation offset direction vector D_unit, meaning the dot product between each row vector of P and D_unit is 0. L-1 basis vectors orthogonal to D_unit are generated from the standard orthogonal basis using the Gram-Schmidt orthogonalization method, forming the projection matrix P. Second, a learnable modulation intensity parameter vector α is set, with dimension L, each element initialized to 0.5, and a value ranging from 0 to 1. The input to the activation direction modulation sublayer is the original node activation vector X output from the key network layer nodes, and the output is the pre-modulated activation vector Y.

[0096] Step S153: Connect the input of the activation direction modulation sublayer to the output of the key network layer node, and the activation direction modulation sublayer receives the original node activation vector output by the key network layer node as input.

[0097] The input of the activation direction modulation sublayer is directly connected to the output of the key network layer node, allowing the original node activation vector X output by the key network layer node to be directly input into the activation direction modulation sublayer for processing. The dimension of the original node activation vector X is L (e.g., 15360), matching the input dimension of the activation direction modulation sublayer.

[0098] Step S154: Inside the activation direction modulation sublayer, the original node activation vector is first decomposed into a parallel offset component and a vertical offset component using the projection matrix. Then, the learnable modulation intensity parameter vector is multiplied element-wise with the parallel offset component to obtain the modulated parallel offset component. Finally, the modulated parallel offset component is vector-added with the original vertical offset component to synthesize the pre-modulated activation vector.

[0099] The processing of the activation direction modulation sublayer is as follows: First, the projection of the original node activation vector X onto the activation offset direction vector D_unit is calculated, resulting in the parallel offset component X_parallel = (X·D_unit)*D_unit, where "·" represents the dot product operation. Then, the projection of X onto the orthogonal space spanned by the projection matrix P is calculated, resulting in the vertical offset component X_orthogonal = P*(P^T*X), where P^T represents the transpose of the projection matrix P. Next, the learnable modulation intensity parameter vector α is multiplied element-wise with the parallel offset component X_parallel, resulting in the modulated parallel offset component X_parallel_modulated = α*X_parallel. Finally, the modulated parallel offset component is added to the vertical offset component to obtain the initially modulated activation vector Y = X_parallel_modulated + X_orthogonal. Through this method, the component along the activation offset direction is scaled by the modulation intensity parameter vector α, thereby reducing the activation offset caused by adversarial perturbations.

[0100] Step S155: Construct the residual path selection sublayer, which contains a learnable path selection gating parameter and a residual connection unit.

[0101] The residual path selection sublayer contains a learnable path selection gating parameter γ and a residual connection unit. The path selection gating parameter γ is a scalar, initialized to 0.5, and its value ranges from 0 to 1. The sigmoid function ensures that its value is between 0 and 1. The residual connection unit directly introduces the original node activation vector X output by the key network layer nodes into the residual path selection sublayer.

[0102] Step S156: Connect the input of the residual path selection sublayer to the output of the activation direction modulation sublayer to receive the pre-modulated activation vector as input. At the same time, directly connect the residual input of the residual path selection sublayer to the output of the key network layer node to receive the original node activation vector as another input.

[0103] The input of the residual path selection sublayer is connected to the output of the activation direction modulation sublayer, receiving the pre-modulated activation vector Y; the residual input is directly connected to the output of the key network layer node, receiving the original node activation vector X. Thus, the residual path selection sublayer has two inputs: the modulated vector Y and the original vector X.

[0104] Step S157: Inside the residual path selection sub-layer, a fusion coefficient between zero and one is calculated using the learnable path selection gating parameter. Based on the fusion coefficient, the pre-modulated activation vector is weighted and fused with the original node activation vector to generate the final modulated output activation vector.

[0105] The processing of the residual path selection sublayer is as follows: First, the path selection gating parameter γ is used to calculate the fusion coefficient g = sigmoid(γ) using the sigmoid function, ensuring that the value of g is between 0 and 1. Then, the pre-modulated activation vector Y is weighted and fused with the original node activation vector X, with the fusion formula Z = g*Y + (1-g)*X, where Z is the final modulated output activation vector. When g is close to 1, the modulated vector Y is mainly output; when g is close to 0, the original vector X is mainly output. Through the learnable path selection gating parameter γ, the model can adaptively adjust the modulation intensity, enhancing adversarial robustness while maintaining model performance.

[0106] Step S158: The activation direction modulation sublayer and the residual path selection sublayer are encapsulated and integrated to form a dynamic path deflection unit for the key network layer node. The input interface of the dynamic path deflection unit receives the original output from the key network layer node, and the output interface of the dynamic path deflection unit outputs the new activation vector after direction modulation and path selection.

[0107] The activation direction modulation sublayer and the residual path selection sublayer are encapsulated according to their input-output relationship to form a dynamic path deflection unit. The input interface of this dynamic path deflection unit receives the raw output X from the key network layer nodes, processes it through the activation direction modulation sublayer to obtain Y, and then processes it through the residual path selection sublayer to obtain the final output Z. As an independent module, the dynamic path deflection unit can be embedded into the output of the key network layer nodes of the model to modulate the activation vector. The learnable parameters within the unit (modulation intensity parameter vector α and path selection gating parameter γ) are adjusted during subsequent joint optimization training.

[0108] Step S160: Embed the dynamic path deflection unit into the output of the corresponding key network layer node in the initial generative artificial intelligence model to construct an intermediate robust enhancement model containing an activation direction modulation mechanism.

[0109] By modifying the network structure of the initial generative AI model, dynamic path deflection units are embedded into the outputs of key network layer nodes, forming an intermediate robustness enhancement model. This intermediate robustness enhancement model, while retaining the original model structure, adds a dynamic path deflection mechanism, which can modulate the activation vectors of key network layers to enhance the defense against adversarial examples.

[0110] Step S161: parse the original computation graph of the initial generative artificial intelligence model. The original computation graph contains a complete data flow path from the input layer through each network layer node to the output layer. Each network layer node has a clear input connection source node and an output connection target node.

[0111] The raw computation graph of the initial generative AI model is analyzed using model visualization tools (such as Netron) to obtain the connection relationships between nodes in each network layer. The raw computation graph shows the flow path of data from the input layer to the output layer in the form of a directed graph, with each network layer node clearly showing the source node for its input and the target node for its output. For example, the output of the input layer connects to the input of the first encoder layer, the output of the first encoder layer connects to the input of the second encoder layer, and so on. The analysis results are stored as a JSON file, containing information such as the name, type, list of input nodes, and list of output nodes for each node.

[0112] Step S162: Based on the key network layer nodes and their specific positions in the original computation graph, determine the output connection target node list for each key network layer node in the original computation graph.

[0113] Based on the parsed original computation graph information, the position of each key network layer node in the computation graph is located, and its output connection target node list is extracted. For example, if the key network layer node is "encoder_layer_5", its output connection target nodes might be "encoder_layer_6" and "decoder_layer_3" (assuming the decoder layer needs the encoder layer's output as input). The output connection target node list records which subsequent network layer nodes the output data of this key network layer node flows to.

[0114] Step S163: For each critical network layer node, disconnect the direct connection between the original output of the critical network layer node and the output connection target node list in the original computation graph.

[0115] In the original computation graph, by modifying the connection relationships of nodes, the direct connections between key network layer nodes and each node in the output connection target node list are disconnected. For example, for the "encoder_layer_5" node, its direct input connections with the "encoder_layer_6" and "decoder_layer_3" nodes are disconnected, so that these target nodes no longer directly receive the output of "encoder_layer_5".

[0116] Step S164: Connect the input of the dynamic path deflection unit to the original output of the key network layer node, and receive the original activation vector output by the key network layer node.

[0117] For each key network layer node, a corresponding dynamic path deflection unit instance is created, and the input of this unit is connected to the original output of the key network layer node. In this way, the original activation vector X output by the key network layer node will first be input into the dynamic path deflection unit for processing.

[0118] Step S165: Connect the output of the dynamic path deflection unit to the input of each target node in the output connection target node list, which should originally receive the original output of the key network layer node.

[0119] The output of the dynamic path deflection unit is connected to the input of each target node in the output connection target node list. For example, the output of the dynamic path deflection unit corresponding to "encoder_layer_5" is connected to the inputs of "encoder_layer_6" and "decoder_layer_3", so that the modulated activation vector Z can replace the original activation vector X as input to subsequent network layer nodes.

[0120] Step S166: Through the above connection operation, the dynamic path deflection unit is inserted as an intermediary module between the key network layer node and its subsequent network layer nodes, so that the output of the key network layer node must first be modulated by the dynamic path deflection unit before it can be transmitted to the subsequent network layer.

[0121] Through the connection operations in steps S163 to S165, the dynamic path deflection unit is inserted between the key network layer node and its subsequent network layer nodes, forming a new data flow path of "key network layer node → dynamic path deflection unit → subsequent network layer node". The output of the key network layer node is no longer directly passed to the subsequent network layer, but must be modulated by the dynamic path deflection unit to achieve dynamic adjustment of the activation vector.

[0122] Step S167: Repeat the operation of disconnecting the original connection and inserting dynamic path deflection units for all identified key network layer nodes, update the original computation graph, and obtain a new computation graph containing multiple dynamic path deflection unit nodes.

[0123] Repeat steps S163 to S166 for all nodes in the candidate critical network layer node set, inserting dynamic path deflection units at the output of each critical network layer node. After completing all insertion operations, update the original computation graph to obtain a new computation graph containing multiple dynamic path deflection unit nodes. In the new computation graph, dynamic path deflection units appear as new nodes between critical network layer nodes and their subsequent network layer nodes.

[0124] Step S168: Reconstruct the network structure of the initial generative artificial intelligence model according to the new computation graph, and use the dynamic path deflection unit as a newly added trainable module of the model to form an overall network together with the original network layer nodes, thereby constructing an intermediate state robust enhancement model containing activation direction modulation mechanism.

[0125] Based on the new computational graph, the network structure of the initial generative AI model is reconstructed using a deep learning framework (such as PyTorch). The dynamic path deflection unit is implemented as a custom PyTorch module and added to the appropriate location in the model. The learnable parameters of the dynamic path deflection unit (modulation intensity parameter vector α and path selection gating parameter γ) are registered as model parameters and participate in training along with the weight parameters of the original network layer nodes. The reconstructed model is the intermediate-state robust enhancement model, which incorporates the activation direction modulation mechanism.

[0126] Step S170: Use the adversarial example test set to jointly optimize and train the intermediate state robustness enhancement model, and simultaneously adjust the internal modulation parameters of the dynamic path deflection unit and the original weight parameters of the initial generative artificial intelligence model to generate the target generative artificial intelligence model.

[0127] The intermediate state robustness enhancement model is trained using an adversarial example test set. The internal modulation parameters of the dynamic path deflection unit and the original weight parameters of the model are simultaneously optimized through the backpropagation algorithm to improve the model's defense against adversarial examples and generate a target generative artificial intelligence model.

[0128] Step S171: Set the intermediate robustness enhancement model to the training state, activate the updatable properties of the original weight parameters of all original network layer nodes in the model, and at the same time activate the updatable properties of the internal modulation parameters of all dynamic path deflection units. The internal modulation parameters include the modulation intensity parameter vector in the activation direction modulation sublayer and the path selection gating parameter in the residual path selection sublayer.

[0129] Before training begins, the intermediate robustness enhancement model is set to training mode (model.train()). By setting the requires_grad attribute of the parameter to True, the updatable properties of the original weight parameters (such as attention weights in the encoder and decoder layers, feedforward neural network weights, etc.) and the internal modulation parameters (modulation intensity parameter vector α and path selection gating parameter γ) of all original network layer nodes are activated, ensuring that these parameters can be updated through backpropagation during training.

[0130] Step S172: Randomly select an adversarial sample unit from the adversarial sample test set and input the adversarial sample unit from the training batch into the intermediate state robustness enhancement model.

[0131] A batch of adversarial sample units is randomly selected from the adversarial sample test set, with a batch size of 32. The adversarial sample units contain samples of four attack types, ensuring a roughly balanced proportion of samples of each attack type during the sampling process. After preprocessing the selected adversarial sample units (such as normalization and word segmentation), they are input into the intermediate robustness enhancement model.

[0132] Step S173: Perform forward propagation calculation in the intermediate state robustness enhancement model. When the data flows through each node embedded with a key network layer, the key network layer node first calculates its original output activation vector, and then sends the original output activation vector to the dynamic path deflection unit connected to its output end.

[0133] The intermediate-state robustness enhancement model performs forward propagation computation on the input adversarial example units. When data flows through key network layer nodes, it first outputs the original activation vector X according to the computation method of the original model, and then sends X to the dynamic path deflection unit connected to its output. The dynamic path deflection unit modulates X and outputs the modulated activation vector Z, which continues to be passed to subsequent network layer nodes. The computation process of non-key network layer nodes is the same as the original model, and their outputs are directly passed to subsequent network layers.

[0134] Step S174: Inside the dynamic path deflection unit, the original output activation vector is sequentially processed by the modulation of the activation direction modulation sublayer and the fusion of the residual path selection sublayer to generate the final modulated output activation vector, and the final modulated output activation vector is then passed to subsequent network layers.

[0135] Inside the dynamic path deflection unit, the original output activation vector X first enters the activation direction modulation sublayer. After decomposition, modulation, and parallel / vertical component synthesis, a preliminary modulated activation vector Y is obtained. Y then enters the residual path selection sublayer, where it is weighted and fused with the original activation vector X to obtain the final modulated output activation vector Z. Z is then passed to subsequent network layer nodes of the key network layer nodes to continue participating in the forward propagation calculation.

[0136] Step S175: After processing by all network layer nodes and all dynamic path deflection units, obtain the model prediction output results corresponding to the adversarial example units of the current training batch from the output layer of the intermediate robustness enhancement model.

[0137] After the forward propagation computation is completed, the model's prediction output for the adversarial example units of the current training batch is obtained from the output layer of the intermediate robustness enhancement model. For the image generation model, the output is the generated image tensor; for the text generation model, the output is the generated text sequence or probability distribution.

[0138] Step S176: Obtain the real label information corresponding to the adversarial example units of the current training batch, and calculate the classification loss value of the current training batch based on the model prediction output and the real label information.

[0139] Obtain the ground truth labels for the adversarial example units in the current training batch. The ground truth labels are the labels of the original clean sample units (such as the class labels of images or the reference sequence of text). Calculate the classification loss based on the model's predicted output and the ground truth labels. For classification tasks, the cross-entropy loss function is used; for generation tasks, the generator loss function or mean squared error loss function from a Generative Adversarial Network (GAN) is used. The loss value is averaged over the batch dimension to obtain the average loss for the current training batch.

[0140] Step S177: Based on the classification loss value, calculate the gradient of the classification loss value relative to all updatable parameters in the intermediate robustness enhancement model using the backpropagation algorithm. The updatable parameters include the original weight parameters of all original network layer nodes and the internal modulation parameters of all dynamic path deflection units.

[0141] Using PyTorch's automatic differentiation mechanism, backpropagation is performed based on the classification loss value to obtain the gradient of the loss function with respect to all updatable parameters in the model (original weight parameters and internal modulation parameters of the dynamic path deflection unit). The gradient calculation starts from the output layer and propagates back along the network to the input layer, covering all network layer nodes and dynamic path deflection units involved in the calculation.

[0142] Step S178: Based on the calculated gradient value, and in conjunction with the preset gradient descent optimizer, all updatable parameters of the intermediate robustness enhancement model are updated synchronously to reduce the classification loss value.

[0143] The Adam optimizer is used to update the model parameters. The initial learning rate of the optimizer is set to 0.0001, the β1 parameter to 0.9, the β2 parameter to 0.999, and the weight decay coefficient to 0.0001. Based on the calculated gradient values, the optimizer updates all updatable parameters in the gradient descent direction, using the formula: parameter = parameter - learning rate × gradient. By updating the parameters, the model's classification loss on adversarial examples is reduced, improving the model's adversarial robustness.

[0144] Step S179: Repeat the steps of training batch extraction, forward propagation calculation, loss calculation, gradient backpropagation and parameter update until the preset total number of joint optimization training rounds is reached. Then, solidify the final model parameters to generate the target generative artificial intelligence model.

[0145] Repeat steps S172 to S178 of the training process, with a preset total of 100 rounds for joint optimization training. During training, model validation is performed every 5 rounds, using a validation set to evaluate model performance. If the validation loss does not decrease for 10 consecutive rounds, training is stopped early. After training, the final model parameters (including the original weight parameters and the internal modulation parameters of the dynamic path deflection unit) are solidified and saved as a model file to generate the target generative artificial intelligence model.

[0146] For example, the method may also include:

[0147] Step S180: Perform cross-layer path consistency constraint reinforcement processing on the target generative artificial intelligence model to generate a final generative artificial intelligence model with global collaborative defense capabilities.

[0148] To further enhance the global collaborative defense capability of the target generative artificial intelligence model, cross-layer path consistency constraint reinforcement processing is applied. By adding an additional loss function, the modulation output features between different key network layer nodes are made to remain consistent while moving closer to the original path.

[0149] Step S181: Obtain the dynamic path deflection unit corresponding to each of the key network layer nodes in the target generative artificial intelligence model, and extract the final modulated output activation vector of each dynamic path deflection unit under the current input sample as the current layer modulation output feature of the key network layer node.

[0150] For a target generative artificial intelligence model, a batch of validation samples (including clean samples and adversarial samples) is input. During the forward propagation of the model, the final modulated output activation vector Z of the dynamic path deflection unit corresponding to each key network layer node is extracted and used as the current layer modulation output feature of that key network layer node. The above features reflect the activation state of each key layer after modulation processing.

[0151] Step S182: Extract the original layer representation vector corresponding to each key network layer node from the original inference path topology graph as the original path reference benchmark for the key network layer node, and extract the average perturbation layer representation vector corresponding to each key network layer node under multiple attack types from the perturbation inference path topology graph set as the perturbation path comparison benchmark for the key network layer node.

[0152] The original layer representation vector V_original of each key network layer node is extracted from the original inference path topology graph and used as the original path reference benchmark. From the perturbation inference path topology graph set, the perturbation layer representation vector of each key network layer node under the four attack types is averaged to obtain the average perturbation layer representation vector V_perturb_avg, which is used as the perturbation path comparison benchmark.

[0153] Step S183: For each key network layer node, calculate the first feature similarity between the current layer modulation output feature and the original path reference benchmark, and simultaneously calculate the second feature similarity between the current layer modulation output feature and the perturbation path comparison benchmark.

[0154] The cosine similarity is used to calculate the similarity of the first and second features. The first feature similarity is S1 = cosine_similarity(Z, V_original), and the second feature similarity is S2 = cosine_similarity(Z, V_perturb_avg). The cosine similarity value ranges from -1 to 1; a larger value indicates greater similarity between the two vectors.

[0155] Step S184: Construct a local path preservation loss term for the key network layer node based on the first feature similarity and the second feature similarity. The local path preservation loss term increases when the first feature similarity is lower than a preset first similarity threshold and increases when the second feature similarity is higher than a preset second similarity threshold.

[0156] The local path preservation loss term L_local is calculated as follows: L_local = max(0, first_threshold - S1) + max(0, S2 - second_threshold), where first_threshold is the first similarity threshold (e.g., 0.8) and second_threshold is the second similarity threshold (e.g., 0.3). When S1 is lower than first_threshold, the first term increases; when S2 is higher than second_threshold, the second term increases, thereby ensuring that the current layer modulation output feature Z is similar to the original path reference benchmark V_original, but dissimilar to the perturbation path comparison benchmark V_perturb_avg.

[0157] Step S185: Obtain the pre- and post-order connections between all key network layer nodes in the target generative artificial intelligence model, construct a key network layer node sequence based on the pre- and post-order connections, and arrange the key network layer nodes in the order of data flow from front to back in the key network layer node sequence.

[0158] Based on the network structure of the target generative artificial intelligence model, determine the sequential connections between key network layer nodes, such as "encoder_layer_2→encoder_layer_5→decoder_layer_3→decoder_layer_7". Arrange the key network layer nodes into a sequence according to the order of data flow from front to back, such as [encoder_layer_2,encoder_layer_5,decoder_layer_3,decoder_layer_7].

[0159] Step S186: For any two adjacent key network layer nodes in the key network layer node sequence, firstly, the current layer modulation output features of the previous key network layer node and the current layer modulation output features of the next key network layer node are processed to unify the feature dimensions, and then the cross-layer feature transfer consistency parameter between the two feature vectors after unification is calculated.

[0160] For two adjacent key network layer nodes (such as encoder_layer_5 and decoder_layer_3), if their current layer modulation output feature dimensions are different (e.g., 15360 and 20480 respectively), a linear transformation is used to convert the dimension of the previous layer feature to the same dimension as the next layer feature. Then, the cosine similarity between the transformed previous layer feature and the next layer feature is calculated and used as the cross-layer feature propagation consistency parameter C.

[0161] Step S187: Construct a path coherence constraint loss term between adjacent nodes based on the cross-layer feature transmission consistency parameter. The path coherence constraint loss term increases when the cross-layer feature transmission consistency parameter is lower than a preset consistency threshold.

[0162] The path coherence constraint loss term L_coherence = max(0, coherence_threshold - C), where coherence_threshold is the consistency threshold (e.g., 0.6). When the cross-layer feature transmission consistency parameter C is lower than coherence_threshold, L_coherence increases, prompting the modulation output features of adjacent key network layer nodes to maintain coherent transmission.

[0163] Step S188: Accumulate the path coherence constraint loss terms corresponding to all adjacent nodes in the key network layer node sequence to generate the global path coherence total loss value.

[0164] The path coherence constraint loss term L_coherence for all adjacent node pairs in the key network layer node sequence is summed to obtain the global path coherence total loss value L_total_coherence=sum(L_coherenceforalladjacentpairs).

[0165] Step S189: Accumulate the local path preservation loss terms of each key network layer node to generate the total global path preservation loss value.

[0166] The local path preservation loss L_local of all key network layer nodes is summed to obtain the total global path preservation loss L_total_local = sum(L_localforallkeynodes).

[0167] Step S1810: The total loss value of global path coherence and the total loss value of global path maintenance are weighted and summed to generate the total loss function of cross-layer path consistency constraints.

[0168] The total loss function for cross-layer path consistency constraints is L_consistency = w1 * L_total_coherence + w2 * L_total_local, where w1 and w2 are weight coefficients, set to 0.5 and 0.5 respectively, to balance the importance of global path coherence and global path preservation.

[0169] Step S1811: Use the adversarial example test set to perform a second round of joint optimization training on the target generative artificial intelligence model. On the basis of the original classification loss function, add the cross-layer path consistency constraint total loss function, and synchronously update the internal modulation parameters of all dynamic path deflection units and the original weight parameters of all original network layer nodes.

[0170] The target generative AI model is subjected to a second round of joint optimization training using an adversarial example test set. The training process is similar to step S170, but the loss function is a weighted sum of the original classification loss function L_classification and the total loss function L_consistency for cross-layer path consistency constraints, i.e., total loss L = L_classification + 0.3 * L_consistency, where 0.3 is the weight coefficient of the constraint loss. The internal modulation parameters of all dynamic path deflection units and the original weight parameters of the original network layer nodes are updated synchronously through backpropagation.

[0171] Step S1812: During the second round of joint optimization training, by minimizing the total loss function of the cross-layer path consistency constraint, the modulation output features between adjacent key network layer nodes are made to maintain coherent transmission, while the modulation output features of each key network layer node are made to move closer to the original path reference benchmark and further away from the perturbation path comparison benchmark.

[0172] In the second round of training, by minimizing the total loss L, the model not only reduces the classification loss on adversarial examples, but also maintains a high degree of consistency in the modulation output features of adjacent key network layer nodes through the constraint of L_consistency. At the same time, the modulation output features of each key network layer node are closer to the original path reference benchmark and farther away from the perturbation path comparison benchmark, thereby enhancing the model's global collaborative defense capability.

[0173] Step S1813: After completing the second round of joint optimization training, the final generative artificial intelligence model with cross-layer path consistency constraint reinforcement processing is obtained.

[0174] The second round of joint optimization training consists of 50 rounds. After training, the model parameters are saved to obtain the final generative AI model. This final generative AI model has stronger adversarial example defense capabilities and global path consistency.

[0175] Step S190: Perform adaptive sparse pruning processing of the dynamic path deflection unit on the final generative artificial intelligence model to generate a lightweight deployment target generative artificial intelligence model.

[0176] To reduce the computational complexity and number of parameters of the final generative AI model and facilitate practical deployment, adaptive sparse pruning is performed on the dynamic path deflection units to remove dynamic path deflection units that contribute little to the model performance.

[0177] Step S191: Obtain the real-time response data generated by all dynamic path deflection units in the final generative artificial intelligence model under the current validation set input. The real-time response data includes the specific values ​​of the learnable path selection gating parameters in the residual path selection sublayer within each dynamic path deflection unit.

[0178] A pre-defined validation set is input into the final generative AI model. During the model's forward propagation, the specific values ​​of the path selection gating parameter γ in the residual path selection sublayer within each dynamic path deflection unit are recorded. The validation set includes clean samples and adversarial samples with various attack types to comprehensively evaluate the response of the dynamic path deflection unit.

[0179] Step S192: Calculate the actual contribution score of each dynamic path deflection unit under the current input sample based on the specific value of the path selection gating parameter. The actual contribution score is equal to the average value of the path selection gating parameter on multiple input samples.

[0180] For each dynamic path deflection unit, the average value of its path selection gating parameter γ across all samples in the validation set is calculated as the actual contribution score S_gate for that unit. The closer the value of S_gate is to 1, the greater the contribution of that unit to the modulation output; the closer it is to 0, the smaller the contribution.

[0181] Step S193: Sort all dynamic path deflection units from high to low according to their actual contribution scores to generate a dynamic path deflection unit contribution ranking list.

[0182] Sort the actual contribution scores S_gate of all dynamic path deflection units from high to low to obtain a contribution ranking list. Units at the top of the list have higher contributions, and units at the bottom have lower contributions.

[0183] Step S194: Select dynamic path deflection units whose actual contribution scores are lower than the preset contribution retention threshold from the contribution ranking list to form a candidate set of target deflection units to be evaluated.

[0184] The preset contribution retention threshold is 0.2. Dynamic path deflection units with S_gate below 0.2 are selected from the contribution ranking list to form a candidate set of target deflection units to be evaluated. These units are considered to be candidates that can be pruned.

[0185] Step S195: For each deflection unit to be evaluated in the candidate set of target deflection units to be evaluated, the function of the deflection unit is temporarily masked in the final generative artificial intelligence model, so that the data directly bypasses the deflection unit and flows from the corresponding key network layer node to the subsequent network layer node.

[0186] For each unit in the candidate set of deflection units to be evaluated, its function is temporarily disabled by modifying the model computation graph, that is, disconnecting the input and output connection of the unit, so that the output of the key network layer node is directly passed to the subsequent network layer node without going through the modulation processing of the unit.

[0187] Step S196: In the shielded state, the preset verification sample set is input into the final generative artificial intelligence model for forward propagation calculation to obtain the first output result of the model on the verification sample set after shielding the deflection unit.

[0188] A pre-defined validation sample set (containing clean and adversarial samples) is input into the model with the biased units to be evaluated masked, and forward propagation is performed to obtain the first output result. The size of the validation sample set is approximately 10% of the training set to ensure the accuracy of the evaluation.

[0189] Step S197: Compare the original output of the model on the verification sample set when the deflection unit is not masked with the first output, and calculate the change in output caused by the masking operation.

[0190] Calculate the difference between the original output (output without the deflection unit masked) and the first output (output with the deflection unit masked). For classification tasks, use the decrease in accuracy as the variable; for generation tasks, use the decrease in SSIM or BLEU scores as the variable. The variable Δ reflects the impact of masking the deflection unit on model performance.

[0191] Step S198: Determine whether the change in the output result exceeds a preset change tolerance threshold. If the change in the output result does not exceed the change tolerance threshold, mark the current deflection unit to be evaluated as a removable deflection unit.

[0192] The preset tolerance threshold for change is 1% (for accuracy or generation metrics). If Δ≤1%, it is considered that masking the deflection unit has little impact on model performance, and it is marked as a removable deflection unit.

[0193] Step S199: If the change in the output result exceeds the change tolerance threshold, the current deflection unit to be evaluated is marked as an unremovable deflection unit and removed from the candidate set of target deflection units to be evaluated.

[0194] If Δ>1%, it is considered that masking the deflection unit will significantly reduce the model performance, so it is marked as a non-removable deflection unit and removed from the candidate set of target deflection units to be evaluated.

[0195] Step S1910: Summarize all cells marked as removable deflection cells to generate a final list of removable deflection cells.

[0196] All dynamic path deflection units marked as removable deflection units are aggregated to generate a final list of removable deflection units, which includes the unit's name and the network layer node information it belongs to.

[0197] Step S1911: Based on the final removable deflection unit list, perform a pruning operation on the network structure of the final generative artificial intelligence model, completely delete each removable deflection unit in the final removable deflection unit list from the model, and restore the direct connection between the output end of the corresponding key network layer node and the input end of the subsequent network layer node.

[0198] Based on the final list of removable deflection units, each removable deflection unit is completely removed from the model, while the direct connection between the output of the key network layer node and the input of the subsequent network layer node is restored, which is the reverse of the operation in steps S163 to S165.

[0199] Step S1912: Reconstruct the structure of the pruned model, remove the internal parameters corresponding to all deleted dynamic path deflection units, and retain the weight parameters of all original network layer nodes and the internal modulation parameters of the dynamic path deflection units that have not been removed.

[0200] The model is restructured after pruning. The internal parameters (modulation intensity parameter vector α and path selection gating parameter γ) of the removed dynamic path deflection units are deleted, while the weight parameters of the original network layer nodes and the internal modulation parameters of the dynamic path deflection units that were not removed are retained, thereby reducing the number of parameters in the model.

[0201] Step S1913: The reconstructed model is fine-tuned on the adversarial example test set for a set number of rounds to adapt the reconstructed model to the pruned network structure. After the fine-tuning training is completed, a lightweight target generative artificial intelligence model with adaptive sparse pruning processing by dynamic path deflection units is obtained.

[0202] The reconstructed model was fine-tuned on an adversarial example test set, with 20 epochs of fine-tuning and a learning rate of 0.00001 to adapt to the pruned network structure. After fine-tuning, the model parameters were saved, resulting in a lightweight, deployable generative AI model. This model maintains high adversarial robustness while having lower computational complexity and fewer parameters, making it easier to deploy in practical applications.

[0203] In one exemplary embodiment, an AIGC model robustness enhancement system based on adversarial example defense is provided. This system can be a terminal, server, etc., and its internal structure diagram can be as follows: Figure 2 As shown, the AIGC model robustness enhancement system based on adversarial example defense includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements an AIGC model robustness enhancement method based on adversarial example defense. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the shell of the AIGC model robustness enhancement system based on adversarial sample defense, or an external keyboard, touchpad, or mouse, etc.

[0204] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A robustness enhancement method for AIGC models based on adversarial example defense, characterized in that, The method includes: Obtain the original neuron activation pattern sequence generated when the initial generative artificial intelligence model propagates forward layer by layer on a preset clean sample set, and construct the original inference path topology map of the initial generative artificial intelligence model for normal input based on the original neuron activation pattern sequence. The clean sample set is perturbed by a preset target perturbation generator to generate an adversarial sample test set covering multiple attack types. The adversarial sample test set is then input into the initial generative artificial intelligence model for forward propagation calculation. The perturbation neuron activation pattern sequence generated by the initial generative artificial intelligence model during layer-by-layer forward propagation is obtained. Based on the perturbation neuron activation pattern sequence, a perturbation inference path topology graph set for the initial generative artificial intelligence model for the perturbed input is constructed. The original inference path topology is compared with each perturbation inference path topology in the set of perturbation inference path topology by performing a layer-by-layer node activation difference analysis to identify key network layer nodes in the initial generative artificial intelligence model that have significant shifts in activation mode under perturbation input, as well as the activation shift direction vectors corresponding to the key network layer nodes. The significant shift is determined by comparing the total cross-attack type shift of the network layer nodes with a preset shift threshold. Based on the key network layer node and the activation offset direction vector, a dynamic path deflection unit oriented to a specific network layer node is constructed. The dynamic path deflection unit includes an activation direction modulation sublayer and a residual path selection sublayer that can be embedded in the output of the key network layer node. The dynamic path deflection unit is embedded into the output of the corresponding key network layer node in the initial generative artificial intelligence model to construct an intermediate state robust enhancement model containing an activation direction modulation mechanism. The intermediate state robust enhancement model is jointly optimized and trained using the adversarial example test set. The internal modulation parameters of the dynamic path deflection unit and the original weight parameters of the initial generative artificial intelligence model are adjusted simultaneously to generate the target generative artificial intelligence model.

2. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 1, characterized in that, The process of obtaining the original neuron activation pattern sequence generated during the layer-by-layer forward propagation of the initial generative artificial intelligence model on a preset clean sample set, and constructing the original inference path topology graph of the initial generative artificial intelligence model for normal input based on the original neuron activation pattern sequence, includes: Obtain multiple clean sample units from the preset clean sample set. The clean sample units are original input text units or original image units without any added perturbation information. The multiple clean sample units are sequentially input into the initial generative artificial intelligence model, and layer-by-layer forward propagation calculations are performed after the input layer of the initial generative artificial intelligence model. During the layer-by-layer forward propagation computation, an activation value capture hook function is set after each network layer of the initial generative artificial intelligence model. The activation value capture hook function captures the original activation tensor output by the current network layer for the clean sample unit of the current input. For each captured raw activation tensor, the activation value corresponding to each neuron in the raw activation tensor is recorded to generate the raw neuron activation vector of the current network layer under the input of the current clean sample unit. The original neuron activation vectors output by the same network layer for all clean sample units are summarized, and the average neuron activation vector of the network layer under all clean sample unit inputs is calculated. The average neuron activation vector is then used as the original layer representation vector of the network layer. Obtain the network structure connection information of the initial generative artificial intelligence model, which includes the sequential connection methods between each network layer in the model and the direction of data flow. Based on the network structure connection information, the original inference path topology of the initial generative artificial intelligence model for normal input is constructed using each network layer of the initial generative artificial intelligence model as a topology graph node, the original layer representation vector corresponding to each network layer as a node feature attribute, and the data flow direction between network layers as a directed connection edge. In the original inference path topology graph, each node carries the average activation mode information of the network layer under clean sample input, and the directed edges between nodes represent the computational dependency relationship between the output of the previous layer and the input of the next layer.

3. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 1, characterized in that, The clean sample set is perturbed using a preset target perturbation generator to generate an adversarial sample test set covering multiple attack types, including: Obtain the internal module configuration information of the preset target perturbation generator, which includes a gradient information attack module, a decision boundary attack module, a transformation-based attack module, and a score-based attack module; Each clean sample unit in the clean sample set is sequentially input into the gradient information attack module. The gradient information attack module calculates the gradient of the loss function of the initial generative artificial intelligence model relative to the clean sample unit, and adds perturbation to the clean sample unit in the gradient ascent direction to generate preliminary adversarial sample units based on gradient information. Each clean sample unit in the clean sample set is sequentially input into the decision boundary attack module. The decision boundary attack module performs a binary search near the decision boundary of the initial generative artificial intelligence model to find the perturbation point that is closest to the clean sample unit and can cause the model to misclassify, thereby generating an initial adversarial sample unit based on the decision boundary. Each clean sample unit in the clean sample set is sequentially input into the transformation-based attack module. The transformation-based attack module performs spatial transformation or geometric distortion operations on the clean sample units to change the semantic structure of the samples while keeping human perception unchanged, thereby generating a preliminary adversarial sample unit based on transformation. Each clean sample unit in the clean sample set is sequentially input into the score-based attack module. The score-based attack module estimates the gradient direction of the output probability of the initial generative artificial intelligence model by sampling, and adds perturbation to the clean sample unit in the estimated gradient direction to generate a preliminary adversarial sample unit based on score. The initial adversarial sample units generated by the gradient information attack module, the decision boundary attack module, the transformation-based attack module, and the score-based attack module are collected to form an original adversarial sample pool containing the generation results of multiple attack types. The attack effectiveness of the initial adversarial sample units in the original adversarial sample pool is verified. Invalid adversarial sample units that cannot successfully attack the initial generative artificial intelligence model are filtered out, and valid adversarial sample units that can cause the model to produce incorrect output are retained. All valid adversarial sample units are categorized and stored according to the attack module type from which they originate, generating an adversarial sample test set covering multiple attack types.

4. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 1, characterized in that, The step involves inputting the adversarial example test set into the initial generative AI model for forward propagation computation, obtaining the perturbation neuron activation pattern sequence generated by the initial generative AI model during layer-by-layer forward propagation, and constructing a perturbation inference path topology graph set for the initial generative AI model for the perturbed input based on the perturbation neuron activation pattern sequence, including: Select a subset of adversarial samples corresponding to the first attack type from the adversarial sample test set, and input multiple adversarial sample units from the subset of adversarial samples corresponding to the first attack type into the initial generative artificial intelligence model in sequence; After the input layer of the initial generative artificial intelligence model, layer-by-layer forward propagation computation is performed, and after each network layer, the perturbation activation tensor of the adversarial example unit output of the current network layer for the current input is captured by the activation value capture hook function. For each captured perturbation activation tensor, the activation value corresponding to each neuron in the perturbation activation tensor is recorded to generate the perturbation neuron activation vector of the current network layer under the input of the current adversarial example unit. The perturbation neuron activation vectors output by all adversarial sample units corresponding to the first attack type of the same network layer are summarized, and the average perturbation neuron activation vector of the network layer under the input of all adversarial sample units corresponding to the first attack type is calculated. The average perturbation neuron activation vector is used as the first perturbation layer representation vector of the network layer for the first attack type. Based on the network structure connection information, the network layers of the initial generative artificial intelligence model are used as nodes in the topology graph, the first perturbation layer representation vectors corresponding to each network layer are used as node feature attributes, and the data flow direction between network layers is used as directed connection edges to construct the first perturbation inference path topology graph of the initial generative artificial intelligence model for the first type of attack. From the adversarial sample test set, select adversarial sample subsets corresponding to the second attack type, the third attack type and so on up to the last attack type in sequence. For each adversarial sample subset corresponding to the attack type, repeat the above steps of forward propagation calculation, perturbation neuron activation vector recording, average perturbation neuron activation vector calculation and perturbation inference path topology graph construction. The perturbation inference path topology map for each attack type is obtained separately. The perturbation inference path topology maps corresponding to all attack types are summarized to form the set of perturbation inference path topology maps for the perturbation input of the initial generative artificial intelligence model.

5. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 4, characterized in that, The step of performing a layer-by-layer node activation difference comparison analysis between the original inference path topology graph and each perturbation inference path topology graph in the set of perturbation inference path topology graphs to identify key network layer nodes in the initial generative artificial intelligence model that have undergone significant shifts in activation patterns under perturbation input, and the activation shift direction vectors corresponding to the key network layer nodes, includes: Extract the original layer representation vector corresponding to each network layer node from the original inference path topology graph as the first comparison benchmark; Each perturbation inference path topology graph is sequentially obtained from the set of perturbation inference path topology graphs, and the perturbation layer representation vector corresponding to each network layer node is extracted from the currently obtained perturbation inference path topology graph as the second comparison object; For each network layer node, calculate the element-wise difference between the original layer representation vector and the perturbation layer representation vector to generate the activation offset vector of the network layer node under the current attack type. Based on the value of each element in the activation offset vector, determine the activation change magnitude and direction of each neuron in the network layer node under the current attack type; The sum of squares of all elements in the activation offset vector is used to obtain the overall activation offset of the network layer node under the current attack type. The total cross-attack type offset of the same network layer node is obtained by summing the combined activation offsets of the same network layer node under different attack types. Based on the total offset of all network layer nodes across attack types, network layer nodes whose total offset value across attack types exceeds a preset offset threshold are selected as the set of candidate key network layer nodes whose activation mode has significantly shifted. For each candidate key network layer node in the candidate key network layer node set, its activation offset vectors under different attack types are superimposed to generate a synthetic comprehensive activation offset direction vector. The integrated activation offset direction vector is normalized to obtain the activation offset direction vector corresponding to the key network layer node. The activation offset direction vector indicates the overall trend direction of the internal neuron activation mode of the key network layer node deviating from the original state when facing multiple attacks.

6. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 5, characterized in that, The step of constructing a dynamic path deflection unit oriented towards a specific network layer node based on the key network layer node and the activation offset direction vector includes: Obtain the activation offset direction vector corresponding to the key network layer node, and use the activation offset direction vector as the core modulation reference for constructing the activation direction modulation sublayer; The activation direction modulation sub-layer is constructed, which contains a learnable modulation intensity parameter vector and a fixed projection matrix. Each row of the projection matrix is ​​orthogonal to the activation offset direction vector, and is used to decompose the input feature vector into components along the offset direction and components perpendicular to the offset direction. The input of the activation direction modulation sublayer is connected to the output of the key network layer node, and the activation direction modulation sublayer receives the original node activation vector output by the key network layer node as input; Inside the activation direction modulation sublayer, the original node activation vector is first decomposed into a parallel offset component and a vertical offset component using the projection matrix. Then, the learnable modulation intensity parameter vector is multiplied element-wise with the parallel offset component to obtain the modulated parallel offset component. Finally, the modulated parallel offset component is vector-added with the original vertical offset component to synthesize the pre-modulated activation vector. Construct the residual path selection sublayer, which contains a learnable path selection gating parameter and a residual connection unit. Connect the input of the residual path selection sublayer to the output of the activation direction modulation sublayer to receive the pre-modulated activation vector as input. At the same time, connect the residual input of the residual path selection sublayer directly to the output of the key network layer node to receive the original node activation vector as another input. Inside the residual path selection sub-layer, a fusion coefficient between zero and one is calculated using the learnable path selection gating parameter. Based on the fusion coefficient, the pre-modulated activation vector is weighted and fused with the original node activation vector to generate the final modulated output activation vector. The activation direction modulation sublayer and the residual path selection sublayer are encapsulated and integrated to form a dynamic path deflection unit for the key network layer node. The input interface of the dynamic path deflection unit receives the original output from the key network layer node, and the output interface of the dynamic path deflection unit outputs the new activation vector after direction modulation and path selection.

7. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 6, characterized in that, The step of embedding the dynamic path deflection unit into the output of the corresponding key network layer node in the initial generative artificial intelligence model to construct an intermediate robust enhancement model containing an activation direction modulation mechanism includes: The original computation graph of the initial generative artificial intelligence model is analyzed. The original computation graph contains a complete data flow path from the input layer through each network layer node to the output layer. Each network layer node has a clear input connection source node and output connection target node. Based on the key network layer nodes and their specific positions in the original computation graph, determine the output connection target node list for each key network layer node in the original computation graph; For each critical network layer node, disconnect the direct connection between the original output of the critical network layer node and the list of output connection target nodes in the original computation graph; The input of the dynamic path deflection unit is connected to the original output of the key network layer node to receive the original activation vector output by the key network layer node. Connect the output of the dynamic path deflection unit to the input of each target node in the output connection target node list, which should originally receive the original output of the key network layer node; Through the above connection operations, the dynamic path deflection unit is inserted as an intermediary module between the key network layer node and its subsequent network layer nodes, so that the output of the key network layer node must first be modulated by the dynamic path deflection unit before it can be transmitted to the subsequent network layer. Repeat the operation of disconnecting the original connection and inserting dynamic path deflection units for all identified key network layer nodes, update the original computation graph, and obtain a new computation graph containing multiple dynamic path deflection unit nodes; The network structure of the initial generative artificial intelligence model is reconstructed based on the new computation graph. The dynamic path deflection unit is added as a trainable module to the model and together with the original network layer nodes, they form an overall network, thereby constructing an intermediate state robust enhancement model that includes an activation direction modulation mechanism.

8. The method for enhancing the robustness of AIGC models based on adversarial example defense according to claim 7, characterized in that, The process of jointly optimizing and training the intermediate-state robustness enhancement model using the adversarial example test set, and simultaneously adjusting the internal modulation parameters of the dynamic path deflection unit and the original weight parameters of the initial generative AI model to generate the target generative AI model includes: The intermediate robustness enhancement model is set to the training state, and the updatable properties of the original weight parameters of all original network layer nodes in the model are activated. At the same time, the updatable properties of the internal modulation parameters of all dynamic path deflection units are activated. The internal modulation parameters include the modulation intensity parameter vector in the activation direction modulation sublayer and the path selection gating parameter in the residual path selection sublayer. Randomly select an adversarial sample unit from the adversarial sample test set and input the adversarial sample unit from the training batch into the intermediate state robustness enhancement model; In the intermediate robustness enhancement model, forward propagation calculation is performed. When the data flows through each node embedded with a key network layer, the key network layer node first calculates its original output activation vector, and then sends the original output activation vector to the dynamic path deflection unit connected to its output. Inside the dynamic path deflection unit, the original output activation vector is sequentially processed by the modulation of the activation direction modulation sublayer and the fusion of the residual path selection sublayer to generate the final modulated output activation vector, and the final modulated output activation vector is then passed to subsequent network layers. After processing by all network layer nodes and all dynamic path deflection units, the model prediction output result corresponding to the adversarial example unit of the current training batch is obtained from the output layer of the intermediate robustness enhancement model. Obtain the real label information corresponding to the adversarial example units in the current training batch, and calculate the classification loss value of the current training batch based on the model prediction output and the real label information; Based on the classification loss value, the gradient of the classification loss value with respect to all updatable parameters in the intermediate state robustness enhancement model is calculated using the backpropagation algorithm. The updatable parameters include the original weight parameters of all original network layer nodes and the internal modulation parameters of all dynamic path deflection units. Based on the calculated gradient value, and combined with the preset gradient descent optimizer, all updatable parameters of the intermediate robustness enhancement model are updated synchronously to reduce the classification loss value. Repeat the steps of training batch extraction, forward propagation calculation, loss calculation, gradient backpropagation and parameter update until the preset total number of joint optimization training rounds is reached. Finally, the model parameters are solidified to generate the target generative artificial intelligence model.

9. A robustness enhancement system for AIGC models based on adversarial example defense, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the AIGC model robustness enhancement method based on adversarial example defense as described in any one of claims 1 to 8 by executing the machine-executable instructions.

10. A computer program product, characterized in that, The computer program product includes machine-executable instructions stored in a computer-readable storage medium. The processor of the AIGC model robustness enhancement system based on adversarial example defense reads the machine-executable instructions from the computer-readable storage medium and executes the machine-executable instructions, causing the AIGC model robustness enhancement system based on adversarial example defense to perform the AIGC model robustness enhancement method based on adversarial example defense as described in any one of claims 1 to 8.

Images