Transformer model pruning method, system and storage medium

Abstract

The application relates to a Transformer model pruning method and system and a storage medium, the pruning method fuses time domain, frequency domain, cross-layer gradient and sensitive data through dimension redundancy detection, avoids single dimension error pruning of key structures, converts pruning structure search into an attack and defense optimization problem through red and blue double-population game collaborative differential evolution, makes the finally reserved structure not only have high compression rate but also naturally have anti-robustness, solves the problem that a traditional pruned model is fragile, and through an alternating anti-collaborative optimization mechanism, the most destructive interference can be adaptively generated, a truly robust pruning structure is screened out, and the pruned model can guarantee robustness and accuracy in various application scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a Transformer model pruning method, system, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, neural network models based on the Transformer architecture have been widely used in fields such as natural language processing, speech recognition, and computer vision. However, these models have a large number of parameters and high computational complexity, posing challenges for deployment on resource-constrained devices. Model pruning, as an effective model compression technique, has emerged to address this challenge. This technique aims to reduce the complexity, computational load, and storage requirements of a model by removing redundant connections, parameters, or computational units from the neural network, while maintaining model performance as much as possible.

[0003] However, the pruning methods used in existing neural network models based on the Transformer architecture have many technical problems that urgently need to be solved.

[0004] On the one hand, the core flaw of existing pruning methods lies in their single-dimensional detection of redundant features and lack of global optimization capabilities. Traditional dynamic token pruning and information entropy-based pruning methods largely rely on temporal attention weights or local feature entropy values ​​for redundancy judgment. This single-dimensional detection approach has significant limitations. Taking speech recognition tasks as an example, high-frequency spectral components play a crucial role in phoneme recognition; however, existing methods, lacking frequency domain analysis, are prone to mispruning such critical structures, leading to irreversible loss of model accuracy. Simultaneously, most existing pruning strategies employ layer-by-layer local optimization, failing to fully consider the global nature of cross-layer information flow. In deep networks, once a token is mispruned, it is difficult to recover it through subsequent layers, resulting in a broken information transmission path in the pruned model and severely impacting the overall model performance.

[0005] On the other hand, existing pruning methods require significant time for model fine-tuning after pruning, and the accuracy recovery is unstable, resulting in low model recovery efficiency. In image classification tasks, the dynamic token pruning method, lacking cross-layer gradient propagation analysis, leads to a more than 5% drop in model accuracy after pruning, while the retraining time required for the recovery process reaches more than 50% of the original training time, severely limiting the model's efficiency in practical applications. More seriously, existing pruning methods severely neglect model robustness; pruned models experience a sharp performance drop when facing noise interference or adversarial attacks. For example, the BERT pruned model experiences a 15% drop in accuracy under text adversarial attacks, fully exposing its vulnerability in complex application scenarios.

[0006] Furthermore, existing pruning methods, while pursuing model compression, often implement pruning decisions and adversarial robustness enhancement as two independent stages: first, structural pruning is performed, and then adversarial training or fine-tuning is used to restore the model's robustness. This two-stage approach has inherent flaws: the pruning stage's assessment of redundant structures does not consider the vulnerabilities that may be exposed in the subsequent adversarial fine-tuning stage. This means that some structures judged as redundant in static detection actually play a crucial role in adversarial defense, and once pruned, they cannot be recovered through subsequent fine-tuning, resulting in irreversible accuracy loss and robustness degradation. Therefore, there is an urgent need for a technical solution that can coordinate pruning structure search and adversarial robustness enhancement within the same optimization framework.

[0007] In summary, existing pruning methods used in Transformer-based neural network models have significant shortcomings in redundant feature detection, global optimization, model recovery efficiency, and robustness. A more efficient and comprehensive pruning technique is urgently needed to improve the performance of models in practical applications. Summary of the Invention

[0008] This invention provides a Transformer model pruning method, system, and storage medium, aiming to solve the technical problem of insufficient optimization performance of existing pruning methods used for neural network models based on the Transformer architecture.

[0009] To address the aforementioned technical problems, in a first aspect, the present invention provides a Transformer model pruning method, comprising the following steps: Obtain the Transformer model to be pruned as the original model; The original model is pre-trained until it converges to obtain a pre-trained model. Data is then collected from the pre-trained model to obtain multi-dimensional data, including time-domain data, frequency-domain data, cross-layer data, and adversarial-sensitive data. Based on the multi-dimensional data, the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude are obtained. A comprehensive redundancy score is calculated based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Attention heads with a comprehensive redundancy score higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures. A red-blue dual-population game model is constructed using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population. The model is then optimized using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies the preset fitness. The pre-trained model is pruned according to the target pruning structure to obtain a simplified model; Using the original model as the teacher model and the simplified model as the student model, the simplified model is trained using a double knowledge distillation method to align the KL divergence loss between the attention matrices of the student model and the teacher model, and to align the cross-entropy loss between the output of the student model and the output of the teacher model. The output of the student model after training is then used as the pruned model corresponding to the Transformer model.

[0010] Furthermore, the time-domain data includes: the weight matrix of each attention head in the pre-trained model, and the activation values ​​corresponding to the weight matrix; The frequency domain data includes: spectral features extracted by performing a fast Fourier transform on the time domain data; The cross-layer data includes: the inter-layer gradient propagation path between the middle layer and the attention head connection of the pre-trained model, and the activation intensity of the feedforward network; The adversarial sensitive data includes: the perturbation magnitude of the response of the mid-level and attention heads of the pre-trained model to Gaussian noise or fast gradient sign-based adversarial attacks during pre-training.

[0011] Furthermore, based on the multi-dimensional data, the steps of obtaining the cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude, and calculating a comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity, and marking attention heads with comprehensive redundancy scores higher than a preset threshold as candidate pruning structures, and the rest as retained structures, include: The cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model are calculated based on the time-domain data and the frequency-domain data. The inter-layer score of the inter-layer gradient propagation path is calculated based on the cross-layer data; The sensitivity of each attention head to the amplitude of the disturbance is calculated based on the aforementioned adversarial sensitivity data; After normalizing the cosine similarity, spectral similarity, inter-layer score, and sensitivity, the geometric mean method is used to calculate the comprehensive redundancy score for each attention head. The comprehensive redundancy score is defined as Score and satisfies the following relationship: Score = (S_cos×S_spec×S_layer×S_adv)^{1 / 4}; Where S_cos is the normalized cosine similarity, S_spec is the normalized spectral similarity, S_layer is the normalized inter-layer score, and S_adv is the normalized sensitivity. Attention heads whose overall redundancy score is higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures.

[0012] Furthermore, the step of constructing a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and optimizing it using an alternating adversarial cooperative differential evolution algorithm to obtain the target pruning structure that satisfies the preset fitness, includes the following sub-steps: The Blue Army population is constructed based on the candidate pruning structure, and the initial individuals of the population are selected based on the adversarial sensitive data to construct the Red Army population. Each individual in the Blue Army population is encoded as a pruning mask vector, which indicates whether to retain or remove each attention head in the candidate pruning structure. The perturbation patterns corresponding to the top K% of attention heads in the adversarial sensitive data are selected as the initial individuals of the Red Army population, and each individual in the Red Army population is encoded as a noise interference vector, which simulates perturbations against the pre-trained model. The current blue force population is fixed, and mutation and crossover operations are performed on the red force population based on the attention head corresponding to the preserved structure to generate attack interference targeting the key preserved structure; then, the attack fitness of each individual in the red force population is calculated according to the alternating adversarial cooperative differential evolution algorithm. The attack fitness is positively correlated with the accuracy decrease of the pre-trained model under interference. Based on the attack fitness, individuals in the red force population whose noise interference vector is greater than a preset interference threshold are selected to form an evolved red force population. The Red Army population is fixed, and the Blue Army population is subjected to mutation and crossover operations according to the alternating adversarial cooperative differential evolution algorithm. The pruned structure individuals with the slowest decrease in comprehensive redundancy score under the attack interference of the Red Army population are selected. At the same time, the defense fitness of each individual in the Blue Army population is calculated. The defense fitness is obtained by weighting the compression ratio of the pre-trained model, hardware latency, and robust accuracy under the individual attacks of the Red Army population. The attack fitness of the Red Army population and the defense fitness of the Blue Army population are calculated alternately in each generation iteration, forming a closed loop of mutual game and cooperative evolution between the Red and Blue sides. Iteratively perform crossover and mutation of the Red Army population and the Blue Army population until the rate of change of the defense fitness within a consecutive preset number of generations is less than a preset threshold. Then, output the candidate pruning structure corresponding to the individual in the current Blue Army population that satisfies the preset fitness as the target pruning structure.

[0013] Furthermore, the step of pruning the pre-trained model according to the target pruning structure to obtain a simplified model also includes: In the pre-trained model, skip connections are added between different structural levels that are pruned according to the target pruning structure.

[0014] Furthermore, the defense fitness is defined as... It satisfies the following relationship: ; in, The compression ratio of the pre-trained model. The inference latency of the pre-trained model on the specified reference hardware platform. This represents the worst-case verification accuracy of the current blue team population under attack from the red team population.

[0015] Secondly, the present invention also provides a Transformer model pruning system, comprising: The initialization module is used to obtain the Transformer model to be pruned as the original model. The pre-training module is used to pre-train the original model until it converges to obtain a pre-trained model, and to collect data from the pre-trained model to obtain multi-dimensional data including time domain data, frequency domain data, cross-layer data and adversarial sensitive data. The redundancy detection module is used to obtain the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude based on the multi-dimensional data. It calculates a comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Attention heads with a comprehensive redundancy score higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures. The differential evolution module is used to construct a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and to optimize it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies a preset fitness. The pruning module is used to prune the pre-trained model according to the target pruning structure to obtain a simplified model. The knowledge distillation module is used to learn the parameters of the simplified model using the original model as the teacher model and the simplified model as the student model, by employing a dual knowledge distillation method. This aligns the KL divergence loss between the attention matrices of the student model and the teacher model, and aligns the cross-entropy loss between the outputs of the student model and the teacher model. The output of the student model after learning is then used as the pruned model corresponding to the Transformer model.

[0016] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a Transformer model pruning program stored in the memory and executable on the processor, wherein the processor, when executing the Transformer model pruning program, implements the steps of the Transformer model pruning method as described in any of the above embodiments.

[0017] Fourthly, the present invention also provides a storage medium storing a Transformer model pruning program, wherein when the Transformer model pruning program is executed by a processor, it implements the steps of the Transformer model pruning method as described in any of the above embodiments.

[0018] The beneficial effect achieved by this invention lies in proposing a pruning method for neural network models based on the Transformer architecture, which has the following advantages: First, by simultaneously fusing time domain, frequency domain, cross-layer gradient, and adversarial sensitive data through dimensional redundancy detection, we avoid miscutting key structures from a single dimension. Second, by transforming the pruning structure search into an offensive and defensive adversarial optimization problem through red-blue dual-population game-based collaborative differential evolution, the final retained structure not only has a high compression rate but also naturally possesses adversarial robustness, thus solving the problem of the fragility of the traditional pruning model. Third, the alternating adversarial collaborative optimization mechanism can adaptively generate the most destructive interference, screen out the truly robust pruning structure, and ensure the robustness and accuracy of the pruned model in various application scenarios.

[0019] Fourth, by constructing a closed-loop feedback system with multi-dimensional redundancy detection and red-blue dual-population game, the redundancy confidence of pruning decisions is continuously and dynamically recalibrated under the drive of actual adversarial effects. This breaks the linear technical mindset of pruning first and then enhancing in existing technologies, and fundamentally eliminates the inherent defect of structural misjudgment in traditional two-stage processes that cannot be retrospectively corrected. Attached Figure Description

[0020] Figure 1This is a flowchart of the steps of the Transformer model pruning method provided in the embodiments of the present invention; Figure 2 This is a line graph comparing the ablation experiments of the Transformer model pruning method provided in this embodiment of the invention. Figure 3 This is a line graph comparing the performance of the Transformer model pruning method provided in this embodiment of the invention with existing methods; Figure 4 This is a schematic diagram of the Transformer model pruning structure of the system provided in the embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of the Transformer model pruning method provided in this embodiment of the invention. The Transformer model pruning method includes the following steps: S101. Obtain the Transformer model to be pruned as the original model.

[0023] In this embodiment of the invention, the Transformer model refers to an application model with this architecture, such as a recognition model, translation model, or medical image processing model that integrates LiDAR and camera. Since the method proposed in this embodiment of the invention aims to optimize the model architecture, any existing application model that conforms to the Transformer architecture can be used as the original model defined in this embodiment of the invention, and pruning optimization can be achieved according to the relevant methods proposed in this embodiment of the invention.

[0024] S102. Pre-train the original model until it converges to obtain a pre-trained model, and collect data from the pre-trained model to obtain multi-dimensional data including time domain data, frequency domain data, cross-layer data and adversarial sensitive data.

[0025] In this embodiment of the invention, the purpose of pre-training is to bring it to a stable state (convergence) to ensure that subsequent pruning is based on a mature model parameter distribution.

[0026] Specifically, in this embodiment of the invention, the redundancy of the model structure is acquired and characterized from multiple perspectives, including time-domain data, frequency-domain data, cross-layer data, and adversarial-sensitive data. The time-domain data includes: the weight matrix of each layer's attention head in the pre-trained model, and the activation values ​​corresponding to the weight matrix; the time-domain data belongs to the time-series features output by each layer of the model in sequence processing. The frequency domain data includes: spectral features extracted by performing a fast Fourier transform on the time domain data; the frequency domain data can reflect the signal fluctuation pattern. The cross-layer data includes: the inter-layer gradient propagation path between the middle layers and the attention head connection of the pre-trained model, and the activation intensity of the feedforward network; the cross-layer data can reflect the interaction information between different layers in the model; The adversarial-sensitive data includes the magnitude of the response perturbation to Gaussian noise or Fast Gradient Sign Method (FGSM) adversarial attacks; the adversarial-sensitive data reflects the differences in the model's response when input adversarial examples, and is used to detect the model's robustness redundancy to noise.

[0027] Based on the collection of time-domain data, frequency-domain data, cross-layer data, and adversarial-sensitive data, the redundancy of the model structure can be comprehensively characterized from four dimensions: time, frequency, hierarchical interaction, and robustness, avoiding information loss caused by single-dimensional pruning.

[0028] S103. Based on the multi-dimensional data, obtain the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude. Calculate the comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Mark attention heads with a comprehensive redundancy score higher than a preset threshold as candidate pruning structures, and mark the rest as retained structures.

[0029] In this embodiment of the invention, step S103 includes: The cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model are calculated based on the time-domain data and the frequency-domain data. The inter-layer score of the inter-layer gradient propagation path is calculated based on the cross-layer data; The sensitivity of each attention head to the amplitude of the disturbance is calculated based on the aforementioned adversarial sensitivity data; After normalizing the cosine similarity, spectral similarity, inter-layer score, and sensitivity, the geometric mean method is used to calculate the comprehensive redundancy score for each attention head. The comprehensive redundancy score is defined as Score and satisfies the following relationship: Score = (S_cos×S_spec×S_layer×S_adv)^{1 / 4}; Where S_cos is the normalized cosine similarity, S_spec is the normalized spectral similarity, S_layer is the normalized inter-layer score, and S_adv is the normalized sensitivity. Attention heads whose overall redundancy score is higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures.

[0030] S104. Construct a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and optimize it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies the preset fitness.

[0031] Step S104 aims to simulate the evolution of attack and defense, seeking not only the structure with the fewest parameters but also the most resilient structure. In this embodiment of the invention, an optimal solution based on the evolution of attack and defense is achieved by constructing a red-blue dual-population game model. Specifically, step S104 includes the following sub-steps: S1041. Construct the Blue Army population based on the candidate pruning structure, and select initial individuals of the population based on the adversarial sensitive data to construct the Red Army population; wherein, each individual in the Blue Army population is encoded as a pruning mask vector, the pruning mask vector is used to indicate the retention or removal of each attention head in the candidate pruning structure; the perturbation pattern corresponding to the top K% of the attention heads in the adversarial sensitive data is selected as the initial individuals of the Red Army population, and each individual of the Red Army population is encoded as a noise interference vector, the noise interference vector is used to simulate the perturbation against the pre-trained model.

[0032] The initial population of the blue team is generated based on the candidate pruning structure obtained in step S103. Each blue team individual is encoded as a binary mask, representing the decision to retain or prune the attention head or multilayer perceptron (MLP) layer in the pre-trained model.

[0033] The Red Army population initializes itself using the adversarial-sensitive data collected in step S102. Each Red Army individual is encoded as a noise injection vector, specifically designed to perturb regions with high adversarial sensitivity in the pre-trained model.

[0034] S1042. Fix the current blue force population, perform mutation and crossover operations on the red force population based on the attention head corresponding to the preserved structure, and generate attack interference targeting the key preserved structure; then calculate the attack fitness of each individual in the red force population according to the alternating adversarial cooperative differential evolution algorithm. The attack fitness is positively correlated with the accuracy decrease of the pre-trained model under interference. Based on the attack fitness, select individuals in the red force population whose noise interference vector is greater than a preset interference threshold to form an evolved red force population.

[0035] In the mutation, crossover, and evolutionary process of the red-blue dual-population game model, the blue population is first fixed to optimize the red population. Specifically, the red team's goal is to maximize the loss function of the pruned model. The fitness function of individuals in the red team population is defined as: ; in, Indicates the first among the Red Army groups j individual attackers R j Attack adaptability, For the Blue Army The first in Individual, This represents the original loss value of the pruning model. This represents the pruning model after adding attack interference.

[0036] If an individual significantly reduces the performance of a current pruning model, then that individual is retained.

[0037] Unlike the independent evolutionary pattern of a single population in the standard difference evolutionary algorithm, in this embodiment of the invention, the fitness functions of the Red and Blue populations are calculated alternately and used as inputs to each other in each generation, forming a co-evolutionary pattern of mutual restraint between the two populations, rather than a simple splicing of two independent optimization processes. This makes the search of the pruning structure and the enhancement of adversarial robustness fundamentally inseparable. Based on this, subsequent steps will fix the evolved Red population and perform targeted selection on the Blue population.

[0038] S1043. Fix the evolved Red Army population, and perform mutation and crossover operations on the Blue Army population according to the alternating adversarial cooperative differential evolution algorithm. Select the pruned structure individuals with the slowest decrease in comprehensive redundancy score under the attack interference of the evolved Red Army population. At the same time, calculate the defense fitness of each individual in the Blue Army population. The defense fitness is obtained by weighting the compression ratio of the pre-trained model, hardware latency, and robust accuracy under the individual attacks of the evolved Red Army population. The attack fitness of the Red Army population and the defense fitness of the Blue Army population are calculated alternately in each generation iteration, forming a closed loop of mutual game and cooperative evolution between the Red and Blue sides.

[0039] The Evolutionary Red Army population increases the level of interference with the pre-trained model, so the goal of the Blue Army population is to maintain high accuracy and low latency in the pre-trained model even when attacked by the Evolutionary Red Army population.

[0040] In this embodiment of the invention, the defense fitness is defined as... It satisfies the following relationship: ; in, The compression ratio of the pre-trained model. The inference latency of the pre-trained model on the specified reference hardware platform. This represents the worst-case verification accuracy of individuals in the current Blue Army population under attacks from the Red Army population. Only those individuals that have had redundant structures removed (without affecting accuracy) and have retained key anti-interference structures (to resist the Red Army) can achieve a higher defensive fitness.

[0041] S1044. Iteratively perform crossover and mutation of the Red Army population and the Blue Army population until the rate of change of the defense fitness within a consecutive preset number of generations is less than a preset threshold. Output the candidate pruning structure corresponding to the individual in the current Blue Army population that satisfies the preset fitness as the target pruning structure.

[0042] Step S1044 is used to control the convergence of the iteration. As the iteration proceeds, the attack interference and destructive performance of the Red Army population gradually improves, and the anti-disturbance capability of each candidate pruning structure in the Blue Army population continues to strengthen. When the rate of change of the defense fitness of the Blue Army population within a consecutive preset number of generations is less than a preset threshold, it indicates that the candidate pruning structure in the Blue Army population has become robust, and the Red Army population can no longer cause a significant decrease in accuracy. Then, the candidate pruning structure corresponding to the individual in the current Blue Army population that satisfies the preset fitness is output as the target pruning structure.

[0043] In this embodiment of the invention, the multi-dimensional redundancy detection performed in step S103 and the red-blue dual-population game model constructed in step S104 form a deeply coupled closed-loop optimization system, producing significant synergistic technical effects, specifically reflected in the following three aspects: First, the detection results provide prior knowledge for the game, avoiding ineffective searches. In existing pruning methods, the initial population of the differential evolution algorithm is usually randomly generated, resulting in a large number of ineffective pruning structures being evaluated, leading to low search efficiency. In this embodiment of the invention, through weighted fusion in step S103, the time domain, frequency domain, cross-layer, and adversarial sensitive data are integrated into the comprehensive redundancy score of each attention head, and the candidate pruning structure of the blue team population is constructed based on this. At the same time, the initial individuals of the red team population are not random noise, but are specifically selected from the perturbation patterns corresponding to the top K% of attention heads in the adversarial sensitive data. This prior knowledge-guided population initialization allows both the red and blue teams to focus on the regions in the model with the greatest pruning potential and the greatest vulnerability to attack before the game begins. Compared with random initialization, the search efficiency is improved, and irreversible accuracy loss caused by blind pruning is avoided.

[0044] Second, the confidence of redundant detection is corrected in reverse during the game process, achieving dynamic recalibration. Single static redundant detection is easily limited by the parameter distribution of the pre-trained model. For example, some attention heads may appear redundant on clean data, but become crucial under adversarial interference. In this embodiment of the invention, in each iteration of the red-blue game, the blue team faces targeted attacks on the preserved structure generated by the red team. The defensive fitness of individual blue team members includes the evaluation metric of "the slowest decrease in overall redundancy score under attack interference." This means that if an attention head is marked as a candidate for pruning (redundancy) in static detection, but its overall redundancy score drops sharply under the red team's attack (i.e., the model accuracy is significantly impaired), then this structure will be re-evaluated as a preserved structure in subsequent iterations. Conversely, if an attention head scores highly (important) in static detection, but its score remains stable under the red team's attack, then its redundancy is further confirmed. This game-driven dynamic adjustment mechanism for redundant confidence is something that neither a single detection method nor a single game optimization can achieve. Its direct effect is that the final selected target pruned structure simultaneously achieves Pareto optimality in three indicators: compression ratio, conventional accuracy, and adversarial robustness.

[0045] Third, the coupling system produced a synergistic effect. Furthermore, to verify the synergistic effect of the coupling system, an ablation study was conducted on the method implemented in the above embodiments, and the results are shown in Table 1 below.

[0046] Table 1 Ablation experiment results of the multi-dimensional redundancy detection coupled with the red-blue game.

[0047] To further illustrate the synergistic effect of the coupled system, please refer to... Figure 2 , Figure 2 This is a line graph showing the data corresponding to the experimental results in Table 1. Figure 2The four vertical axes represent compression rate (%), normal accuracy decrease (%), accuracy decrease under adversarial attacks (%), and search time (relative value). Each configuration corresponds to a line graph, which is distinguished by different black and white line styles and markers.

[0048] As can be seen, compared to the combination of uncoupled multi-dimensional detection and red-blue game theory (i.e., detection followed by game theory, but the game initialization does not utilize the detection results), the coupling scheme of this embodiment reduces the conventional accuracy decrease from 2.5% to 1.2% and the accuracy decrease under adversarial attacks from 4.2% to 2.1% at the same compression rate, while actually reducing the search time. Since increasing coupling usually incurs computational overhead, the above results demonstrate that this embodiment reduces ineffective iterations through prior knowledge guidance, achieving a simultaneous improvement in efficiency and effectiveness.

[0049] Unlike existing technologies that implement pruning decisions and adversarial training as sequential, independent stages, the red-blue dual-population game proposed in this invention constructs a bidirectionally coupled closed-loop co-evolutionary framework through the alternating adversarial co-evolutionary differential evolution algorithm, rather than a unidirectional attack-defense simulation or two-stage independent operations. In this framework, the attack fitness of the red population and the defense fitness of the blue population are calculated alternately and mutually drive each other within the same iteration cycle: the optimization direction of the red population is dynamically guided by the vulnerable structures currently retained by the blue population, while the selection criteria of the blue population are dynamically adjusted by the attack interference generated in real time by the red population. The two are deeply integrated and mutually reinforcing within the same optimization loop. Furthermore, the initial individuals of the red population are not randomly generated, but are guided and initialized based on the perturbation patterns corresponding to the top K% of attention heads in adversarial sensitivity in step S103, ensuring that the starting point of the game focuses on the most vulnerable areas of the model. Through the aforementioned bidirectional coupling mechanism and guided initialization strategy, the pruning structure is simultaneously endowed with adversarial robustness during the search evolution process, overcoming the irreversible damage to the pruning structure caused by the separation of the pruning objective and the robustness optimization objective in the traditional two-stage scheme.

[0050] It should be noted that in the ablation experiment results shown in Table 1 above, the configuration of "multi-dimensional detection + red-blue game (uncoupled, i.e., separate optimization)" represents the most likely technical path that existing technologies will take when facing the dual requirements of pruning and robustness: that is, to first complete redundant detection independently, and then to perform adversarial search optimization independently. Under this configuration, the conventional accuracy decreases by 2.5% and the adversarial accuracy decreases by 4.2% at a compression rate of 60%, which constitutes the upper limit of the technical effect of the conventional linear combination scheme; The coupling scheme proposed in this invention reduces the above two indicators to 1.2% and 2.1% respectively under the same compression ratio, and the search time is reduced from 1.2x to 0.9x. This nonlinear performance leap does not come from individual improvements in each step, but rather from the closed-loop feedback mechanism between steps S103 and S104: the detection result guides the initial direction of the game search, and the adversarial feedback generated during the game process corrects the redundant confidence of the detection. The two drive each other and are mutually causal within the same iteration cycle, forming a new optimization paradigm different from any linear combination scheme. It is difficult to produce the above synergistic effect by simply superimposing the existing methods.

[0051] S105. Prune the pre-trained model according to the target pruning structure to obtain a simplified model.

[0052] Based on the optimized target pruning structure, redundant layers or modules in the pre-trained model are removed to obtain a simplified model with fewer parameters. Simultaneously, to enhance the information flow within the model, step S105 also includes: In the pre-trained model, skip connections are added between different structural levels that have been pruned according to the target pruning structure to restore the topology of the pruned model.

[0053] S106. Using the original model as the teacher model and the simplified model as the student model, the simplified model is trained using a double knowledge distillation method to align the attention matrix of the student model with that of the teacher model. The output of the student model after training is then used as the pruning model corresponding to the Transformer model.

[0054] The dual knowledge distillation method enables the teacher model (the original, unpruned Transformer model with complete knowledge) and the student model (the pruned, simplified model that needs to learn the teacher's knowledge) to align their attention mechanisms through a loss function, ensuring that the semantic understanding ability of the pruned model does not degrade. To achieve the alignment of the attention matrix, the loss function used in this embodiment satisfies the following: KL divergence loss = Σ(Teacher attention distribution × log(Teacher distribution / Student distribution)).

[0055] The alignment of the attention matrix, combined with the addition of the skip connection structure in step S105, improves the recovery effect of the model after pruning.

[0056] The Transformer architecture neural network model pruned using the methods described above exhibits better optimization in terms of inference latency (differential evolution algorithm selects structures with worse latency performance for pruning), privacy protection (reducing structures with strong attention requirements), and adversarial attack defense (selecting structures with worse adversarial sensitivity performance for pruning). At the same time, the pruning method can also ensure the original accuracy performance and reduce model complexity.

[0057] To verify the technical effectiveness of the pruning method proposed in this invention, this invention also provides data from pruning experiments conducted on a BERT-base model using the GLUE benchmark set as the task. In this invention, the BERT-base model is used as the Transformer model to be pruned, and the GLUE benchmark task is used as the verification task. Comparison methods include L1 norm pruning, SparseGPT pruning, and differential evolution pruning without introducing a red-blue game mechanism. In the experiments, the model compression ratios were set to 50%, 60%, and 70%, respectively, and tests were conducted under the same training epochs, the same hardware platform, and the same batch size. Adversarial perturbations were generated using the Fast Gradient Sign Method (FGSM), and the perturbation boundary ε was set to 0.01. Evaluation metrics included the decrease in accuracy under normal conditions, the decrease in accuracy under adversarial attacks, and the reduction in inference latency. The performance comparison data of this invention and the existing methods used for comparison under different compression ratio conditions are shown in Table 2 below.

[0058] Table 2. Performance comparison results of various pruning methods under different compression ratios.

[0059] Correspondingly, please refer to Figure 3 , Figure 3 (a)-(e) in the table are line graphs corresponding to the data shown in Table 2. Figure 3 The pruning methods were compared across five dimensions, including the decrease in accuracy of conventional methods (…). Figure 3 (a) in the text, and the decrease in accuracy under adversarial attacks ( Figure 3 (b) of the above), the proportion of inference delay reduction ( Figure 3 (c) in the middle), robust gap ( Figure 3 (d) and the comprehensive deployment efficiency index ( Figure 3 (e)). Among them, the robustness gap is defined as the difference between the accuracy decrease under adversarial attacks and the normal accuracy decrease, which is used to characterize the additional performance loss generated by the model when subjected to adversarial perturbations; the comprehensive deployment efficiency index is defined as the ratio of the inference latency reduction ratio to the sum of the normal accuracy decrease and the accuracy decrease under adversarial attacks, which is used to measure the comprehensive balance between inference acceleration and performance maintenance of the pruned model.

[0060] From Table 2 and Figure 3 It can be seen that, under compression ratios of 50%, 60%, and 70%, the conventional accuracy reduction of the pruning method proposed in this embodiment is only 0.8%, 1.2%, and 2.4%, respectively, all lower than other comparative methods. Under FGSM adversarial attack conditions, the accuracy reduction of the pruning method proposed in this embodiment is 1.5%, 2.1%, and 4.0%, respectively, also significantly lower than other pruning methods. Especially at a compression ratio of 60%, the conventional accuracy reduction of the pruning method proposed in this embodiment is only 1.2%, lower than the 5.8% of the L1 norm pruning method, the 3.5% of the SparseGPT pruning method, and the 2.7% of the differential evolution pruning method; its accuracy reduction under adversarial attacks is only 2.1%, also lower than the 12.3% of the L1 norm pruning method, the 7.6% of the SparseGPT pruning method, and the 4.9% of the differential evolution pruning method. The above results show that the pruning method proposed in this embodiment can effectively maintain model accuracy at high compression ratios and significantly improve the stability of the pruned model under adversarial perturbations.

[0061] Furthermore, the pruning method proposed in this embodiment of the invention achieves inference latency reductions of 31.2%, 42.5%, and 53.6% at compression rates of 50%, 60%, and 70%, respectively, indicating that it can achieve significant inference acceleration while reducing model complexity. Meanwhile, the robustness gaps corresponding to the pruning method proposed in this embodiment of the invention are only 0.7%, 0.9%, and 1.6%, respectively, indicating that the model's performance difference under normal input and adversarial perturbation inputs is small, demonstrating good robust stability. In terms of the overall deployment efficiency index, the pruning method proposed in this embodiment of the invention achieves scores of 13.57, 12.88, and 8.38, respectively, all higher than other comparative methods, indicating that the pruning method proposed in this embodiment of the invention can achieve a better balance between model compression, inference efficiency, and accuracy preservation.

[0062] As the compression ratio increases from 50% to 70%, the reduction in inference latency for each pruning method shows an upward trend. However, traditional pruning methods are usually accompanied by significant loss of conventional accuracy and decreased adversarial robustness. In contrast, the pruning method proposed in this embodiment of the invention can still control the decrease in conventional accuracy to 2.4% and the decrease in accuracy under adversarial attacks to 4.0% under a high compression ratio of 70%, while achieving a 53.6% reduction in inference latency. This demonstrates that the pruning method proposed in this embodiment of the invention, through joint redundancy detection of time-domain data, frequency-domain data, cross-layer data, and adversarial-sensitive data, can more accurately distinguish between critical and redundant structures, reducing the probability of mistakenly pruning critical attention heads under high compression ratios. At the same time, through a red-blue dual-population game-based collaborative differential evolution mechanism, it can continuously screen for pruning structures that are more robust to perturbations during the pruning search process, so that the final simplified model has a high compression ratio, low inference latency, and strong adversarial robustness, making it more suitable for deployment on resource-constrained devices or in real-time inference scenarios.

[0063] The beneficial effect achieved by this invention lies in proposing a pruning method for neural network models based on the Transformer architecture, which has the following advantages: First, by simultaneously fusing time domain, frequency domain, cross-layer gradient, and adversarial sensitive data through dimensional redundancy detection, we avoid miscutting key structures from a single dimension. Second, by transforming the pruning structure search into an offensive and defensive adversarial optimization problem through red-blue dual-population game-based collaborative differential evolution, the final retained structure not only has a high compression rate but also naturally possesses adversarial robustness, thus solving the problem of the fragility of the traditional pruning model. Third, the alternating adversarial collaborative optimization mechanism can adaptively generate the most destructive interference, screen out the truly robust pruning structure, and ensure the robustness and accuracy of the pruned model in various application scenarios.

[0064] This invention also provides a Transformer model pruning system 200, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the Transformer model pruning structure of the system provided in this embodiment of the invention, which includes: Initialization module 201 is used to obtain the Transformer model to be pruned as the original model; The pre-training module 202 is used to pre-train the original model until it converges to obtain a pre-trained model, and to collect data from the pre-trained model to obtain multi-dimensional data including time domain data, frequency domain data, cross-layer data and adversarial sensitive data. The redundancy detection module 203 is used to obtain the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude based on the multi-dimensional data, and to calculate a comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score and the sensitivity, and to mark attention heads with a comprehensive redundancy score higher than a preset threshold as candidate pruning structures, and the rest as retained structures; The differential evolution module 204 is used to construct a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and to optimize it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies a preset fitness. The pruning module 205 is used to prune the pre-trained model according to the target pruning structure to obtain a simplified model. The knowledge distillation module 206 is used to learn the parameters of the simplified model using the original model as the teacher model and the simplified model as the student model, by using a dual knowledge distillation method to align the KL divergence loss between the attention matrices of the student model and the teacher model, and to align the cross-entropy loss between the output of the student model and the output of the teacher model. The output of the student model after learning is used as the pruned model corresponding to the Transformer model.

[0065] The Transformer model pruning system 200 can implement the steps in the Transformer model pruning method in the above embodiments and achieve the same technical effect, as described in the above embodiments, and will not be repeated here.

[0066] This invention also provides a computer device, please refer to... Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a Transformer model pruning program stored in the memory 302 and capable of running on the processor 301.

[0067] The processor 301 calls the Transformer model pruning program stored in the memory 302 and executes the steps in the Transformer model pruning method provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S101. Obtain the Transformer model to be pruned as the original model.

[0068] S102. Pre-train the original model until it converges to obtain a pre-trained model, and collect data from the pre-trained model to obtain multi-dimensional data including time domain data, frequency domain data, cross-layer data and adversarial sensitive data.

[0069] Specifically, the time-domain data includes: the weight matrix of each attention head in the pre-trained model, and the activation values ​​corresponding to the weight matrix; The frequency domain data includes: spectral features extracted by performing a fast Fourier transform on the time domain data; The cross-layer data includes: the inter-layer gradient propagation path between the middle layer and the attention head connection of the pre-trained model, and the activation intensity of the feedforward network; The adversarial sensitive data includes: the perturbation magnitude of the response of the mid-level and attention heads of the pre-trained model to Gaussian noise or fast gradient sign-based adversarial attacks during pre-training.

[0070] S103. Based on the multi-dimensional data, obtain the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude. Calculate the comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Mark attention heads with a comprehensive redundancy score higher than a preset threshold as candidate pruning structures, and mark the rest as retained structures.

[0071] Step S103 includes: The cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model are calculated based on the time-domain data and the frequency-domain data. The inter-layer score of the inter-layer gradient propagation path is calculated based on the cross-layer data; The sensitivity of each attention head to the amplitude of the disturbance is calculated based on the aforementioned adversarial sensitivity data; After normalizing the cosine similarity, spectral similarity, inter-layer score, and sensitivity, the geometric mean method is used to calculate the comprehensive redundancy score for each attention head. The comprehensive redundancy score is defined as Score and satisfies the following relationship: Score = (S_cos×S_spec×S_layer×S_adv)^{1 / 4}; Where S_cos is the normalized cosine similarity, S_spec is the normalized spectral similarity, S_layer is the normalized inter-layer score, and S_adv is the normalized sensitivity. Attention heads whose overall redundancy score is higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures.

[0072] S104. Construct a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and optimize it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies the preset fitness.

[0073] Step S104 includes the following sub-steps: The Blue Army population is constructed based on the candidate pruning structure, and the initial individuals of the population are selected based on the adversarial sensitive data to construct the Red Army population. Each individual in the Blue Army population is encoded as a pruning mask vector, which indicates whether to retain or remove each attention head in the candidate pruning structure. The perturbation patterns corresponding to the top K% of attention heads in the adversarial sensitive data are selected as the initial individuals of the Red Army population, and each individual in the Red Army population is encoded as a noise interference vector, which simulates perturbations against the pre-trained model. The current blue force population is fixed, and mutation and crossover operations are performed on the red force population based on the attention head corresponding to the preserved structure to generate attack interference targeting the key preserved structure; then, the attack fitness of each individual in the red force population is calculated according to the alternating adversarial cooperative differential evolution algorithm. The attack fitness is positively correlated with the accuracy decrease of the pre-trained model under interference. Based on the attack fitness, individuals in the red force population whose noise interference vector is greater than a preset interference threshold are selected to form an evolved red force population. The Red Army population is fixed, and the Blue Army population is subjected to mutation and crossover operations according to the alternating adversarial cooperative differential evolution algorithm. The pruned structure individuals with the slowest decrease in comprehensive redundancy score under the attack interference of the Red Army population are selected. At the same time, the defense fitness of each individual in the Blue Army population is calculated. The defense fitness is obtained by weighting the compression ratio of the pre-trained model, hardware latency, and robust accuracy under the individual attacks of the Red Army population. The attack fitness of the Red Army population and the defense fitness of the Blue Army population are calculated alternately in each generation iteration, forming a closed loop of mutual game and cooperative evolution between the Red and Blue sides. Iteratively perform crossover and mutation of the Red Army population and the Blue Army population until the rate of change of the defense fitness within a consecutive preset number of generations is less than a preset threshold. Then, output the candidate pruning structure corresponding to the individual in the current Blue Army population that satisfies the preset fitness as the target pruning structure.

[0074] The defense fitness is defined as It satisfies the following relationship: ; in, The compression ratio of the pre-trained model. The inference latency of the pre-trained model on the specified reference hardware platform. This represents the worst-case verification accuracy of the current blue team population under attack from the red team population.

[0075] S105. Prune the pre-trained model according to the target pruning structure to obtain a simplified model.

[0076] Step S105 also includes: In the pre-trained model, skip connections are added between different structural levels that are pruned according to the target pruning structure.

[0077] S106. Using the original model as the teacher model and the simplified model as the student model, the simplified model is trained using the double knowledge distillation method to align the KL divergence loss between the attention matrices of the student model and the teacher model, and to align the cross-entropy loss between the output of the student model and the output of the teacher model. The output of the student model after training is then used as the pruned model corresponding to the Transformer model.

[0078] The computer device 300 provided in this embodiment of the invention can implement the steps in the Transformer model pruning method as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0079] This invention also provides a storage medium storing a Transformer model pruning program. When executed by a processor, the Transformer model pruning program implements the various processes and steps of the Transformer model pruning method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0080] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by instructing related hardware (such as mobile phones, computers, servers, air conditioners, or network devices, etc.) through a Transformer model pruning program. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0081] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0082] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form without departing from the spirit and scope of the claims of the present invention, and all such changes are within the protection scope of the present invention.

Claims

1. A Transformer model pruning method, characterized in that, Includes the following steps: Obtain the Transformer model to be pruned as the original model; The original model is pre-trained until it converges to obtain a pre-trained model. Data is then collected from the pre-trained model to obtain multi-dimensional data, including time-domain data, frequency-domain data, cross-layer data, and adversarial-sensitive data. Based on the multi-dimensional data, the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude are obtained. A comprehensive redundancy score is calculated based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Attention heads with a comprehensive redundancy score higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures. A red-blue dual-population game model is constructed using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population. The model is then optimized using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies the preset fitness. The pre-trained model is pruned according to the target pruning structure to obtain a simplified model; Using the original model as the teacher model and the simplified model as the student model, the simplified model is trained using a double knowledge distillation method to align the KL divergence loss between the attention matrices of the student model and the teacher model, and to align the cross-entropy loss between the output of the student model and the output of the teacher model. The output of the student model after training is then used as the pruned model corresponding to the Transformer model.

2. The Transformer model pruning method according to claim 1, characterized in that, The time-domain data includes: the weight matrix of each attention head in the pre-trained model, and the activation values ​​corresponding to the weight matrix; The frequency domain data includes: spectral features extracted by performing a fast Fourier transform on the time domain data; The cross-layer data includes: the inter-layer gradient propagation path between the middle layer and the attention head connection of the pre-trained model, and the activation intensity of the feedforward network; The adversarial sensitive data includes: the perturbation magnitude of the response of the mid-level and attention heads of the pre-trained model to Gaussian noise or fast gradient sign-based adversarial attacks during pre-training.

3. The Transformer model pruning method according to claim 2, characterized in that, The steps include obtaining the cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model based on the multi-dimensional data, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude; calculating a comprehensive redundancy score based on the cosine similarity, spectral similarity, inter-layer score, and sensitivity; and marking attention heads with comprehensive redundancy scores higher than a preset threshold as candidate pruning structures, while marking the rest as retained structures. The cosine similarity and spectral similarity between different attention heads within the same layer of the pre-trained model are calculated based on the time-domain data and the frequency-domain data. The inter-layer score of the inter-layer gradient propagation path is calculated based on the cross-layer data; The sensitivity of each attention head to the amplitude of the disturbance is calculated based on the aforementioned adversarial sensitivity data; After normalizing the cosine similarity, spectral similarity, inter-layer score, and sensitivity, the geometric mean method is used to calculate the comprehensive redundancy score for each attention head. The comprehensive redundancy score is defined as Score and satisfies the following relationship: Score = (S_cos×S_spec×S_layer×S_adv)^{1 / 4}; Where S_cos is the normalized cosine similarity, S_spec is the normalized spectral similarity, S_layer is the normalized inter-layer score, and S_adv is the normalized sensitivity. Attention heads whose overall redundancy score is higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures.

4. The Transformer model pruning method according to claim 1, characterized in that, The steps of constructing a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and optimizing it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies a preset fitness, include the following sub-steps: The Blue Army population is constructed based on the candidate pruning structure, and the initial individuals of the population are selected based on the adversarial sensitive data to construct the Red Army population. Each individual in the Blue Army population is encoded as a pruning mask vector, which indicates whether to retain or remove each attention head in the candidate pruning structure. The perturbation patterns corresponding to the top K% of attention heads in the adversarial sensitive data are selected as the initial individuals of the Red Army population, and each individual in the Red Army population is encoded as a noise interference vector, which simulates perturbations against the pre-trained model. The current blue force population is fixed, and mutation and crossover operations are performed on the red force population based on the attention head corresponding to the preserved structure to generate attack interference targeting the key preserved structure; then, the attack fitness of each individual in the red force population is calculated according to the alternating adversarial cooperative differential evolution algorithm. The attack fitness is positively correlated with the accuracy decrease of the pre-trained model under interference. Based on the attack fitness, individuals in the red force population whose noise interference vector is greater than a preset interference threshold are selected to form an evolved red force population. The Red Army population is fixed, and the Blue Army population is subjected to mutation and crossover operations according to the alternating adversarial cooperative differential evolution algorithm. The pruned structure individuals with the slowest decrease in comprehensive redundancy score under the attack interference of the Red Army population are selected. At the same time, the defense fitness of each individual in the Blue Army population is calculated. The defense fitness is obtained by weighting the compression ratio of the pre-trained model, hardware latency, and robust accuracy under the individual attacks of the Red Army population. The attack fitness of the Red Army population and the defense fitness of the Blue Army population are calculated alternately in each generation iteration, forming a closed loop of mutual game and cooperative evolution between the Red and Blue sides. Iteratively perform crossover and mutation of the Red Army population and the Blue Army population until the rate of change of the defense fitness within a consecutive preset number of generations is less than a preset threshold. Then, output the candidate pruning structure corresponding to the individual in the current Blue Army population that satisfies the preset fitness as the target pruning structure.

5. The Transformer model pruning method according to claim 2, characterized in that, The step of pruning the pre-trained model according to the target pruning structure to obtain a simplified model further includes: In the pre-trained model, skip connections are added between different structural levels that are pruned according to the target pruning structure.

6. The Transformer model pruning method according to claim 4, characterized in that, The defense fitness is defined as It satisfies the following relationship: ； in, The compression ratio of the pre-trained model. The inference latency of the pre-trained model on the specified reference hardware platform. This represents the worst-case verification accuracy of the current blue team population under attack from the red team population.

7. A Transformer model pruning system, characterized in that, include: The initialization module is used to obtain the Transformer model to be pruned as the original model. The pre-training module is used to pre-train the original model until it converges to obtain a pre-trained model, and to collect data from the pre-trained model to obtain multi-dimensional data including time domain data, frequency domain data, cross-layer data and adversarial sensitive data. The redundancy detection module is used to obtain the cosine similarity and spectral similarity between different attention heads in the same layer of the pre-trained model, the inter-layer score of the inter-layer gradient propagation path, and the sensitivity of each attention head to the perturbation amplitude based on the multi-dimensional data. It calculates a comprehensive redundancy score based on the cosine similarity, the spectral similarity, the inter-layer score, and the sensitivity. Attention heads with a comprehensive redundancy score higher than a preset threshold are marked as candidate pruning structures, and the rest are marked as retained structures. The differential evolution module is used to construct a red-blue dual-population game model using the candidate pruning structure as the blue team population and the adversarial sensitive data as the red team population, and to optimize it using an alternating adversarial cooperative differential evolution algorithm to obtain a target pruning structure that satisfies a preset fitness. The pruning module is used to prune the pre-trained model according to the target pruning structure to obtain a simplified model. The knowledge distillation module is used to learn the parameters of the simplified model using the original model as the teacher model and the simplified model as the student model, by employing a dual knowledge distillation method. This aligns the KL divergence loss between the attention matrices of the student model and the teacher model, and aligns the cross-entropy loss between the outputs of the student model and the teacher model. The output of the student model after learning is then used as the pruned model corresponding to the Transformer model.

8. A computer device, characterized in that, include: The processor includes a memory, a processor, and a Transformer model pruning program stored in the memory and executable on the processor, wherein the processor, when executing the Transformer model pruning program, implements the steps of the Transformer model pruning method as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a Transformer model pruning program, which, when executed by a processor, implements the steps of the Transformer model pruning method as described in any one of claims 1-6.

Images