Neural network pruning system and method based on layer-by-layer pruning rate optimization

Abstract

The invention relates to network pruning, in particular to a neural network pruning system and method based on layer-by-layer pruning rate optimization, and the system comprises a modeling module which identifies a prunable layer of a neural network to be pruned and constructs a pruning rate optimization problem model; the evaluation module is used for providing search direction guidance for the global optimization module and supporting rapid fine adjustment of the neural network pruned by adopting the pruning rate vector; the global optimization module operates three mechanisms of an improved sparrow search algorithm ISSA, integrated double-role topology cooperation, enhanced lens imaging reverse learning based on stagnation monitoring and dynamic intensity modulation, and later local fine search, and solves the optimal pruning rate vector of the whole network for the pruning rate optimization problem model; the pruning execution module is used for performing unstructured pruning on a prunable layer of the neural network to be pruned according to the optimal pruning rate vector; the method can overcome the defects that scene adaptability and structural safety are lacked, and it is difficult to effectively achieve balance of high precision and high compression rate on the edge side.

Description

Technical Field

[0001] This invention relates to network pruning, and more specifically to a neural network pruning system and method based on layer-by-layer pruning rate optimization. Background Technology

[0002] Deep neural networks have made groundbreaking progress in fields such as computer vision, natural language processing, and complex industrial engineering predictions. However, as model performance improves, the number of parameters and computational complexity increase exponentially, posing a significant challenge to their deployment on resource-constrained devices such as mobile terminals, embedded sensors, and edge computing gateways. Particularly in the fields of industrial internet and intelligent manufacturing, tasks such as optimizing robot grinding and polishing parameters, predicting the strength of building materials, analyzing the aerodynamic performance of aircraft, and real-time assessing building energy efficiency often require real-time inference using embedded chips with limited computing power.

[0003] Network pruning, a mainstream model compression technique, aims to reduce model size and inference latency by removing redundant connections. Unstructured pruning, in particular, has attracted significant attention due to its high compression ratio. However, in practical industrial applications, existing network pruning techniques face serious challenges: 1) Lack of scene adaptability and structural security, and long R&D cycle: Industrial scenarios vary greatly, from image recognition on mobile devices to regression prediction in industrial sites. The model structures corresponding to different tasks are very different. Existing solutions usually rely on manual experience or cumbersome "test-label-re-prune" processes to set layer-by-layer pruning thresholds. However, when faced with new tasks, R&D personnel need to repeatedly try and fail to find suitable pruning parameters, resulting in an excessively long model adaptation cycle, which cannot meet the needs of rapid iteration in the industry. 2) Difficulty in balancing accuracy and efficiency on low-power devices: Existing solutions often have huge computational overhead, making it difficult to run directly on edge devices. Or, due to the use of uniform / coarse pruning strategies, they are prone to destroying the key feature extraction layer structure in the model (for example, when processing weak sensor vibration signals or aerodynamic data). This leads to a significant decrease in prediction accuracy while pursuing lightweight design, and problems such as "over-pruning and signal distortion" or "under-pruning and slow inference" are likely to occur, which cannot meet industrial-grade safety and real-time standards.

[0004] Therefore, there is an urgent need for a neural network pruning system and method that can automatically search for the globally optimal pruning strategy in a single run for different industrial scenarios, with low computational overhead and the ability to effectively achieve a balance between high precision and high compression rate at the edge. Summary of the Invention

[0005] (a) Technical problems to be solved In view of the above-mentioned shortcomings of the existing technology, the present invention provides a neural network pruning system and method based on layer-by-layer pruning rate optimization, which can effectively overcome the shortcomings of the existing technology, such as lack of scene adaptability and structural security, and difficulty in achieving a balance between high precision and high compression rate at the edge.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A neural network pruning system based on layer-by-layer pruning rate optimization includes a modeling module, an evaluation module, a global optimization module, and a pruning execution module. The modeling module identifies the pruningable layers of the neural network to be pruned and constructs a model for optimizing the pruning rate. The evaluation module provides search direction guidance for the global optimization module and supports rapid fine-tuning of the neural network after pruning using the pruning rate vector. The global optimization module runs the improved Sparrow Search Algorithm ISSA, which integrates three mechanisms: dual-role topological collaboration, enhanced lens imaging back learning based on stagnation monitoring and dynamic intensity modulation, and post-processing local fine search. In a single run, it directly solves the optimal pruning rate vector of the entire network for the pruning rate optimization problem model. The pruning execution module performs unstructured pruning on the pruningable layers of the neural network to be pruned based on the optimal pruning rate vector, generating a lightweight neural network adapted to edge devices with limited memory and computing power.

[0007] Preferably, the modeling module identifies the pruningable layers of the neural network to be pruned and constructs a pruning rate optimization problem model, including: Identify all prunable layers in a neural network, including convolutional layers and fully connected layers; In order to ensure the accuracy of the final classification or regression output mapping of the neural network, active optimization of the final output layer of the neural network is excluded when constructing the variables to be optimized.

[0008] Preferably, the modeling module constructs a pruning rate optimization problem model, including: The task of allocating pruning rates for each pruning layer is mapped to a pruning rate optimization problem model in an L-dimensional continuous space, and the variable to be optimized is defined as the pruning rate vector. ; in, Let l be the pruning rate of the l-th pruning layer, where l = 1, 2, ..., L, and L is the number of pruning layers. , , These are the minimum pruning rate, the maximum pruning rate, and the minimum pruning rate, respectively. This is used to ensure that the neural network can achieve basic parameter compression and avoid insignificant pruning effects; maximum pruning rate. This is used to prevent excessive pruning from causing a sharp decline in neural network performance and to ensure the usability of the pruned neural network.

[0009] Preferably, the evaluation module provides search direction guidance for the global optimization module, including: To minimize the fitness function As a guide, it directs the search direction of the global optimization module; the fitness function It can be expressed as follows: ; in, To use the pruning rate vector The accuracy of the pruned neural network on the validation set. To use the pruning rate vector The ratio of the total number of parameters in the pruned neural network to the total number of parameters in the original neural network. To balance the weighting coefficients of accuracy and compression ratio, Weighting coefficient Adjustments should be made based on the actual deployment environment of the neural network.

[0010] Preferably, the evaluation module supports rapid fine-tuning of the neural network after pruning using a pruning rate vector, including: To support high-frequency iterative evaluation within a single run, a pruning rate vector is used. The pruned neural network undergoes rapid fine-tuning in a preset number of rounds; After fine-tuning, obtain the accuracy of the fine-tuned neural network on the validation set. Substitute into the fitness function Calculate the pruning rate vector The fitness value provides a basis for updating the population position.

[0011] Preferably, the core component of the global optimization module is an improved Sparrow Search Optimization Engine (ISSA Engine) with a restructured architecture. This engine is custom-designed for the high-dimensional continuous space characteristics of the pruning rate optimization problem model, eliminating the need for manually preset layer-by-layer pruning thresholds. The improved Sparrow Search Optimization Engine (ISSA Engine) is executed collaboratively by three core modules: a dual-role topology collaboration module, a stagnation perception and reverse breakthrough module (ELLO), and a late-stage neighborhood fine mining module, to achieve full automation of the optimization process and single-time global convergence.

[0012] Preferably, the dual-role topology collaboration module performs lightweight reconstruction of the population topology, reconstructing the population topology into a two-level model of "discoverer" and "joiner", eliminating the "watcher" role in the standard sparrow search algorithm, so as to significantly reduce the floating-point operation and memory usage during algorithm operation, reduce computational redundancy, and improve search efficiency. The dual-role topology collaboration module drives the population to perform dynamic position updates based on fitness values ​​within an L-dimensional continuous solution space. Each dimension of the solution space corresponds to the pruning rate of a pruning layer. Through a lightweight two-level population topology structure, the population can achieve efficient exploration and information interaction within the solution space, laying the foundation for the search for the global optimal solution.

[0013] Preferably, the stagnation perception and reverse breakthrough module ELLO monitors the update frequency of the global optimal solution in real time. By counting the number of consecutive times the global optimal solution has not been updated, it determines whether the algorithm has entered a stagnation state. When the number of consecutive times without updating exceeds a preset stagnation threshold, the module automatically activates the enhanced lens imaging reverse learning mechanism. Based on the overall state of the current population, it dynamically calculates the reverse solution, forcing the population to jump out of the local optimal attraction domain, avoiding premature convergence of the algorithm, and ensuring the global exploration capability of the search process. Specifically, this includes: 1) By generating inverse solutions, the algorithm is ensured to effectively reflect from the currently stagnant solution space to a region of potential better solutions: ; Where X represents the original location of the sparrow individual, X opp For the reverse solution, g best X is the current globally optimal solution. mean L represents the average position of the current population, rand(dim) is a random vector with the same dimension as L, where each element is uniformly distributed in the range [0,1], and intensity is the dynamic learning intensity factor. 2) Dynamic Intensity Modulation: This is achieved by dynamically learning the intensity factor. The intensity factor is positively correlated with the count of consecutive unupdated global optimal solutions and negatively correlated with the iteration progress. This is to provide a larger reverse perturbation intensity in the early stages of the algorithm, enhancing its global exploration capability and helping the population traverse a wider solution space; and to reduce the reverse perturbation intensity in the later stages of the algorithm, avoiding disruption of individual solutions tending towards convergence and ensuring stable convergence of the algorithm. ; Among them, intensity base Here, is the initial learning intensity factor, and 'progress' is the iteration progress normalization coefficient. The closer it is to 1, the closer the algorithm is to convergence. t is the current iteration number, T is the maximum iteration number, and count' is the preset stall threshold.

[0014] Preferably, the late-stage neighborhood refinement module is automatically triggered when the iteration progress of a single run is within a preset progress interval, so as to perform local refinement optimization on the current global optimal solution in the later stage of the algorithm. This module first locks the currently found global optimal solution, and then applies small-step random perturbations that follow the corresponding distribution in its neighborhood to generate multiple candidate solutions and evaluate them. Through high-frequency local development, better solutions in the neighborhood are mined, improving the accuracy of the optimal pruning rate vector. Specifically, it includes: 1) Triggering condition: Automatically triggered when the normalization coefficient of the iteration progress is within the preset progress range. The preset progress range is set based on the algorithm convergence law to ensure that the module starts after the population has initially converged to a relatively optimal solution region. This avoids premature triggering that would render local development ineffective and allows sufficient iterations to complete local fine optimization. 2) Generation of small-step random perturbations: ; in, Let L be a random perturbation step size vector with the same dimension as L, ub and lb be the upper and lower bound vectors of the search space, respectively, N(0,1) be a random vector with the same dimension as L that follows a standard normal distribution to ensure the randomness and rationality of the perturbation, and k be the perturbation coefficient, k≤0.02, used to control the perturbation intensity; The later-stage neighborhood refinement module utilizes the generated random perturbation step size vector The current global optimal solution is perturbed with a small step size, and multiple candidate solutions are generated in its neighborhood and evaluated. A better pruning rate vector is selected and a better solution in the neighborhood is discovered.

[0015] The neural network pruning method based on layer-by-layer pruning rate optimization includes the following steps: S1. Identify all pruning layers in the neural network. Pruning layers include convolutional layers and fully connected layers. In order to ensure the accuracy of the final classification or regression output mapping of the neural network, when constructing the variables to be optimized, exclude the active optimization processing of the final output layer of the neural network. S2. Map the task of allocating pruning rates for each pruning layer into a pruning rate optimization problem model in an L-dimensional continuous space, defining the variable to be optimized as the pruning rate vector. ; S3. Initialize the parameters of the improved sparrow search algorithm ISSA and start the improved sparrow search optimization engine ISSAEngine; S4. The dual-role topological collaboration module is used to drive the population to perform dynamic position updates based on fitness values ​​in the L-dimensional continuous solution space, thereby achieving global exploration, reducing computational redundancy, and improving search efficiency. S5. During the iteration process, the global optimal solution update status is monitored in real time through the ELLO stagnation perception and reverse breakthrough module. When the algorithm gets stuck in a stagnation state, the enhanced lens imaging reverse learning mechanism is activated to generate a reverse solution. The dynamic learning intensity factor is used to force the population to jump out of the local optimal attraction domain, avoid premature convergence of the algorithm, and ensure the global exploration capability of the search process. S6. When the iteration progress reaches the later stage of the algorithm, the late-stage neighborhood fine mining module is automatically triggered. It applies small-step random perturbations that follow the corresponding distribution in the neighborhood of the current global optimal solution, performs high-precision local development, mines better solutions in the neighborhood, and improves the accuracy of the optimal pruning rate vector. S7. After a single run, output the optimal pruning rate vector. ; S8. Based on the optimal pruning rate vector Unstructured pruning of the prunable layers of the neural network: Calculate the L1 norm of the weight parameters of each prunable layer and sort all L1 norms in ascending order; then, based on the optimal pruning rate vector... Determine the pruning rate of each pruning layer. Based on the ranking of the pruning layers, mark the weight parameters with the lowest pruning rate as to be removed and construct a binary mask. Use the binary mask to remove redundant weight parameters while keeping the structure of the final output layer unchanged. Generate a lightweight neural network adapted to edge devices with limited memory and computing power, and export it for subsequent deployment.

[0016] (III) Beneficial Effects Compared with the prior art, the neural network pruning system and method based on layer-by-layer pruning rate optimization provided by the present invention has the following beneficial effects: 1) Achieve single-run global optimization without manually presetting layer-by-layer pruning thresholds: The complex task of allocating pruning rates for each pruning layer is transformed into a single-run global optimization task, completely abandoning the traditional mode of manually presetting layer-by-layer pruning thresholds, and achieving zero human intervention and automation in the entire model compression process. 2) Significantly improved optimization accuracy and algorithm robustness: By integrating stagnation perception and reverse breakthrough, and subsequent neighborhood fine mining mechanism, the problem of premature convergence of traditional swarm intelligence optimization algorithms is effectively solved; in benchmark function test, the average solution error of the present invention is reduced by 3 to 12 orders of magnitude compared with the standard algorithm, ensuring that when facing complex industrial tasks, the global optimal solution can be accurately locked and the local optimum can be avoided. 3) Balancing low accuracy loss with extremely high compression ratio: This invention not only achieves a compression ratio of 84.81% with an accuracy loss of only 0.59% in basic tasks such as MNIST-LeNet-5, but also demonstrates excellent structural security in deep network tasks such as VGG-16. Experiments have shown that even under extreme conditions with a compression ratio exceeding 88%, this invention can still automatically identify and preserve the connection paths of key layers of the model, effectively preventing the catastrophic failure of "layer collapse" common in existing technologies (such as global amplitude pruning), and ensuring that the model still has industrial-grade predictive accuracy under extreme compression ratios. 4) Possesses strategy self-discovery capability and strong generalization: This invention can autonomously discover the "shallow conservative and deep aggressive" pruning distribution characteristics that conform to the feature extraction mechanism of neural networks. In various engineering regression tasks such as building energy efficiency prediction, the mean square error (MSE) of the pruned model is as low as 0.008681, which verifies the strong versatility of this invention in different industrial scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram illustrating the principle of optimizing the layer-by-layer pruning rate of the neural network using the improved sparrow search algorithm ISSA in this invention; Figure 3 This is a point map showing the population diversity distribution during the search process of the Standard Sparrow Search Algorithm (SSA). Figure 4 This is a point map showing the population diversity distribution of the improved sparrow search algorithm ISSA during the search process in this invention. Figure 5 This is a line graph comparing the population diversity distribution of the improved sparrow search algorithm ISSA and the standard sparrow search algorithm SSA during the search process. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] This embodiment takes the deployment of a deep learning model in a resource-constrained edge computing scenario (such as an embedded controller or mobile terminal in an industrial field) as an example. This system runs on a computing platform equipped with a regular CPU or a low-power AI chip, aiming to solve the technical challenge of traditional methods failing to quickly acquire high-precision, lightweight neural networks under low computing power conditions.

[0021] The following describes the specific functional modules and technical effects of the neural network pruning system based on layer-by-layer pruning rate optimization provided by this invention, using concrete examples. The system functional modules include: a modeling module, an evaluation module, a global optimization module, and a pruning execution module. The modeling module identifies the pruningable layers of the neural network to be pruned and constructs a model for optimizing the pruning rate. The evaluation module provides search direction guidance for the global optimization module and supports rapid fine-tuning of the neural network after pruning using the pruning rate vector. The global optimization module runs the improved Sparrow Search Algorithm ISSA, which integrates three mechanisms: dual-role topological collaboration, enhanced lens imaging back learning based on stagnation monitoring and dynamic intensity modulation, and post-processing local fine search. In a single run, it directly solves the optimal pruning rate vector of the entire network for the pruning rate optimization problem model. The pruning execution module performs unstructured pruning on the pruningable layers of the neural network to be pruned based on the optimal pruning rate vector, generating a lightweight neural network adapted to edge devices with limited memory and computing power.

[0022] I. Modeling Module The modeling module identifies the pruningable layers of the neural network to be pruned and constructs a model for optimizing the pruning rate, including: Identify all prunable layers in a neural network, including convolutional layers and fully connected layers; In order to ensure the accuracy of the final classification or regression output mapping of the neural network, active optimization of the final output layer of the neural network is excluded when constructing the variables to be optimized.

[0023] The modeling module constructs a model for the pruning rate optimization problem, including: The task of allocating pruning rates for each pruning layer is mapped to a pruning rate optimization problem model in an L-dimensional continuous space, and the variable to be optimized is defined as the pruning rate vector. ; in, Let l be the pruning rate of the l-th pruning layer, where l = 1, 2, ..., L, and L is the number of pruning layers. , , These are the minimum pruning rate, the maximum pruning rate, and the minimum pruning rate, respectively. This is used to ensure that the neural network can achieve basic parameter compression and avoid insignificant pruning effects. This indicates that the minimum pruning ratio for each pruning layer is 20%; the maximum pruning rate... This is used to prevent excessive pruning from causing a sharp decline in neural network performance and to ensure the usability of the pruned neural network. This indicates that the maximum pruning ratio for each pruning layer is 90%.

[0024] II. Evaluation Module The evaluation module provides search direction guidance for the global optimization module, including: To minimize the fitness function As a guide, it directs the search direction of the global optimization module; the fitness function It can be expressed as follows: ; in, To use the pruning rate vector The accuracy of the pruned neural network on the validation set. To use the pruning rate vector The ratio of the total number of parameters in the pruned neural network to the total number of parameters in the original neural network. To balance the weighting coefficients of accuracy and compression ratio, Weighting coefficient Adjustments can be made based on the actual deployment environment of the neural network. For example, for battery-powered sensors in the field (which are sensitive to energy consumption), the weighting coefficients can be appropriately reduced. To achieve a higher compression ratio; for flaw detection equipment (sensitive to accuracy), the weighting coefficient should be appropriately increased. In exchange for higher precision.

[0025] The evaluation module supports rapid fine-tuning of neural networks pruned using pruning rate vectors, including: To support high-frequency iterative evaluation within a single run, a pruning rate vector is used. After pruning, the neural network undergoes rapid fine-tuning for a preset number of rounds (no less than 3 rounds) (the default parameter configuration is: learning rate of 0.01 and batch size of 128). After fine-tuning, obtain the accuracy of the fine-tuned neural network on the validation set. Substitute into the fitness function Calculate the pruning rate vector The fitness value provides a basis for updating the population position.

[0026] III. Global Optimization Module The core component of the global optimization module is an improved Sparrow Search Optimization Engine (ISSAEngine) that has undergone architectural restructuring, such as... Figure 2 As shown, the engine is custom-designed for the high-dimensional continuous space characteristics of the pruning rate optimization problem model, eliminating the need for manual pre-setting of layer-by-layer pruning thresholds; The improved Sparrow Search Optimization Engine (ISSA Engine) is executed collaboratively by three core modules: a dual-role topology collaboration module, a stagnation perception and reverse breakthrough module (ELLO), and a late-stage neighborhood fine mining module. This enables full automation of the optimization process and single-time global convergence.

[0027] Dual-role topology collaboration module Considering the memory and computing power limitations of edge devices, the dual-role topology collaboration module performs lightweight reconstruction of the population topology, reconstructing the population topology into a two-level model of "discoverer" and "joiner", eliminating the "watcher" role in the standard sparrow search algorithm, so as to significantly reduce the floating-point operation and memory usage during algorithm operation, reduce computational redundancy, and improve search efficiency. The dual-role topology collaboration module drives the population to perform dynamic position updates based on fitness values ​​within an L-dimensional continuous solution space. Each dimension of the solution space corresponds to the pruning rate of a pruning layer. Through a lightweight two-level population topology structure, the population can achieve efficient exploration and information interaction within the solution space, laying the foundation for the search for the global optimal solution.

[0028] ELLO Module for Stagnation Perception and Reverse Breakthrough To address the issue of traditional algorithms easily getting trapped in local optima due to the complex and highly unstructured characteristics of industrial data, the ELLO stagnation detection and reverse breakthrough module monitors the update frequency of the global optimum in real time. By counting the number of consecutive times the global optimum has not been updated, it determines whether the algorithm has stagnated. When the number of consecutive times without an update exceeds a preset stagnation threshold, the module automatically activates the enhanced lens imaging reverse learning mechanism. Based on the overall state of the current population, it dynamically calculates the reverse solution, forcing the population to escape the local optimum attraction domain, avoiding premature convergence, and ensuring the global exploration capability of the search process. Specifically, this includes: 1) By generating inverse solutions, the algorithm is ensured to effectively reflect from the currently stagnant solution space to a region of potential better solutions: ; Where X represents the original location of the sparrow individual, X opp For the reverse solution, g best X is the current globally optimal solution. mean L represents the average position of the current population, rand(dim) is a random vector with the same dimension as L, where each element is uniformly distributed in the range [0,1], and intensity is the dynamic learning intensity factor. 2) Dynamic Intensity Modulation: This is achieved through dynamically learning the intensity factor. The intensity factor is positively correlated with the count of consecutive unupdated global optimal solutions and negatively correlated with the iteration progress. In the early stages of the algorithm (when progress is small), a larger reverse perturbation intensity is applied to enhance the algorithm's global exploration capability and help the population traverse a wider solution space. In the later stages of the algorithm (when progress is large), the reverse perturbation intensity is reduced to avoid disrupting individual solutions that tend towards convergence, thus ensuring stable convergence of the algorithm. ; Among them, intensity base The initial learning intensity factor is set to 0.1, and progress is the iteration progress normalization coefficient. The closer it is to 1, the closer the algorithm is to convergence. t is the current iteration number, T is the maximum iteration number (default is 200), and count' is the preset stall threshold.

[0029] Later-stage neighborhood fine mining module To meet the stringent requirements of industrial applications for model accuracy (such as Top-1 accuracy or mean squared error, MSE), a later-stage neighborhood refinement module is automatically triggered when the iteration progress of a single run falls within a preset progress range. This allows for local refinement of the current global optimum in the later stages of the algorithm (e.g., progress > 0.6). This module first identifies the current global optimum, then applies small-step random perturbations following a corresponding distribution to its neighborhood, generating and evaluating multiple candidate solutions. Through high-frequency local refinement, it uncovers better solutions within the neighborhood, improving the accuracy of the optimal pruning rate vector. Specifically, this includes: 1) Triggering conditions: Automatically triggered when the normalization coefficient of the iteration progress is within the preset progress range (50%~70%). The preset progress range is set based on the algorithm convergence law to ensure that the module starts after the population has initially converged to a better solution region, which avoids premature triggering that would lead to ineffective local development, and reserves sufficient iterations to complete local fine optimization. 2) Generation of small-step random perturbations: ; in, is a random perturbation step size vector with the same dimension as L, ub and lb are the upper bound (default value 0.9) and lower bound vector (default value 0.2) of the search space, respectively, N(0,1) is a random vector with the same dimension as L that follows a standard normal distribution to ensure the randomness and rationality of the perturbation, k is the perturbation coefficient, k≤0.02, used to control the perturbation intensity; The later-stage neighborhood refinement module utilizes the generated random perturbation step size vector The current global optimal solution is perturbed with a small step size, and multiple candidate solutions are generated in its neighborhood and evaluated. A better pruning rate vector is selected and a better solution in the neighborhood is discovered.

[0030] Based on the aforementioned neural network pruning system based on layer-by-layer pruning rate optimization, this application also discloses a neural network pruning method based on layer-by-layer pruning rate optimization, such as... Figure 1 As shown, it includes the following steps: S1. Identify all pruning layers in the neural network. Pruning layers include convolutional layers and fully connected layers. In order to ensure the accuracy of the final classification or regression output mapping of the neural network, when constructing the variables to be optimized, exclude the active optimization processing of the final output layer of the neural network. S2. Map the task of allocating pruning rates for each pruning layer into a pruning rate optimization problem model in an L-dimensional continuous space, defining the variable to be optimized as the pruning rate vector. ; S3. Initialize the parameters of the improved sparrow search algorithm ISSA (including population size, stagnation threshold, etc.) and start the improved sparrow search optimization engine ISSA Engine; S4. The dual-role topological collaboration module is used to drive the population to perform dynamic position updates based on fitness values ​​in the L-dimensional continuous solution space, thereby achieving global exploration, reducing computational redundancy, and improving search efficiency. S5. During the iteration process, the global optimal solution update status is monitored in real time through the ELLO stagnation perception and reverse breakthrough module. When the algorithm gets stuck in a stagnation state, the enhanced lens imaging reverse learning mechanism is activated to generate a reverse solution. The dynamic learning intensity factor is used to force the population to jump out of the local optimal attraction domain, avoid premature convergence of the algorithm, and ensure the global exploration capability of the search process. S6. When the iteration progress reaches the later stage of the algorithm, the late-stage neighborhood fine mining module is automatically triggered. It applies small-step random perturbations that follow the corresponding distribution in the neighborhood of the current global optimal solution, performs high-precision local development, mines better solutions in the neighborhood, and improves the accuracy of the optimal pruning rate vector. S7. After a single run, output the optimal pruning rate vector. ; S8. Based on the optimal pruning rate vector Unstructured pruning of the prunable layers of the neural network: Calculate the L1 norm of the weight parameters of each prunable layer and sort all L1 norms in ascending order; then, based on the optimal pruning rate vector... Determine the pruning rate of each pruning layer. Based on the ranking of the pruning layers, mark the weight parameters with the lowest pruning rate as to be removed and construct a binary mask. Use the binary mask to remove redundant weight parameters while keeping the structure of the final output layer unchanged. Generate a lightweight neural network adapted to edge devices with limited memory and computing power, and export it for subsequent deployment.

[0031] To verify the effectiveness and superiority of this invention, this embodiment constructs a comprehensive testing system covering everything from underlying algorithm benchmarks, basic vision tasks, deep network structure security to industrial engineering regression. The test results are as follows: I. Algorithm Optimization Performance Benchmark Test Before applying the improved Sparrow Search Algorithm ISSA to pruning, the performance of the improved ISSA Engine itself was first verified on eight standard benchmark functions. Experiments show that, thanks to its lightweight two-level population topology and stagnation awareness and back-breakthrough mechanisms, the ISSA algorithm of this invention achieves a 3-12 order of magnitude improvement in solution accuracy on complex multimodal functions compared to the standard Sparrow Search Algorithm SSA, and its convergence speed is improved by approximately 40%. This means that when facing the high-dimensional non-convex optimization problem of neural network pruning, this invention possesses the underlying mathematical capability to lock in the global optimum (rather than getting trapped in local suboptimal solutions).

[0032] II. Basic Visual Classification Task (MNIST-LeNet-5): In the classic MNIST handwritten digit recognition task, the LeNet-5 network is pruned using the ISSA algorithm of this invention, and compared with SSA and other commonly used optimization algorithms (such as PSO and GWO). After pruning, the model is retrained for 20 rounds. The experimental results are shown in Table 1 below: Table 1. Optimization results of pruning rate (MNIST-LeNet-5)

[0033] As shown in Table 1, the ISSA algorithm of this invention achieves a compression rate of up to 84.81% while only decreasing the Top-1 accuracy by 0.59%, significantly outperforming SSA and other commonly used optimization algorithms. It achieves the best performance in the accuracy-compression rate tradeoff, meaning that the neural network pruned using the ISSA algorithm of this invention can be easily deployed on IoT terminals with minimal memory. Figures 3 to 5As shown, the ISSA of this invention effectively maintains the diversity of the population during the search process (the index is much higher than that of SSA) through the stagnation perception and reverse breakthrough mechanism, avoids premature convergence, and ensures the global optimality of the search results.

[0034] III. Security and Collapse Resistance Verification of Deep Network Structures (ResNet-20 and VGG-16) To verify the robustness of this invention in processing deep overparameterized networks, extreme compression tests were conducted on ResNet-20 and VGG-16 networks on the CIFAR-10 dataset, with a focus on comparing the ISSA of this invention, manually set uniform pruning, and the industry-standard global amplitude pruning.

[0035] VGG-16: Under extreme conditions where the remaining parameters are only 11.39% (i.e., a compression ratio as high as 88.61%), traditional global amplitude pruning, unable to perceive the dependencies between layers, incorrectly removes all connections in key layers, causing the model to "collapse" and the accuracy to plummet to 10.00% (i.e., completely ineffective, equivalent to random guessing). In contrast, the ISSA of this invention can automatically identify and preserve the connection paths of key layers without manual intervention. The experimental results are shown in Table 2 below. Under the same extreme compression ratio, ISSA maintains the accuracy at 86.21%, effectively avoiding the risk of "collapse" and ensuring the usability of the model under extreme compression.

[0036] ResNet-20: With a parameter surplus of 11.47% (i.e., a compression rate of approximately 88.53%), the ISSA algorithm of this invention demonstrates superior optimization capabilities compared to the manual uniform pruning strategy. Experimental results are shown in Table 2 below. The ISSA-optimized model achieves an accuracy of 86.70%, higher than the 86.46% of uniform pruning. This proves that in deep network tasks where collapse does not occur, this invention can automatically find a pruning rate distribution that is better than human experience.

[0037] Table 2 Comparison of the ultimate compression performance of deep networks (CIFAR-10 dataset)

[0038] As shown in Table 2, in the VGG-16 task, both the ISSA proposed in this invention and uniform pruning can maintain an accuracy of over 86%, while global amplitude pruning suffers a catastrophic "hierarchical collapse" (accuracy of only 10.00%). This proves that ISSA has a safety mechanism to identify and avoid "collapse traps" during automatic global optimization. In the ResNet-20 task, the accuracy of the ISSA proposed in this invention (86.70%) surpasses that of uniform pruning (86.46%), proving that this invention has the ability to finely mine the optimal solution in the non-collapse interval.

[0039] IV. Verification of the Generalization Ability of Industrial Engineering Prediction Tasks To verify the universality of this invention in the fields of intelligent construction and aerospace, the standard BP neural network was pruned and optimized on the following three engineering datasets: concrete strength prediction - simulating on-site material testing in construction; airfoil aerodynamic performance - simulating aerodynamic analysis in aircraft design; and building energy efficiency assessment - simulating energy consumption monitoring in intelligent buildings. The experimental results (mean squared error, MSE) are shown in Table 3 below: Table 3 Optimization results of pruning rate (Engineering dataset - BP)

[0040] Table 3 shows that in three different engineering tasks, the ISSA proposed in this invention achieved a lower mean square error compared to SSA, demonstrating its superior optimization accuracy. Particularly in the energy dataset, ISSA achieved the best prediction accuracy with the smallest parameter reduction rate (15.80%). This proves that the invention can automatically adapt to the data characteristics of different industrial scenarios and output an optimal model that balances accuracy and efficiency without human intervention, greatly reducing the threshold for implementing industrial AI models.

[0041] In summary, this invention achieves efficient compression in basic tasks through a single global optimization, solves the problem of hierarchical collapse in deep networks under extreme compression, and has strong generalization ability in industrial scenarios.

[0042] More importantly, without introducing any artificial prior rules, the optimal pruning rate vector ultimately output by the ISSA Engine of this invention is... It exhibits a pruning distribution characteristic of "shallow conservatism and deep radicalism," and this characteristic highly matches the intrinsic mechanism of industrial signal processing, namely: Assigning a low pruning rate to the feature extraction layer near the input layer in the neural network, with a value range of [0.2, 0.5], in industrial applications, this strategy can ensure that the key features of the original signal (such as weak vibration waveforms, airflow textures, etc.) are not destroyed, effectively preventing signal distortion; Assigning a higher pruning rate to the semantic layer near the output layer in the neural network, with values ​​ranging from [0.6, 0.9], effectively compresses redundant parameters at higher levels responsible for logical judgments, significantly reducing inference latency and storage space consumption, and directly addressing the pain point of limited resources on edge devices. The feature extraction layer refers to the pruning layer located on the network input side, and the semantic layer refers to the pruning layer located on the network output side; each accounts for approximately 50% of the total number of pruning layers.

[0043] The pruning distribution characteristics of "shallow conservatism and deep radicalism" are highly compatible with the information processing mechanism of neural networks, achieving an optimal balance between accuracy and compression rate, thus proving the intelligence of this invention.

[0044] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network pruning system based on layer-by-layer pruning rate optimization, characterized in that: It includes a modeling module, an evaluation module, a global optimization module, and a pruning execution module; The modeling module identifies the pruningable layers of the neural network to be pruned and constructs a model for optimizing the pruning rate. The evaluation module provides search direction guidance for the global optimization module and supports rapid fine-tuning of the neural network after pruning using the pruning rate vector. The global optimization module runs the improved Sparrow Search Algorithm ISSA, which integrates three mechanisms: dual-role topological collaboration, enhanced lens imaging back learning based on stagnation monitoring and dynamic intensity modulation, and post-processing local fine search. In a single run, it directly solves the optimal pruning rate vector of the entire network for the pruning rate optimization problem model. The pruning execution module performs unstructured pruning on the pruningable layers of the neural network to be pruned based on the optimal pruning rate vector, generating a lightweight neural network adapted to edge devices with limited memory and computing power.

2. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 1, characterized in that: The modeling module identifies the pruningable layers of the neural network to be pruned and constructs a pruning rate optimization problem model, including: Identify all prunable layers in a neural network, including convolutional layers and fully connected layers; In order to ensure the accuracy of the final classification or regression output mapping of the neural network, active optimization of the final output layer of the neural network is excluded when constructing the variables to be optimized.

3. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 2, characterized in that: The modeling module constructs a model for the pruning rate optimization problem, including: The task of allocating pruning rates for each pruning layer is mapped to a pruning rate optimization problem model in an L-dimensional continuous space, and the variable to be optimized is defined as the pruning rate vector. ; in, Let l be the pruning rate of the l-th pruning layer, where l = 1, 2, ..., L, and L is the number of pruning layers. , , These are the minimum pruning rate, the maximum pruning rate, and the minimum pruning rate, respectively. This is used to ensure that the neural network can achieve basic parameter compression and avoid insignificant pruning effects; maximum pruning rate. This is used to prevent excessive pruning from causing a sharp decline in neural network performance and to ensure the usability of the pruned neural network.

4. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 1, characterized in that: The evaluation module provides search direction guidance for the global optimization module, including: To minimize the fitness function As a guide, it directs the search direction of the global optimization module; the fitness function It can be expressed as follows: ； in, To use the pruning rate vector The accuracy of the pruned neural network on the validation set. To use the pruning rate vector The ratio of the total number of parameters in the pruned neural network to the total number of parameters in the original neural network. To balance the weighting coefficients of accuracy and compression ratio, Weighting coefficient Adjustments should be made based on the actual deployment environment of the neural network.

5. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 4, characterized in that: The evaluation module supports rapid fine-tuning of neural networks pruned using pruning rate vectors, including: To support high-frequency iterative evaluation within a single run, a pruning rate vector is used. The pruned neural network undergoes rapid fine-tuning in a preset number of rounds; After fine-tuning, obtain the accuracy of the fine-tuned neural network on the validation set. Substitute into the fitness function Calculate the pruning rate vector The fitness value provides a basis for updating the population position.

6. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 1, characterized in that: The core component of the global optimization module is an improved sparrow search optimization engine, ISSA Engine, which has been restructured. This engine is customized for the high-dimensional continuous space characteristics of the pruning rate optimization problem model, eliminating the need for manual preset of layer-by-layer pruning thresholds. The improved Sparrow Search Optimization Engine (ISSA Engine) is executed collaboratively by three core modules: a dual-role topology collaboration module, a stagnation perception and reverse breakthrough module (ELLO), and a late-stage neighborhood fine mining module, to achieve full automation of the optimization process and single-time global convergence.

7. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 6, characterized in that: The dual-role topology collaboration module performs lightweight reconstruction of the population topology, reconstructing the population topology into a two-level model of "discoverer" and "joiner", eliminating the "watcher" role in the standard sparrow search algorithm, thereby significantly reducing the floating-point operation and memory usage during algorithm operation, reducing computational redundancy, and improving search efficiency. The dual-role topology collaboration module drives the population to perform dynamic position updates based on fitness values ​​within an L-dimensional continuous solution space. Each dimension of the solution space corresponds to the pruning rate of a pruning layer. Through a lightweight two-level population topology structure, the population can achieve efficient exploration and information interaction within the solution space, laying the foundation for the search for the global optimal solution.

8. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 7, characterized in that: The stagnation perception and reverse breakthrough module ELLO monitors the update frequency of the global optimal solution in real time. By counting the number of consecutive times the global optimal solution has not been updated, it determines whether the algorithm has entered a stagnation state. When the number of consecutive times without updating exceeds a preset stagnation threshold, the module automatically activates the enhanced lens imaging reverse learning mechanism. Based on the overall state of the current population, it dynamically calculates the reverse solution, forcing the population to jump out of the local optimal attraction domain, avoiding premature convergence of the algorithm, and ensuring the global exploration capability of the search process. Specifically, this includes: 1) By generating inverse solutions, the algorithm is ensured to effectively reflect from the currently stagnant solution space to a region of potential better solutions: ； Where X represents the original location of the sparrow individual, X opp For the reverse solution, g best X is the current globally optimal solution. mean L represents the average position of the current population, rand(dim) is a random vector with the same dimension as L, where each element is uniformly distributed in the range [0,1], and intensity is the dynamic learning intensity factor. 2) Dynamic Intensity Modulation: This is achieved by dynamically learning the intensity factor. The intensity factor is positively correlated with the count of consecutive unupdated global optimal solutions and negatively correlated with the iteration progress. This is to provide a larger reverse perturbation intensity in the early stages of the algorithm, enhancing its global exploration capability and helping the population traverse a wider solution space; and to reduce the reverse perturbation intensity in the later stages of the algorithm, avoiding disruption of individual solutions tending towards convergence and ensuring stable convergence of the algorithm. ； Among them, intensity base Here, is the initial learning intensity factor, and 'progress' is the iteration progress normalization coefficient. The closer it is to 1, the closer the algorithm is to convergence. t is the current iteration number, T is the maximum iteration number, and count' is the preset stall threshold.

9. The neural network pruning system based on layer-by-layer pruning rate optimization according to claim 8, characterized in that: The late-stage neighborhood refinement module is automatically triggered when the iteration progress of a single run is within a preset progress interval. This allows for local refinement of the current global optimum in the later stages of the algorithm. The module first locks the currently found global optimum, then applies small-step random perturbations following a corresponding distribution to its neighborhood, generating multiple candidate solutions and evaluating them. Through high-frequency local development, it mines better solutions within the neighborhood, improving the accuracy of the optimal pruning rate vector. Specifically, this includes: 1) Triggering condition: Automatically triggered when the normalization coefficient of the iteration progress is within the preset progress range. The preset progress range is set based on the algorithm convergence law to ensure that the module starts after the population has initially converged to a relatively optimal solution region. This avoids premature triggering that would render local development ineffective and allows sufficient iterations to complete local fine optimization. 2) Generation of small-step random perturbations: ； in, Let L be a random perturbation step size vector with the same dimension as L, ub and lb be the upper and lower bound vectors of the search space, respectively, N(0,1) be a random vector with the same dimension as L that follows a standard normal distribution to ensure the randomness and rationality of the perturbation, and k be the perturbation coefficient, k≤0.02, used to control the perturbation intensity; The later-stage neighborhood refinement module utilizes the generated random perturbation step size vector The current global optimal solution is perturbed with a small step size, and multiple candidate solutions are generated in its neighborhood and evaluated. A better pruning rate vector is selected and a better solution in the neighborhood is discovered.

10. A neural network pruning method based on layer-by-layer pruning rate optimization, applied to the neural network pruning system based on layer-by-layer pruning rate optimization as described in claim 1, characterized in that: Includes the following steps: S1. Identify all pruning layers in the neural network. Pruning layers include convolutional layers and fully connected layers. In order to ensure the accuracy of the final classification or regression output mapping of the neural network, when constructing the variables to be optimized, exclude the active optimization processing of the final output layer of the neural network. S2. Map the task of allocating pruning rates for each pruning layer into a pruning rate optimization problem model in an L-dimensional continuous space, defining the variable to be optimized as the pruning rate vector. ; S3. Initialize the parameters of the improved sparrow search algorithm ISSA and start the improved sparrow search optimization engine ISSA Engine; S4. The dual-role topological collaboration module is used to drive the population to perform dynamic position updates based on fitness values ​​in the L-dimensional continuous solution space, thereby achieving global exploration, reducing computational redundancy, and improving search efficiency. S5. During the iteration process, the global optimal solution update status is monitored in real time through the ELLO stagnation perception and reverse breakthrough module. When the algorithm gets stuck in a stagnation state, the enhanced lens imaging reverse learning mechanism is activated to generate a reverse solution. The dynamic learning intensity factor is used to force the population to jump out of the local optimal attraction domain, avoid premature convergence of the algorithm, and ensure the global exploration capability of the search process. S6. When the iteration progress reaches the later stage of the algorithm, the late-stage neighborhood fine mining module is automatically triggered. It applies small-step random perturbations that follow the corresponding distribution in the neighborhood of the current global optimal solution, performs high-precision local development, mines better solutions in the neighborhood, and improves the accuracy of the optimal pruning rate vector. S7. After a single run, output the optimal pruning rate vector. ; S8. Based on the optimal pruning rate vector Unstructured pruning of the prunable layers of the neural network: Calculate the L1 norm of the weight parameters of each prunable layer and sort all L1 norms in ascending order; then, based on the optimal pruning rate vector... Determine the pruning rate of each pruning layer, mark the weight parameters of the pruning rates that are ranked lower as to be removed according to the ranking of the pruning layers, and construct a binary mask. By using binary masks to remove redundant weight parameters while keeping the structure of the final output layer unchanged, a lightweight neural network adapted to edge devices with limited memory and computing power is generated and exported for subsequent deployment.

Images