A method for evaluating structural redundancy of a feedforward neural network

Abstract

The application relates to a structural redundancy explainable evaluation method of a feedforward neural network, and belongs to the technical field of deep learning explainability and model lightweight. The application measures the task contribution degree of a single neuron by quantifying the linear separability of the output feature of the single neuron to a task target, thereby accurately identifying a redundant neuron; then, redundancy is quantified from a local level to an entire model through controllable layer-by-layer pruning experiments and multidimensional index calculation. The application first generalizes the linear separability measurement from a classification task to a regression scene, proposes an index, and uses the index as a universal contribution degree measurement standard which can be compared across models and datasets, and the evaluation result has good explainability; the application can automatically identify a redundant neuron, can effectively relieve the overfitting problem of an over-parameterized model, can significantly reduce the storage and calculation overhead of the model, and can comprehensively depict the structural redundancy characteristics of a neural network.

Description

Technical Field

[0001] This invention belongs to the field of deep learning interpretability and model lightweighting technology, and relates to an interpretable evaluation method for the structural redundancy of feedforward neural networks, particularly an interpretable evaluation method for the structural redundancy of feedforward neural networks based on regression linear separability. Background Technology

[0002] Deep neural networks have achieved tremendous success in fields such as computer vision, natural language processing, recommender systems, and scientific computing. However, as model performance improves, network structures become increasingly complex, with a dramatic increase in the number of parameters and computational costs. This not only increases the storage and computational overhead of the models but also poses challenges to their deployment on resource-constrained devices such as mobile devices, embedded systems, and edge computing nodes. Particularly in regression tasks, such as house price prediction, energy consumption estimation, industrial control, autonomous driving path planning, and financial time series forecasting, models are often deployed on edge devices or mobile devices with limited computing power and storage. The inference latency and energy consumption issues caused by structural redundancy are particularly prominent. Regression tasks refer to machine learning tasks that predict continuous numerical outputs, as opposed to classification tasks that predict discrete category labels. Inference latency refers to the time required for the model to complete one forward propagation computation. For example, in real-time path planning regression models in autonomous driving, the inference latency directly determines the vehicle's reaction speed; excessive latency can lead to safety hazards. Edge sensors in the Industrial Internet of Things (IIoT) need to run regression models locally to predict device operating status, but these devices typically have computing power at the ARM Cortex-M level and less than 1MB of memory, making it impossible to run original, large-parameter models.

[0003] Research indicates that trained neural networks contain significant parameter redundancy. Frankle and Carbin's Lottery Ticket Hypothesis suggests that a randomly initialized dense network may contain a subnetwork (a smaller, sparsified network structure), which, when trained alone, can achieve performance comparable to the original network. Mukherjee and Huberman further point out that the strong generalization ability of overparameterized networks (networks with a parameter count far exceeding the minimum required for training) can be explained by an "expansion-sparseness" mechanism. A key phenomenon is that the performance of a trained neural network does not significantly decrease after pruning a large number of parameters. These theoretical perspectives provide important theoretical support for the study of neural network structural redundancy. Parameter redundancy refers to the existence of parameters or neuronal structures in the network that do not substantially contribute to the model's task performance and can be removed without significantly affecting performance.

[0004] Neural network redundancy assessment is a key issue in the fields of deep learning interpretability and model lightweighting. By quantitatively assessing the structural redundancy of a network, we can: (1) identify redundant neurons and parameters to guide model pruning and compression; (2) understand the internal working mechanism of the network and provide theoretical guidance for network structure design; and (3) establish interpretable assessment criteria to achieve horizontal comparison between different models. Among them, neurons are the basic computational units of neural networks, receiving input signals, performing weighted summation, and generating output through activation functions; model pruning refers to the technique of removing redundant parameters or neurons in the network to reduce the model size.

[0005] Currently, most existing methods for evaluating neural network redundancy are designed for classification tasks. These methods typically assess the importance of neurons based on metrics such as classification accuracy and feature separability. However, in regression tasks, the target value is continuous, making it impossible to directly apply the evaluation methods used for classification tasks. There is currently a significant gap in the development of quantitative evaluation methods for redundancy metrics specifically designed for regression neural networks. Feature separability refers to the property of whether samples of different classes can be effectively distinguished in the feature space; classification accuracy refers to the proportion of samples correctly predicted by the classification model out of the total number of samples.

[0006] Furthermore, most existing pruning methods require manual setting of pruning ratios or thresholds, lacking adaptability and interpretability. Users find it difficult to intuitively understand which neurons are redundant and the impact of pruning on model performance. Therefore, there is an urgent need for an interpretable structural redundancy assessment method for regressive neural networks, capable of accurately locating redundant neurons, achieving lightweight model compression, and providing interpretable evaluation metrics. Adaptability refers to the method's ability to automatically adjust parameters based on the specific characteristics of the data and model without manual intervention; interpretability means that the evaluation results have clear physical meaning and can be intuitively understood by humans. Summary of the Invention

[0007] The technical problems this invention aims to solve are: most existing neural network redundancy assessment methods are designed for classification tasks and cannot be directly applied to regression tasks; existing methods require manual setting of pruning parameters and lack adaptability; the assessment results lack interpretability, making it difficult to intuitively understand the task contribution of neurons; and there is a lack of a unified, cross-model comparable redundancy quantification standard. To address these technical problems, this invention proposes an interpretable assessment method for the structural redundancy of feedforward neural networks. Here, a feedforward neural network refers to a neural network structure where information propagates unidirectionally from the input layer through hidden layers to the output layer, with no feedback connections between layers; deep learning interpretability refers to the research direction of understanding and explaining the internal working mechanisms and decision-making processes of neural networks; and model lightweighting refers to methods that reduce the number of model parameters and computational load through compression, pruning, and other techniques. The core idea of ​​this invention is to measure the task contribution of a single neuron by quantifying the linear separability of its output features to the task objective, thereby accurately identifying redundant neurons; and then, through controlled layer-by-layer pruning experiments and multi-dimensional index calculations, to quantify redundancy from the local level to the entire model system. Linear separability refers to the property that two sets of data points in the feature space can be completely separated by a hyperplane (i.e., a linear decision boundary in a high-dimensional space); task contribution refers to the quantitative indicator of the degree of contribution of a single neuron to the overall task performance of the model.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An interpretable evaluation method for the structural redundancy of a feedforward neural network, the method comprising the following steps: Step 1: Construct a trained feedforward neural network model and dataset, define the network structure parameters and regression training dataset, and obtain a standardized model and data representation; specifically: Step 1.1, Define the feedforward neural network structure: Given a trained L-layer feedforward neural network model, where L is the total number of hidden layers (i.e., the total number of computational layers between the input and output layers), and l is the layer index with l = 1, 2, ..., L. The number of neurons in the l-th layer is denoted as . ,in It is a positive integer. The input layer dimension is... The output layer has a dimension of 1 or S, where 1 represents a regression task (outputting a single continuous value), and S represents a classification task (outputting the probabilities of S categories), where S is the number of categories. The input layer is the network layer that receives the original data features; the output layer is the network layer that produces the final prediction result; and the hidden layer is the intermediate computational layer located between the input and output layers, performing feature transformations.

[0009] Step 1.2, Define the forward propagation process: The weight matrix of the l-th layer of the feedforward neural network model is denoted as... The bias term is denoted as The nonlinear activation function is denoted as f(·). Here, the weight matrix is ​​a learnable parameter matrix connecting neurons in layer (l-1) and layer l, with each element representing the connection strength between the two neurons; the bias term is a constant offset parameter in the neuron, used to adjust the input threshold of the activation function; the nonlinear activation function is a function that performs a nonlinear transformation on the weighted sum of the neurons, enabling the network to fit complex nonlinear relationships. For dimensional input samples The forward propagation process of the l-th layer is defined as follows: (1) in, This is the output of the l-th layer (i.e., a vector composed of the activation values ​​of all neurons in that layer). For the first Layer output, input layer Forward propagation refers to the process by which input data is processed layer by layer from the input layer through each hidden layer to the output layer. It is the basic method of inference and computation in neural networks.

[0010] Step 1.3, Define the regression training dataset: Given a regression training dataset , where M is the number of training samples (i.e., the total number of samples contained in the training set). Let be the input feature vector of the i-th sample. Let be the regression target value (i.e., the continuous true value to be predicted) for the i-th sample. The range of the regression target value is defined as [a, b], where a is the minimum target value in the training set, and b is the maximum target value in the training set. The training dataset refers to the set of labeled samples used to train the model parameters; each sample contains input features and the corresponding target output.

[0011] Step 2: Based on the feedforward neural network model and training dataset from Step 1, input the training dataset into the model for forward propagation, extract the output features of each neuron in each hidden layer, and obtain the feature representation of the neuron on the training data; specifically: Step 2.1, Forward Propagation to Extract Activation Values: Based on the feedforward neural network model constructed in Step 1 and the regression training dataset Then, input all samples from the training set into the model in sequence, execute the forward propagation process defined in step 1.2, and record the output activation value of each neuron in each hidden layer on all training samples.

[0012] Step 2.2, construct the neuron output feature vector: For the j-th neuron in the l-th hidden layer, where The output activation values ​​of the neuron on the M training samples are arranged in sample order to form the output feature vector of the neuron. ,in, Let i be the output activation value of the i-th sample on this neuron. The purpose of this step is to obtain the feature representation of each neuron on the training data, providing a data foundation for subsequent contribution metric measurement. The output feature vector is a one-dimensional vector composed of the activation values ​​of a single neuron across all training samples, reflecting the neuron's response pattern to different input samples.

[0013] Step 3: Based on the neuron output features extracted in Step 2, the linear separability measure based on Minkowski difference is extended from the classification scenario to the regression scenario. The regression linear separability (RLS) index for each neuron is calculated, yielding the regression linear separability value for each neuron. Specifically: Linear separability measure based on Minkowski difference The Minkowski Difference-based Linear Separability Measure (MD-LSM) is a method that quantifies the linear separability of two point sets by calculating the Minkowski difference. It was first proposed by Zhang et al. in 2023 and has three excellent properties: absolute measurement (the value range is fixed at [0.5,1], independent of data scale), insensitivity to outliers (separability is determined by the overall set distribution), and affine invariance (affine transformations such as translation, scaling, and rotation do not change the measurement value). The Minkowski difference is a set operation. For two sets A and B, the Minkowski difference is defined as AB={ab|a∈A,b∈B}, which is the set composed of the differences of all pairs of elements in the two sets. The Regression Linear Separability (RLS) is an index proposed in this invention that extends the linear separability measure from classification tasks to regression tasks. It is used to measure the degree of linear separability between the output features of neurons and the regression target, and the value range is [0.5,1].

[0014] Step 3.1, Target Value Range Division: Based on the regression target value range [a, b] determined in Step 1.3, the interval [a, b] is divided into N equally spaced partition points, with a partition step size of h = (ba) / (N+1). The nth partition point is... Where n = 1, 2, ..., N, and N is the number of partition points. The number of partition points N determines the accuracy of the integral approximation. When N ≥ 8, the calculation result has converged (the change is less than 0.001), so the default value is N = 10, which strikes a balance between accuracy and computational efficiency. Equal interval partitioning refers to dividing the value interval into several sub-intervals with equal step sizes; partition points refer to the boundary points used to divide the intervals; integral approximation refers to a numerical method that approximates the continuous integral by discrete summation.

[0015] Step 3.2, Binary Subtask Decomposition: For each split point The output feature set of the j-th neuron in the l-th layer is split into two subsets based on the target value: the subset with a target value less than 10 ... subset of and the target value is greater than or equal to subset of In this way, the continuous regression task is transformed into a binary classification subtask (i.e., a classification task with only two classes) at each split point. The binary classification subtask refers to a classification problem that divides samples into two mutually exclusive classes, where the two classes correspond to samples with target values ​​below and above the split point, respectively.

[0016] Step 3.3, Binary classification linear separability calculation: For each split point Calculate two subsets and Linear separability measure between them based on Minkowski difference The Minkowski difference is defined as: for two sets... , Its Minkowski difference is ,in Let A be the Minkowski difference elements, and B be the first set and B be the second set. The i-th element in the first set For the j-th element in the second set, It is an n-dimensional real space.

[0017] Based on this, the linear separability measure for binary classification Defined as: (2) Where I and J are the number of samples in sets A and B, respectively, ω is the projection direction vector (i.e., the direction vector used to project high-dimensional data into one-dimensional space), and P(·) is the probability (proportion) function (i.e., calculating the proportion of elements that satisfy the conditions to the total number of elements). It is the transpose of ω. The value range is [0.5, 1]. =1 indicates that the two sets are completely linearly separable along the optimal projection direction, and the closer the value is to 0.5, the weaker the linear separability. To improve computational efficiency, the mean difference approximation method is used to solve for the optimal projection direction ω, which significantly reduces computational overhead while maintaining accuracy. Here, the optimal projection direction is the direction vector that maximizes the separation of the two sets in the projected one-dimensional space; transpose refers to the row and column interchange operation of a vector or matrix.

[0018] Step 3.4, Calculate the RLS integral: Divide the N partition points The average value (i.e., the discrete integral approximation) is used to obtain the regression linear separability (RLS) of the neuron: (3) in, Let be the output feature vector of the j-th neuron, and y be the regression target value vector. The RLS value ranges from [0.5, 1]. The closer RLS is to 1, the stronger the linear correlation between the neuron's output feature and the regression target, and the greater its contribution to the regression task; the closer RLS is to 0.5, the smaller the neuron's contribution to the regression task, and the more likely it is a redundant neuron. Theoretically, RLS is... Natural generalization in continuous label scenarios: When the regression target value degenerates into two discrete categories, the RLS integral degenerates into a binary classification. Value, i.e., the value of the classification scenario. It is a special case of RLS, thus theoretically unifying the contribution measurement framework for classification and regression tasks. Here, linear correlation refers to the degree of linear relationship between two variables; discrete category refers to the category label taking a finite number of discrete values.

[0019] Step 4: Based on the RLS index obtained in Step 3, the task contribution of each neuron is obtained through exponential mapping normalization, resulting in a neuron contribution distribution with values ​​ranging from [0,1]. Specifically: Based on the RLS value of each neuron calculated in step 3, it is normalized to the [0,1] interval using an exponential mapping function to obtain the task contribution of each neuron. The purpose of exponential mapping is to widen the gap between high-contribution neurons and low-contribution neurons, facilitating subsequent redundant identification. The formula for calculating the task contribution is: (4) in, The contribution of the j-th neuron to the task, with a value range of [0,1]. α represents the regression linear separability of the j-th neuron; α is the scaling factor (i.e., a hyperparameter controlling the steepness of the exponential mapping function), used to adjust the steepness of the contribution distribution, and α=2.5 is the empirical optimal value. The closer the value is to 1, the greater the contribution of that neuron to the task. The closer a value is to 0, the more likely the neuron is to be redundant. Hyperparameters are parameters that are manually set before model training and are not learned from training data; empirical optimal values ​​are parameter values ​​that achieve the best results through extensive experimental verification.

[0020] The design of the exponential mapping originates from the softmax concept in information theory (softmax is a function that normalizes any real-valued vector into a probability distribution, amplifying the differences between elements through exponential transformation). It enhances the discriminative power of contribution through nonlinear transformation, making the difference between high-contribution neurons and low-contribution neurons more significant, thereby improving the accuracy of subsequent pruning. For classification tasks, RLS in formula (3) can be replaced with Multi-Class Linear Separability Measure (MLSM). MLSM measures the difference between all class pairs... The value is obtained by weighted averaging the top V maximum values, where V is the logarithm of the categories. Nonlinear transformation refers to function transformations that do not satisfy the superposition principle, and can introduce nonlinear expressive power; multi-class linear separability measure refers to a linear separability measurement method applicable to multi-class classification tasks.

[0021] Step 5: Based on the neuron task contribution obtained in Step 4, two complementary pruning criteria are designed to prune each hidden layer layer by layer, identifying and deleting redundant neurons to obtain the pruned network structure. The two pruning criteria are dynamic threshold pruning and K-means clustering adaptive pruning, which can be selected according to actual needs. Specifically: Layer-by-layer pruning refers to independently pruning each hidden layer, determining which neurons to retain or delete based on the contribution distribution of neurons in that layer. Dynamic threshold pruning dynamically calculates the pruning threshold based on the statistical characteristics of each layer's neuron contribution (such as minimum value and standard deviation), allowing for flexible control of the pruning intensity through hyperparameters. K-means clustering adaptive pruning automatically clusters neurons into high-contribution and low-contribution classes using the K-means clustering algorithm, eliminating the need for manual setting of pruning parameters. K-means clustering is an unsupervised learning algorithm that divides data points into K clusters, minimizing the sum of distances from each data point to its cluster center. Unsupervised learning refers to machine learning methods that do not require labeling and learn solely based on the inherent structure of the data.

[0022] Step 5.1, Dynamic Threshold Pruning: The pruning threshold is dynamically calculated based on the statistical characteristics of the contribution of neurons in each layer, achieving adaptive pruning for different layers. For the l-th hidden layer, the minimum contribution of all neurons in that layer is first calculated. and standard deviation ,in This represents the average contribution of neurons in layer l. Then, the dynamic pruning threshold is calculated: (5) in, The dynamic pruning threshold for layer l (i.e., the critical value for determining whether a neuron is preserved) is λ, a hyperparameter used to control the pruning ratio. A larger λ indicates a higher pruning ratio; λ can take different values ​​such as 0.5, 0.7, and 0.9. The standard deviation is a statistic measuring the dispersion of data, equal to the square root of the variance; the mean is the arithmetic average of all data values. Finally, for layer l, the retention contribution... Neurons, delete contribution The system uses neurons as a baseline and dynamically determines the threshold based on the minimum contribution of each layer, combined with the standard deviation. This avoids the bias caused by using a uniform threshold across the entire network and achieves adaptive pruning for different layers. The uniform threshold refers to using the same pruning critical value for all layers, without considering the differences in contribution distribution across different layers.

[0023] Step 5.2, K-means clustering adaptive pruning method: The K-means clustering algorithm automatically clusters neurons into two classes: high contribution and low contribution, without requiring manual setting of pruning parameters, achieving fully adaptive redundancy recognition. For the contribution of all neurons in the l-th hidden layer... Treating it as a one-dimensional feature, the K-means clustering algorithm is used to cluster it into K clusters, where The goal of K-means clustering is to minimize the sum of squares within each cluster, i.e. ,in For the k-th cluster, The cluster center of the k-th class, Here, the sum of squares within a cluster refers to the sum of the squared distances from all data points within a cluster to its cluster center, which is the optimization objective of the K-means algorithm; the cluster center refers to the mean of all data points in a cluster, representing the central location of that cluster. This indicates the floor operation, which takes the largest integer not greater than the given number.

[0024] Then, sort the clusters by their center values ​​from largest to smallest, starting with the highest-contribution clusters and adding them sequentially until the percentage of retained neurons reaches the target retention ratio p (default p=0.5), and at least [number missing] neurons are retained. Clusters with smaller cluster centers are classified as low-contribution classes, and their neurons are considered redundant and deleted. Clusters with larger cluster centers are classified as high-contribution classes, and their neurons are retained. The advantage of K-means clustering adaptive pruning is that it does not require manual adjustment of the pruning ratio hyperparameter; it can automatically determine the pruning threshold based on the distribution of contribution, resulting in better objectivity and greater ease of use. The target retention ratio refers to the proportion of neurons expected to be retained out of the total number of neurons in that layer; high-contribution clusters are those with larger cluster center values ​​and higher average neuron contribution.

[0025] Step 6: Based on the pruning results of Step 5, reconstruct the lightweight feedforward neural network model and fine-tune its training. Inherit the weights and bias parameters of the neurons in the original model to obtain a lightweight model with restored performance. Specifically: Step 6.1, Model Reconstruction: Based on the layer-by-layer pruning results in Step 5, remove the deleted neurons and their corresponding connection weights, and construct a new lightweight feedforward neural network structure. For each retained neuron, inherit its corresponding weight matrix columns and bias terms from the original model; for connections between adjacent layers, only weight connections where both ends are retained neurons are retained. The reconstructed model has significantly reduced parameters and computational cost. Here, model reconstruction refers to the process of rebuilding the network topology and parameter matrix based on the pruning results; a lightweight model refers to a compressed model with fewer parameters and less computational cost than the original model.

[0026] Step 6.2, Fine-tuning Training: The pruned model is fine-tuned using the Adam optimizer for 30 epochs, employing a cosine annealing learning rate scheduling strategy with pre-heating to restore model performance. The Adam optimizer is an adaptive learning rate optimization algorithm that combines the advantages of momentum and RMSprop, adaptively adjusting the learning rate of each parameter based on the first and second moments of the gradient. Fine-tuning refers to continuing training with a smaller learning rate based on the pre-trained model parameters to restore or improve performance. An epoch is a training unit that traverses the entire training dataset once. The cosine annealing learning rate scheduling strategy with pre-heating involves linearly increasing the learning rate from 0 to the initial value in the first few epochs (warm-up phase), and then gradually decreasing it according to a cosine function curve (annealing phase). The purpose of fine-tuning is to compensate for the performance loss caused by pruning, allowing the lightweight model to maintain or even improve task performance while reducing parameters and computational cost. Inheriting the original weights instead of random initialization significantly accelerates fine-tuning convergence and avoids a sharp performance drop. Here, convergence speed refers to the number of iterations required for the loss function to decrease to a stable value during model training; random initialization refers to the method of initializing model parameters with random values.

[0027] Step 7: Based on the pruning results of Step 5, calculate the hierarchical redundancy of each hidden layer to obtain the local hierarchical redundancy distribution; specifically: Step 7.1, Hierarchical Redundancy Calculation: Based on the layer-by-layer pruning results in Step 5, calculate the hierarchical redundancy of each hidden layer. The hierarchical redundancy of layer l is defined as the proportion of pruned redundant neurons in that layer to the total number of neurons in that layer: (6) in, This represents the layer redundancy of layer l (i.e., the proportion of redundant neurons in this layer to the total number of neurons). This represents the total number of original neurons in layer l. This represents the number of neurons retained in layer l after pruning. The value range is [0,1], with a larger value indicating higher redundancy in that layer. The hierarchical redundancy index can intuitively reflect the differences in redundancy distribution across different hidden layers, revealing the core distribution pattern that "deep layers have significantly higher redundancy than shallow layers," providing theoretical guidance for network structure design. Specifically, local hierarchical redundancy refers to the redundancy measure of a single hidden layer, corresponding to the global redundancy of the entire model; the core distribution pattern refers to the phenomenon that redundancy increases with increasing layer depth across different network structures and datasets.

[0028] Step 8: Based on the pruned lightweight model obtained in Step 6, calculate three global redundancy evaluation metrics to comprehensively characterize the structural redundancy characteristics of the model from three complementary dimensions: parameter size, computational cost, and task performance; specifically: Step 8.1, Parameter redundancy. ): Defined as the ratio of the number of parameters in the pruned model to the number of parameters in the original model: (7) in, The total number of parameters in the original model (i.e., the total number of all learnable parameters in the model, including weights and biases). This represents the total number of parameters in the model after pruning. The value range is [0,1]. The smaller the value, the higher the degree of parameter compression and the greater the parameter redundancy of the network. For example... =0.3 indicates that only 30% of the original model's parameters were retained after pruning, meaning that 70% of the original network's parameters were redundant structures that did not contribute substantially to the task. The number of parameters directly determines the model's storage footprint, memory pressure during training, and computational overhead for parameter updates, and is a fundamental dimension for evaluating the redundancy of neural network structures. Parameter compression refers to the proportion of removed parameters to the original number of parameters; storage footprint refers to the amount of storage space occupied by the model parameters on disk or in memory.

[0029] Step 8.2, FLOPs redundancy, ): Defined as the ratio of the floating-point computation cost of the pruned model to the floating-point computation cost of the original model: (8) in, The floating-point computational complexity of the original model ( , short for Floating Point Operations, refers to the total number of floating-point operations required for a model to complete one full forward inference operation. This represents the floating-point computation cost of the pruned model. The value range is [0,1]. A smaller value indicates a higher degree of computational compression and greater computational redundancy in the network. Floating-point operations more directly reflect the actual runtime complexity and hardware performance requirements of the model. For a multilayer perceptron composed of fully connected layers... This refers to a typical feedforward neural network model consisting of an input layer, multiple fully connected hidden layers, and an output layer. Since the number of parameters and computational cost of each neuron are in a fixed ratio, therefore... Among them, computational compression refers to the proportion of computational cost removed compared to the original computational cost; time complexity refers to the trend of algorithm runtime as the input size increases; and a fully connected layer is a network layer in which every neuron in the layer is connected to all neurons in the next layer.

[0030] Step 8.3, Accuracy retention ratio ): Used to examine whether pruning operations impair the model's task performance.

[0031] For classification tasks, Defined as the ratio of the classification accuracy of the pruned model to the classification accuracy of the original model: (9) in, This represents the classification accuracy of the original model on the test set. This represents the classification accuracy of the pruned model on the test set. The test set is a collection of independent samples used to evaluate the model's generalization performance and is not used in model training.

[0032] For regression tasks, due to the mean squared error (MSE), The mean of the squared differences between the predicted and actual values ​​is a commonly used performance metric in regression tasks. The smaller the value, the better the performance. Defined as the inverse ratio of the mean squared error of the original model to the mean squared error of the pruned model: (10) in, This represents the mean squared error of the original model on the test set. This represents the mean squared error of the pruned model on the test set.

[0033] The indicator has a clear physical meaning: when The result indicates that the pruned model outperforms the original model, demonstrating that removing redundant structures has a regularization effect (regularization is a technique to prevent model overfitting by constraining model complexity to improve generalization ability; overfitting refers to the phenomenon that a model performs well on the training set but poorly on the test set), effectively alleviating the overfitting problem of overparameterized models. A value of 1 indicates that the performance is completely unaffected after pruning, achieving ideal redundancy removal; This indicates that pruning caused a minor performance loss within an acceptable range; This indicates that excessive pruning has impaired the model's ability to extract core features.

[0034] In this step, the three metrics constitute a complete, unified, and interpretable quantitative evaluation standard for neural network structure redundancy from the three dimensions of storage overhead, computational overhead, and task performance.

[0035] The beneficial effects of this invention are as follows: (1) Theoretical innovation. For the first time, the linear separability measure was extended from classification tasks to regression scenarios, proposing the regression linear separability measure (…). The RLS index fills the theoretical gap in the evaluation of redundancy in regression neural networks. Inheriting the three excellent properties of MD-LSM—absolute metric, insensitivity to outliers, and affine invariance—it can serve as a universal contribution metric that can be compared across models and datasets.

[0036] (2) High interpretability. The task contribution is measured based on the linear separability of neuron output features and regression target, which has clear physical meaning. Users can intuitively understand the degree of contribution of each neuron to the task, as well as the basis for pruning, and the evaluation results have good interpretability.

[0037] (3) Adaptive pruning. The proposed K-means clustering adaptive pruning method does not require manual parameter adjustment and can automatically identify redundant neurons, avoiding the subjectivity and debugging costs caused by manually setting the pruning ratio.

[0038] (4) Regularization effect. Experimental results show that the pruning method proposed in this invention can reduce the test error by more than 20% on narrow and shallow network structures, showing a significant regularization effect and effectively alleviating the overfitting problem of overparameterized models.

[0039] (5) High-efficiency compression. With almost no loss of regression performance, it can achieve more than 70% compression of parameters and computation, significantly reducing the storage and computational overhead of the model and facilitating the deployment of the model on resource-constrained devices.

[0040] (6) The evaluation system is complete. A system including local hierarchical redundancy has been constructed. and global indicators ( The complete quantitative evaluation system of neural networks can achieve multi-level redundancy evaluation from micro-neurons to macro-models, and can comprehensively characterize the structural redundancy characteristics of neural networks. Attached Figure Description

[0041] Figure 1 This is a flowchart of the overall process for an interpretable evaluation method of structural redundancy in a feedforward neural network according to the present invention. The process is divided into a three-stage progressive architecture: stage one is the calculation of neuron task contribution, stage two is the layer-by-layer pruning and model fine-tuning, and stage three is the quantitative evaluation of redundancy. It fully demonstrates all the execution steps from model and dataset preparation, neuron feature extraction, RLS index calculation, neuron task contribution calculation, redundant neuron layer-by-layer pruning, lightweight model reconstruction and fine-tuning, to the calculation and evaluation of hierarchical redundancy and global redundancy indices.

[0042] Figure 2 The diagram shows the validation set loss convergence curves for five MLP regression models, illustrating the training convergence process of different MLP models on the California housing price dataset. The horizontal axis represents the number of training epochs, and the vertical axis represents the mean squared error (MSE) of the validation set. The five curves correspond to the five model structures: MLP-128-64-32, MLP-256-128-64, MLP-256-128-64-32, MLP-512-256-128-64, and MLP-1024-512-256-128-64. All models converged within 150 epochs, validating the stability of the model training. The validation set refers to a subset of samples from the training data used to monitor the training process and adjust hyperparameters. Convergence refers to the state where the model loss function decreases to a stable value and no longer changes significantly.

[0043] Figure 3 To compare the compression performance and performance retention of each model at λ=0.9, a combination of bar charts and line graphs is used: the blue bars represent the parameter compression ratio (i.e., 1-NUM), the orange line represents the performance retention ratio (ACC), and the red dashed line represents the performance baseline of the original model. The horizontal axis represents the five experimental model structures. As can be seen from the graph, at λ=0.9, the parameter compression ratio of all models exceeds 70%, with the large model MLP-1024-512-256-128-64 exhibiting the highest compression ratio. Regarding the performance retention ratio, the ACC of narrow, shallow networks (such as MLP-128-64-32) even exceeds 1, demonstrating the performance improvement brought about by regularization, while the ACC of the large models is slightly lower but still within an acceptable range.

[0044] Figure 4The changes in global redundancy indices for each model under different pruning parameters λ are shown in three subplots: (a) the change in parameter retention ratio NUM with λ, (b) the change in computational retention ratio FLO with λ, and (c) the change in performance retention ratio ACC with λ. The horizontal axis represents the pruning parameter λ (values ​​of 0.5, 0.7, and 0.9), and the vertical axis represents the corresponding global index values. The graphs show that as λ increases, NUM and FLO decrease monotonically (compression ratio increases), while ACC gradually decreases (performance loss increases). When λ = 0.7, most models maintain over 97% performance while compressing approximately 57% of the parameters, achieving a good balance between compression ratio and performance. The red dashed line represents the original model performance baseline (ACC = 1).

[0045] Figure 5 This is a heatmap showing the redundancy R_l of each hidden layer under different pruning methods for each model, comprising five subplots (a)-(e), corresponding to five MLP model structures. The horizontal axis of each subplot represents the hidden layer index (layer 1 to the deepest layer), and the vertical axis represents the pruning method (λ=0.5, λ=0.7, λ=0.9, clustering). The color intensity indicates the level redundancy R_l (the darker the color, the higher the redundancy), and the cells are labeled with specific values. The heatmap clearly shows the core distribution pattern of "deeper layers having higher redundancy than shallower layers," and the trend of higher redundancy as λ increases. The redundancy distribution of the K-means clustering method is similar to that of the λ=0.7 dynamic threshold method.

[0046] Figure 6 A heatmap showing the number of neurons retained in each hidden layer under different pruning methods for each model is presented, comprising five sub-plots (a)-(e), corresponding to five MLP model structures. The horizontal axis of each sub-plot represents the hidden layer index, and the vertical axis represents the pruning method. The cells indicate the number of neurons retained in that layer, with color intensity indicating the number retained. This heatmap complements the hierarchical redundancy heatmap in Figure 5, visually demonstrating the neuron retention status of each layer after pruning. It can be used to guide the structural design of lightweight models: shallow layers need to retain more neurons to extract basic features, while deeper layers can significantly compress the number of neurons. Detailed Implementation

[0047] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.

[0048] Example 1: Redundancy Assessment Method for Feedforward Neural Networks Based on Dynamic Threshold Pruning This embodiment uses a Multilayer Perceptron (MLP) regression model as an example to explain the specific implementation steps of the present invention. This embodiment uses the California Housing Dataset, which contains 20,640 samples, 8 input features, and the target value is the median house price in the region. The model to be evaluated is a 4-layer MLP regression model with an 8-dimensional input layer, 256-128-64-32 neurons in the hidden layers, and a 1-dimensional output layer. The activation function in the hidden layers is ReLU (Rectified Linear Unit, a commonly used non-linear activation function defined as...). The output layer has linear activation. Model training uses a stochastic gradient descent (SGD) optimizer with momentum, an initial learning rate of 0.01, momentum of 0.9, and a weight decay coefficient. The batch size is 256, training lasts 150 epochs, and the learning rate scheduling uses a cosine annealing strategy with preheating (5 epochs). Weight decay is a... Regularization techniques constrain parameter size by adding a sum of squared weights to the loss function; batch size refers to the number of samples used for each parameter update; momentum is an improvement on SGD that accelerates convergence and suppresses oscillations by accumulating historical gradients.

[0049] like Figure 1 As shown in the overall flowchart of the method, the method adopts a three-stage progressive architecture: the first stage is the calculation of the neuron task contribution (steps 1 to 4), the second stage is the layer-by-layer pruning and model fine-tuning (steps 5 to 6), and the third stage is the redundancy quantification and evaluation (steps 7 to 8), realizing a complete closed loop from the quantification of micro-neuron contribution to the evaluation of macro-model redundancy.

[0050] Step 1: Prepare the trained MLP regression model and dataset. Given a trained 4-layer MLP regression model, where L=4 (number of hidden layers), and the number of neurons in the first hidden layer... , second hidden layer 3rd hidden layer 4th hidden layer Input layer dimension The output layer has a dimension of 1 (regression task). Given a regression training dataset... The number of training samples M = 16512 (80% of the training set), and the regression target value ranges from [a,b] to [0.15,5.0] (unit: US$100,000).

[0051] Step 2: Extract the output features of neurons in each hidden layer. (This involves using the training dataset.) The input MLP model is used for forward propagation to extract the output features of each neuron in each hidden layer. For the j-th neuron in the l-th layer, its output feature vector on the training set is... In this embodiment, the first hidden layer extracts output features from 256 neurons, the second hidden layer extracts 128, the third hidden layer extracts 64, and the fourth hidden layer extracts 32, for a total of 480 neurons' output feature vectors.

[0052] Step 3: Calculate regression linear separability. The RLS calculation process includes four sub-steps: target value range division, binary sub-task splitting, binary classification linear separability calculation, and integral summation.

[0053] Target value range division: Divide the target value range [0.15, 5.0] into N=10 equal intervals, with a division step size of... The nth partition point Binary subtask decomposition: For each split point The neuron output features are categorized based on whether the target value is less than or equal to the target value. The subset is split into two parts, positive and negative. Binary classification linear separability calculation: By finding the optimal projection direction, the degree of separation between the two subsets along the projection direction is quantified. Integral summation: The 10 partition points are then used to calculate the linear separability. The average of the values ​​is used to obtain the RLS. Experiments show that the calculation results converge when N≥8 (the change is less than 0.001), so N=10 achieves a good balance between accuracy and efficiency.

[0054] In the calculation, for the j-th neuron in the l-th layer, In this embodiment, the RLS values ​​of neurons in the first hidden layer are distributed between [0.52, 0.91], with a mean of approximately 0.72; those in the second hidden layer are distributed between [0.51, 0.88], with a mean of approximately 0.68; those in the third hidden layer are distributed between [0.50, 0.85], with a mean of approximately 0.63; and those in the fourth hidden layer are distributed between [0.50, 0.82], with a mean of approximately 0.59. It can be observed that as the network depth increases, the mean RLS value of neurons gradually decreases, indicating that the average task contribution of neurons in deeper layers is lower than that in shallower layers. This is consistent with the rule that "deep layers have higher redundancy than shallow layers."

[0055] Step 4, Calculate the neuron's task contribution. Perform exponential mapping normalization on the RLS values ​​to obtain the neuron's task contribution. Where α = 2.5. The purpose of exponential mapping is to widen the gap between high-contribution neurons and low-contribution neurons. For example, a neuron with RLS = 0.9 has a contribution of approximately 0.85 after mapping, while a neuron with RLS = 0.55 has a contribution of only approximately 0.08, significantly amplifying the difference and facilitating subsequent redundant identification.

[0056] Step 5, dynamic threshold pruning. For example... Figure 5 (Hierarchical redundancy heatmap) and Figure 6 As shown in the heatmap of preserved neurons, the dynamic threshold pruning method determines the pruning threshold by calculating the statistics of the contribution of each layer.

[0057] The specific calculations are as follows: (5.1) Calculate the statistics of the contribution of each neuron in each layer. Taking the first hidden layer as an example, the minimum contribution value is... mean Standard deviation (5.2) Calculate the dynamic pruning threshold. When λ=0.7, (5.3) Prune layer by layer. Retain contribution level. Neurons, delete contribution The neurons. The first hidden layer retains approximately 180 neurons, with about 76 pruned. (5.4) Calculate layer redundancy. Layer 1 redundancy. .

[0058] In this embodiment, experiments were conducted with λ = 0.5, 0.7, and 0.9 respectively. When λ = 0.5, the redundancy of each layer was as follows: When λ=0.7, When λ=0.9, It can be clearly observed that as the number of layers increases, the level redundancy increases. The increasing λ value confirms the core principle that "deep layers have higher redundancy than shallow layers." Furthermore, a larger λ value corresponds to a higher pruning ratio in each layer, allowing for flexible selection based on the application scenario's requirements for compression ratio and performance.

[0059] Step 6, Model Reconstruction and Fine-tuning. The lightweight MLP regression model is reconstructed based on the pruning results, retaining the weights and bias parameters of the corresponding neurons in the original model. Taking λ=0.7 as an example, the reconstructed model structure is 8-180-79-32-12-1. Then, the Adam optimizer is used to fine-tune the pruned model for 30 epochs, employing a cosine annealing learning rate scheduling strategy with preheating. After fine-tuning, the mean squared error of the model on the test set recovered from 0.42 after pruning to 0.31, close to the original model's 0.30, indicating that fine-tuning effectively compensated for the performance loss caused by pruning.

[0060] Step 7, Local Hierarchical Redundancy Calculation. As described in Step 5, the redundancy of each level has been calculated. The calculations show that the redundancy distribution analysis indicates that the fourth hidden layer (the deepest layer) has the highest redundancy (λ=0.7). The first hidden layer (the shallowest layer) has the lowest redundancy. This distribution pattern provides an important reference for network structure design: deeper layers can have fewer neurons, while shallower layers need more neurons to extract basic features.

[0061] Step 8, global redundancy assessment. For example... Figure 2 and Figure 3 As shown, three global redundancy evaluation metrics are calculated.

[0062] The specific results are as follows: (8.1) Parameter retention ratio. Taking λ=0.7 as an example, the original model parameters... Parameters after pruning , That is, approximately 56.6% of the parameters were compressed. (8.2) Computational Retention Ratio. Since the number of parameters and computational load in an MLP model are in a fixed ratio, ≈ ≈0.434. (8.3) Performance Preservation Ratio. The original model test set MSE (a_0=0.30), after pruning and fine-tuning , ,belong The acceptable range of slight performance loss.

[0063] Comprehensive analysis: When λ=0.5, This indicates that pruning had a regularizing effect, resulting in a slight performance improvement; when λ=0.7, When λ=0.9, ( (This indicates excessive pruning and significant performance loss). Experimental results show that at λ=0.7, this method can maintain over 97% of the regression performance while compressing approximately 57% of the parameters and computational cost, achieving a good balance between efficient compression and performance preservation. In particular, on narrow and shallow network structures (such as 8-64-32-1), pruning at λ=0.7... The result is as high as 1.22, which means the test error is reduced by about 22%, demonstrating a significant regularization effect.

[0064] Example 2: Redundancy Evaluation Method for Feedforward Neural Networks Based on K-means Clustering Adaptive Pruning The difference between this embodiment and Embodiment 1 lies in the pruning method. This embodiment employs K-means clustering adaptive pruning, eliminating the need for manual setting of pruning parameters. This embodiment uses the same MLP regression model and California house price dataset as Embodiment 1.

[0065] Step 1, Neuron Contribution Calculation. Similar to steps 2 to 4 in Example 1, the task contribution of each neuron is obtained by extracting features through forward propagation, calculating RLS, and performing exponential mapping normalization.

[0066] Step 2, K-means clustering. For the contribution of all neurons in the l-th hidden layer... The K-means clustering algorithm is used to cluster them into K classes. Taking the first hidden layer as an example, Cluster number K-means clustering was performed on the 256 contribution values, resulting in 12 clusters. The cluster centers were calculated and sorted from largest to smallest, with clusters added sequentially starting with the highest contribution values ​​until the retention rate reached p=0.5. Approximately 128 neurons were ultimately retained, and approximately 128 were pruned.

[0067] Step 3, Redundant neuron identification. Compare the size of each cluster center. Clusters with smaller centers are classified as low-contribution classes, and neurons in these classes are identified as redundant neurons. Clusters with larger centers are classified as high-contribution classes, and neurons in these classes are retained.

[0068] Step 4, Pruning and Evaluation. Redundant neurons are removed, while high-contribution neurons are retained. Hierarchical redundancy is calculated. Global Indicators The method is the same as in Example 1. In this example, the result obtained by K-means clustering is: Layer 1 , second floor , third floor , 4th floor global , .

[0069] The advantage of this embodiment is that it completely eliminates the need for manual adjustment of the pruning ratio hyperparameters, automatically identifies redundant neurons, and achieves fully adaptive pruning. Compared with the dynamic thresholding method, the K-means clustering method has a higher compression ratio. vs λ=0.7 ), but performance remains slightly lower ( Experimental results show that this method can reduce test errors by more than 20% on narrow and shallow network structures, demonstrating a significant regularization effect. Both pruning methods have their advantages: the dynamic threshold method allows for flexible control of pruning intensity, suitable for scenarios requiring a fine trade-off between compression ratio and performance; the K-means clustering method requires no parameter tuning, making it suitable for automated deployment and rapid evaluation scenarios.

[0070] Example 3: Redundancy Evaluation Method for Classification Neural Networks The method of this invention is also applicable to classification neural networks. For classification tasks, the task contribution of each neuron is calculated directly using the original MD-LSM metric (multi-class version, i.e., multi-class linear separability measure MLSM), and then redundancy is evaluated using the same pruning criteria and evaluation system as in Example 1 or Example 2.

[0071] This embodiment uses the MNIST handwritten digit classification dataset as an example. This dataset contains 60,000 training samples and 10,000 test samples, with an image size of 28×28 pixels, flattened into a 784-dimensional feature vector, and a total of 10 categories. The model to be evaluated is a 4-layer MLP classification model with a structure of 784-512-256-128-64-10. The hidden layer activation function is ReLU, and the output layer uses softmax (an activation function that normalizes the output vector to a probability distribution, making the sum of all output values ​​equal to 1). The model was trained for 80 epochs, and the validation set accuracy was approximately 97.5%.

[0072] For classification tasks, neuron contributions are calculated using the multi-class linear separability measure (MLSM). Assuming an S=10 class classification problem, the binary linear separability between all class pairs (u,v) is first calculated. Then take the first V largest ones. The values ​​are weighted to obtain the MLSM, and then normalized by exponential mapping to obtain the contribution. The pruning and evaluation procedures are the same as in Example 1.

[0073] For classification tasks, the performance preservation degree is defined as follows: ,in This represents the classification accuracy of the original model. This represents the classification accuracy of the pruned model. Experimental results show that when λ=0.9, This means that while compressing 75% of the parameters, a classification accuracy of over 97% is maintained. K-means clustering can compress the parameters to 16.7% of the original value while maintaining an accuracy of approximately 74.5%.

[0074] The method of this invention can uniformly handle classification and regression tasks, and has good versatility and scalability. Through a unified contribution measurement framework (MLSM for classification and RLS for regression) and a unified evaluation index system, redundancy evaluation across task types is standardized.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for interpretably evaluating the structural redundancy of a feedforward neural network, characterized in that, Includes the following steps: Step 1: Construct a trained feedforward neural network model and dataset, define network structure parameters and regression training dataset, and obtain a standardized model and data representation; Step 2: Based on the feedforward neural network model and training dataset from Step 1, input the training dataset into the model for forward propagation, extract the output features of each neuron in each hidden layer, and obtain the feature representation of the neuron on the training data. Step 3: Based on the neuron output features extracted in Step 2, the linear separability measure based on the Minkowski difference is applied. Extending from classification scenarios to regression scenarios, the regression linear separability index (RLS index) for each neuron is calculated to obtain the regression linear separability value for each neuron. The regression linear separability index for neurons is: (3) in, Let be the output feature vector of the j-th neuron, y be the regression target value vector, and N be the number of split points. For the target value to be less than a subset of The target value is not less than a subset of As the dividing point; The RLS index ranges from [0.5, 1]. The closer RLS is to 1, the stronger the linear correlation between the neuron's output features and the regression target, and the greater its contribution to the regression task. The closer RLS is to 0.5, the smaller the neuron's contribution to the regression task, and the more likely it is to be a redundant neuron. Step 4: Based on the RLS index obtained in Step 3, the task contribution of each neuron is obtained through exponential mapping normalization, resulting in a neuron contribution distribution with a value range of [0,1]. Step 5: Based on the neuron task contribution obtained in Step 4, design two complementary pruning criteria to prune each hidden layer layer by layer, identify and delete redundant neurons, and obtain the pruned network structure, i.e., the layer-by-layer pruning result; the two pruning criteria are dynamic threshold pruning method and K-means clustering adaptive pruning method, which can be selected and used according to actual needs. Layer-by-layer pruning refers to pruning each hidden layer independently, determining which neurons to retain or delete based on the contribution distribution of neurons in the layer; dynamic threshold pruning refers to a method that dynamically calculates the pruning threshold based on the statistical characteristics of the contribution of neurons in each layer, flexibly controlling the pruning intensity through hyperparameters; K-means clustering adaptive pruning refers to using the K-means clustering algorithm to automatically cluster neurons into two classes: high contribution and low contribution. Step 6: Based on the layer-by-layer pruning results in Step 5, reconstruct the lightweight feedforward neural network model and perform fine-tuning training. Inherit the weights and bias parameters of the neurons in the original model to obtain a lightweight model with restored performance. Step 7: Based on the layer-by-layer pruning results in Step 5, calculate the hierarchical redundancy of each hidden layer to obtain the local hierarchical redundancy distribution. Step 8: Based on the pruned lightweight model obtained in Step 6, calculate three global redundancy evaluation indicators to comprehensively characterize the structural redundancy characteristics of the model from three complementary dimensions: parameter scale, computational overhead, and task performance.

2. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 1, characterized in that, The specific steps of the litigation process are as follows: Step 1.1, Define the feedforward neural network structure: Given a trained L-layer feedforward neural network model, where L is the total number of hidden layers in the network model, l is the layer index and l=1,2,…,L; the number of neurons in the l-th layer is denoted as . ,in The input layer dimension is a positive integer. The output layer dimension is 1 or S, where 1 represents a regression task that outputs a single continuous value, and S represents a classification task that outputs the probabilities of S categories. Step 1.2, Define the forward propagation process: The weight matrix of the l-th layer of the feedforward neural network model is denoted as... The bias term is denoted as The nonlinear activation function is denoted as f(·); for dimensional input samples The forward propagation process of the l-th layer is defined as follows: (1) in, This is the output of the l-th layer. For the first Layer output, input layer ; Step 1.3, Define the regression training dataset: Given a regression training dataset Where M is the number of training samples, Let be the input feature vector of the i-th sample. Let be the regression target value for the i-th sample; the range of the regression target value is defined as [a, b], where a is the minimum target value in the training set and b is the maximum target value in the training set; the training dataset refers to the set of labeled samples used to train the model parameters, and each sample contains input features and the corresponding target output.

3. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 2, characterized in that, The specific details of litigation step 2 are as follows: Step 2.1, Forward Propagation to Extract Activation Values: Based on the feedforward neural network model constructed in Step 1 and the regression training dataset Input all samples from the training set into the model in sequence, execute the forward propagation process defined in step 1.2, and record the output activation value of each neuron in each hidden layer on all training samples; Step 2.2, construct the neuron output feature vector: For the j-th neuron in the l-th hidden layer, where The output activation values ​​of the neuron on the M training samples are arranged in sample order to form the output feature vector of the neuron. ,in, Let i be the output activation value of the i-th sample on this neuron. Finally, the feature representation of each neuron on the training data is obtained.

4. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 3, characterized in that, In step 3, the RLS metric is used to extend the linear separability measure from classification tasks to regression tasks, measuring the degree of linear separability between neuron output features and the regression target; specifically: Step 3.1, Target Value Range Division: Based on the regression target value range [a, b] determined in Step 1.3, the interval [a, b] is divided into N equally spaced partition points, with a partition step size of h = (ba) / (N+1). The nth partition point is... , where n=1,2,…,N, and N is the number of partition points; Step 3.2, Binary Subtask Decomposition: For each split point The output feature set of the j-th neuron in the l-th layer is split into two subsets based on the target value: the subset with a target value less than 10 ... subset of and the target value is greater than or equal to subset of ; The continuous regression task is transformed into a binary classification subtask at each split point; Step 3.3, Binary classification linear separability calculation: For each split point Calculate two subsets and Linear separability measure between them based on Minkowski difference The Minkowski difference is defined as: for two sets , Its Minkowski difference is ,in Let A be the Minkowski difference elements, and B be the first set and B be the second set. The i-th element in the first set For the j-th element in the second set, It is an n-dimensional real number space; A measure of linear separability in binary classification Defined as: （2） Where I and J are the number of samples in sets A and B, respectively, ω is the projection direction vector, and P(·) is the probability function. It is the transpose of ω; The value range is [0.5, 1]; =1 indicates that the two sets are completely linearly separable in the optimal projection direction, and the closer the value is to 0.5, the weaker the linear separability. Step 3.4, Calculate the RLS integral: Divide the N partition points The average value is taken to obtain the regression linear separability (RLS) index of the neuron, as shown in formula (3).

5. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 4, characterized in that, Step 4 specifically involves: Based on the RLS index of each neuron calculated in step 3, it is normalized to the [0,1] interval using an exponential mapping function to obtain the task contribution of each neuron; the formula for calculating the task contribution is: (4) in, The contribution of the j-th neuron to the task, with a value range of [0,1]. Let be the regression linear separability of the j-th neuron; α is the scaling factor; The closer the value is to 1, the greater the neuron's contribution to the task. The closer a value is to 0, the more likely the neuron is to be redundant; For classification tasks, simply replace the RLS index in formula (3) with the Multi-Class Linear Separability Measure (MLSM). MLSM measures the linear separability of all class pairs. The value is obtained by taking the weighted average of the top V maximum values, where V is the number of category logs.

6. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 5, characterized in that, In step 4, α = 2.

5.

7. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 6, characterized in that, Step 5 specifically involves: Step 5.1, Dynamic Threshold Pruning: The pruning threshold is dynamically calculated based on the statistical characteristics of the contribution of neurons in each layer, achieving adaptive pruning for different layers; for the l-th hidden layer: First, calculate the minimum contribution of all neurons in this layer. and standard deviation ,in The mean contribution of neurons in layer l; Then, calculate the dynamic pruning threshold: (5) in, λ is the dynamic pruning threshold of the l-th layer, which is the critical value for determining whether a neuron is to be preserved; λ is a hyperparameter used to control the pruning ratio, and the larger λ is, the higher the pruning ratio. Finally, for layer l, the contribution is preserved. Neurons, delete contribution The neurons; by dynamically determining the threshold based on the minimum contribution of each layer and the standard deviation, adaptive pruning of different layers is achieved; Step 5.2, K-means clustering adaptive pruning method: The K-means clustering algorithm is used to automatically cluster neurons into two classes: high contribution and low contribution. No manual setting of pruning parameters is required, and fully adaptive redundancy recognition is achieved. Contribution of all neurons in the l-th hidden layer Treating it as a one-dimensional feature, the K-means clustering algorithm is used to cluster it into K clusters, where The goal of K-means clustering is to minimize the sum of squares within each cluster, i.e. ,in For the k-th cluster, The cluster center of the k-th class, Among them, the sum of squares within a cluster refers to the sum of the squared distances of all data points in each cluster to its cluster center, which is the optimization objective of the K-means algorithm; the cluster center refers to the mean of all data points in a cluster, representing the central position of the cluster. This indicates the floor operation, which takes the largest integer not greater than the given number. Sort the clusters by cluster center value from largest to smallest, and add them sequentially starting with the highest contribution clusters until the percentage of retained neurons reaches the target retention ratio p, and at least one neuron is retained. Clusters with small cluster centers are classified as low-contribution classes, and their neurons are considered redundant and deleted; clusters with large cluster centers are classified as high-contribution classes, and their neurons are retained; where p=0.

5.

8. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 7, characterized in that, Step 6 specifically involves: Step 6.1, Model Reconstruction: Based on the layer-by-layer pruning results in Step 5, remove the deleted neurons and their corresponding connection weights, and construct a new lightweight feedforward neural network structure; for each retained neuron, inherit its corresponding weight matrix column and bias term in the original model; for the connection between two adjacent layers, only retain the weight connection where both ends are retained neurons. Step 6.2, Fine-tuning training: Use the Adam optimizer to fine-tune the pruned model, employing a cosine annealing learning rate scheduling strategy with preheating to restore model performance.

9. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 8, characterized in that, Step 7 specifically involves: Based on the layer-by-layer pruning results in step 5, the layer redundancy of each hidden layer is calculated; the layer redundancy of layer l is defined as the proportion of the number of pruned redundant neurons in that layer to the total number of neurons in that layer: (6) in, represents the level redundancy of the l-th layer, with a value range of [0,1]. This represents the total number of original neurons in layer l. This represents the number of neurons retained in layer l after pruning.

10. The method for interpretably evaluating the structural redundancy of a feedforward neural network according to claim 9, characterized in that, Step 8 specifically involves: Step 8.1, Parameter Retention Ratio Defined as the ratio of the number of parameters in the pruned model to the number of parameters in the original model. (7) in, This represents the total number of parameters in the original model, i.e., the total number of all learnable parameters, including weights and biases. This represents the total number of parameters in the model after pruning. The value range is [0,1]. The smaller the value, the higher the degree of parameter compression and the greater the parameter redundancy of the network. Step 8.2, Retention of Calculation Amount Defined as the ratio of the floating-point computation cost of the pruned model to the floating-point computation cost of the original model. (8) in, This represents the floating-point computation cost of the original model. This represents the floating-point computation cost of the pruned model. The value range is [0,1]. The smaller the value, the higher the degree of computational compression and the greater the computational redundancy of the network. Step 8.3, Performance Retention Ratio Used to examine whether pruning operations impair the model's task performance; For classification tasks, Defined as the ratio of the classification accuracy of the pruned model to the classification accuracy of the original model: (9) in, This represents the classification accuracy of the original model on the test set. This represents the classification accuracy of the pruned model on the test set. For regression tasks, due to mean squared error The smaller the value, the better the performance. Defined as the inverse ratio of the mean squared error of the original model to the mean squared error of the pruned model: (10) in, This represents the mean squared error of the original model on the test set. This represents the mean squared error of the pruned model on the test set. The indicator has a clear physical meaning: when This indicates that the pruned model performs better than the original model, demonstrating that removing redundant structures has a regularization effect. A value of 1 indicates that the performance is completely unaffected after pruning, achieving ideal redundancy removal; This indicates that pruning causes a minor performance loss within an acceptable range; This indicates that excessive pruning has impaired the model's ability to extract core features.

Images