Training-free neural architecture evaluation method based on singular value distribution entropy
By evaluating neural architectures using the Singular Score, the problems of high computational cost and low ranking consistency in neural architecture search are solved, and efficient network architecture performance prediction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA JILIANG UNIV
- Filing Date
- 2026-05-07
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, neural architecture search requires multiple training sessions during the architecture evaluation phase, resulting in high computational costs, and the predictive performance ranking of zero-cost proxy metrics has low consistency with the actual performance ranking.
We use the singular value distribution entropy (SingularScore) as a zero-cost surrogate metric to measure the network's expressive power and select the optimal neural architecture by analyzing the uniformity of the singular value distribution of the activation tensor under random input.
It significantly reduces the computational cost of neural architecture search and improves the consistency between the predicted performance ranking and the actual performance ranking of zero-cost proxy metrics, effectively measuring the expressive power of network architecture.
Smart Images
Figure CN122133703A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural architecture search in deep learning, and more particularly to a training-free neural architecture evaluation method based on singular value distribution entropy. Background Technology
[0002] In recent years, deep neural networks have achieved remarkable results in many fields such as image classification and natural language processing. However, because their architecture design heavily relies on researchers' prior knowledge and experience, it is difficult for beginners to design neural architectures that meet the needs of different tasks. Therefore, architecture search methods that can automate neural network design have attracted widespread attention. However, traditional neural architecture search requires multiple training iterations for each candidate architecture during the architecture evaluation phase, resulting in high computational costs. Therefore, for training-free neural architecture search, using zero-cost surrogate metrics to quickly evaluate network performance is of great significance for reducing computational costs.
[0003] Zero-cost surrogate metrics are closely related to deep network architecture parameters. During architecture initialization, information such as activation functions, gradients, and parameter counts are obtained through forward propagation, backpropagation, and architecture topology analysis. Based on this parameter information, a zero-cost surrogate metric is constructed. Furthermore, the network's performance potential is predicted using a zero-cost surrogate metric evaluation method. When evaluating network performance, Kendall's stau (KT) and Spearman's rank correlation (SPR) are used to measure the consistency between the performance predicted by the zero-cost surrogate metric and the actual performance of the architecture.
[0004] Zero-cost surrogate metrics rank network prediction performance to select the optimal neural architecture, avoiding lengthy training processes. However, current research on zero-cost surrogate metrics faces the challenge that the ranking of network prediction performance by most metrics shows low consistency with the actual performance ranking. To address this issue, this invention proposes a general zero-cost surrogate metric that is highly correlated with actual performance. It primarily predicts the expressive power of the network architecture and selects the optimal architecture by analyzing the activation tensors output by activation layers. Summary of the Invention
[0005] To address the issue of low consistency between network prediction performance ranking and actual performance ranking in existing technologies, the present invention aims to provide a training-free neural architecture evaluation method based on singular value distribution entropy, thereby solving the technical problems mentioned in the background art.
[0006] The core of this invention lies in proposing a novel zero-cost surrogate metric, the Singular Value Distribution Entropy (SingularScore). This metric analyzes the uniformity of the singular value distribution of the activation tensor under random inputs, serving as a performance measure of the network's expressive power in training-free neural architecture search. A network with strong expressive power should acquire important information distributed across multiple feature directions, rather than a few.
[0007] According to the mathematical definition and related theorems of singular values, the magnitude of a singular value is proportional to the variance of the data captured by its corresponding singular vector. Therefore, singular values reflect the importance of each singular vector in representing the data. A high singular value means that the network can capture most of the variance information in the data along that singular vector, which also means that this feature direction is very important. Specifically, as shown below... Figure 2 As shown, the green dots represent the distribution of each value of the activation tensor in the feature space. After singular value decomposition, Figure 2 High singularity on the right and These represent significant variances in the data distribution along their respective directions, containing a large amount of information. Since the singular value ranges differ across activation layers, this invention normalizes the singular values of each layer to a probability value, objectively measuring the number of high singular values through information entropy, thus representing the distribution of singular values. The entropy values of all layers are then summed to serve as a zero-cost proxy metric score for this architecture.
[0008] In summary, by performing singular value decomposition on the activation tensor of a network and quantifying the uniformity of the singular value distribution through information entropy, we can measure the efficiency of the network's utilization of important information in this feature space and predict the network's expressive power.
[0009] To achieve the above technical effects, the present invention provides a training-free neural architecture evaluation method based on singular value distribution entropy, which adopts the following technical solution and includes the following steps: Step 1: Specify the dataset and obtain the image information of the dataset; Step 2: For a given search space, randomly sample neural architectures until a specified number of samples is reached; Step 3: Input random noise that conforms to the image information distribution into the neural architecture to be evaluated, perform a single forward propagation, and obtain the activation tensors output by all activation layers to be evaluated in the neural architecture. Step 4: Perform singular value decomposition on the activation tensor of each activation layer to obtain the corresponding set of singular values; Step 5: For each set of singular values, calculate the information entropy value of its probability distribution; Step 6: Accumulate the information entropy values of all activation layers in a neural architecture to generate a comprehensive score, which serves as the singular value distribution entropy index of the neural architecture to measure its expressive power. Step 7: Repeat steps 3 to 6 for each sampled neural architecture. In the selected high-performance architecture candidate set, select the neural architecture with the highest singular value distribution entropy index score as the optimal network structure.
[0010] The steps for the above-described technical solution of this invention are specified as follows: Step 1 involves obtaining image information from the dataset, including the pixel dimensions (length, width, and number of channels) of the images, as well as the standard batch size for that dataset. Common datasets include CIFAR-10, CIFAR-100, and ImageNet16-120.
[0011] In step 2, the search space for the neural architecture and the number of randomly sampled neural architectures are determined. The setting of the number of random samples will affect the final network performance.
[0012] The specific operations in step 3 include: Based on the image information of the dataset, a random noise tensor conforming to a Gaussian normal distribution is generated, with dimensions of batch size, number of channels, length, and width. The random noise tensor is input into the neural architecture to be evaluated, and a single forward propagation is performed. The activation tensor output after random noise passes through each activation layer in the neural architecture is recorded and denoted as . ,in Indicates the activation layer index. To characterize the number of feature vectors representing the total number of input samples in one forward propagation, This is a feature dimension that characterizes the number of neurons in the current layer.
[0013] The specific steps in step 4 are as follows: The first of the architecture Layer activation tensor Perform singular value decomposition: in, It is The matrix, The column vectors are left singular vectors; It is The matrix, The column vectors are right singular vectors; It is A diagonal matrix is represented as: in, It is a diagonal matrix. , Represents a diagonal matrix rank, The size is also equal to the number of non-zero singular values. Sort by size, conforming to So, the first... The set of singular values of a layer is called the singular value spectrum, which is represented as: in, Indicates the first Layer singular value spectrum quantifies the information contained in each direction by the magnitude of singular values, and measures the importance of information captured by neurons in the current layer in each direction.
[0014] The specific steps in step 5 are as follows: Since the singular value ranges differ across layers, they are normalized. First, the first layer... Singular value set of the layer Calculate the normalized probability of each singular value using the following formula: in, Indicates the first The first layer The normalized probability of each singular value is given, with a numerical range of (0, 1). Then, the uniformity of the singular value distribution in each layer is analyzed, and the singular value distribution entropy of the activation tensor in that layer is calculated based on this probability distribution. The calculation formula is as follows: in, It is a preset constant. Represents the first The sum of the singular value entropy of a layer. If a layer has a large number of high singular values, then the normalized probability values will be concentrated in a specific distribution. Therefore, the layer... It will also be larger.
[0015] The specific steps in step 6 are as follows: The singular value distribution entropy of all activation layers in the neural architecture. The scores are accumulated to obtain the surrogate index score representing the expressive ability of this neural architecture, denoted as: in, This indicates the number of active layers in the architecture. It is the value of the singular value distribution entropy index of the architecture, used to quickly evaluate the potential performance of the neural architecture in terms of expressive power.
[0016] In step 7, steps 3 to 6 are repeated for all sampled architectures to calculate the singular value distribution entropy of each architecture. The sampled network architectures are then sorted according to the magnitude of this index value, and the neural architecture with the largest index value is selected as the optimal neural architecture.
[0017] The beneficial effects of this invention are: To address the high computational cost of traditional neural architecture search methods, this invention proposes a novel and general training-free neural architecture evaluation method based on singular value distribution entropy. This method primarily analyzes the output of each activation layer of the neural architecture, using the singular value distribution of the activation tensor to measure the diversity of output patterns and determine the existence of multiple important information directions in the feature space. This invention effectively solves the problem of low consistency between the predictive performance ranking and the actual network performance ranking of zero-cost surrogate metrics, enabling the measurement of the expressive power of network architectures and significantly reducing the computational cost of neural architecture search. Attached Figure Description
[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is an illustration of the singular value distribution entropy of the present invention; Figure 3 This is a graph showing the relationship between the singular value distribution entropy and the actual performance ranking on the CIFAR-10 dataset. Figure 4 This is a graph showing the relationship between the singular value distribution entropy and the actual performance ranking on the CIFAR-100 dataset. Figure 5 This is a graph showing the relationship between the singular value distribution entropy and the actual performance ranking on the ImageNet16-120 dataset. Detailed Implementation
[0019] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.
[0020] like Figure 1 As shown, the specific steps of an embodiment of the present invention are as follows: Step 1: Specify the dataset and obtain the image information of the dataset, including determining the dataset name, obtaining the pixel length, width, and number of channels of the image, and the batch size of the dataset.
[0021] In this step, the dataset can be a benchmark dataset such as CIFAR-10, CIFAR-100, or ImageNet16-120. The pixel dimensions, width, and number of channels of the image can be obtained directly through the dataset interface provided by torchvision. However, the batch size for data processing needs to be set when configuring the DataLoader and should be adjusted appropriately based on computing resources and model requirements.
[0022] Step 2: Determine the search space for neural architectures and the number of neural architectures to be sampled; based on the given search space, randomly sample neural architectures until the specified number of samples is reached; In this step, the search space can be selected from DARTS, NAS-Bench-201, etc. Furthermore, the number of network architectures to be sampled is preset, and then random sampling is performed within the search space until the preset number of network architectures is reached.
[0023] Step 3: Input random noise that conforms to the image distribution information of the dataset into the neural architecture to be evaluated, perform a single forward propagation, and obtain the activation tensors output by all activation layers to be evaluated in the neural architecture; Specifically, based on the image information in the dataset, a random noise tensor conforming to a Gaussian normal distribution is generated, with dimensions including batch size, number of channels, length, and width. In other words, a Gaussian noise map is randomly generated using a random number generation method (e.g., the `torch.randn` function), and the batch size, number of channels, image length, and image width of this noise map are consistent with the image information in the dataset. This noise map is then used to replace the images in the dataset as input to the neural architecture to be evaluated.
[0024] The random noise tensor is input into the neural architecture to be evaluated, and a single forward propagation is performed. The activation tensor output after random noise passes through each activation layer in the neural architecture is recorded and denoted as . ,in Indicates the activation layer index. This represents the number of feature vectors in one forward propagation. Indicates the feature dimension of the current layer.
[0025] Step 4: For each of the activation tensors, perform singular value decomposition to obtain the corresponding set of singular values; Specifically, regarding the architecture's first... Layer activation tensor To perform singular value decomposition, the expression is: in, It is The matrix, The column vectors are left singular vectors; It is The matrix, The column vectors are right singular vectors; It is A diagonal matrix is represented as: in, It is a diagonal matrix. , Represents a diagonal matrix rank, It also represents the number of non-zero singular values, specifying the specific singular values. Sort by size, conforming to So, the first The set of singular values of a layer is called the singular value spectrum, which is represented as: in, Representing the The singular value set of a layer can measure the importance of information in each direction in the neurons of the current layer and intuitively reflect the distribution of singular value values.
[0026] Step 5: For each set of singular values, calculate the information entropy value of its probability distribution; Specifically, since the singular value ranges of each layer are different, they are normalized. First, the singular value range of the first layer is... Singular value set of the layer Calculate the normalized probability of each singular value using the following formula: in, Indicates the first Layer The normalized probability of each singular value is given, with a numerical range of (0,1). The uniformity of the singular value distribution in each layer is analyzed, and the singular value distribution entropy of the activation tensor in that layer is calculated based on this probability distribution. The calculation formula is as follows: in, It is a preset constant with a size of To avoid the situation where log0 occurs. Represents the first The sum of the singular value information entropy of a layer; if a layer has a large number of high singular values, then the corresponding... It will also be larger.
[0027] Step 6: Calculate the singular value distribution entropy of each activation layer in the neural architecture. The scores are accumulated to obtain the surrogate index score representing the expressive ability of this neural architecture, denoted as: in, This indicates the total number of active layers in the architecture. It is the value of the singular value distribution entropy index of the neural architecture, which is used to quickly evaluate the performance of the neural architecture in terms of expressive power.
[0028] Step 7: Repeat steps 3 to 6 for all sampled architectures to calculate the singular value distribution entropy of each architecture. Sort the sampled network architectures according to the magnitude of the index value. The neural architecture with the largest index value is selected as the optimal neural architecture.
[0029] In practical image recognition or classification tasks, traditional methods require a significant amount of time to train neural networks, and then evaluate the performance of the neural network architecture based on the network's performance on the dataset. This invention proposes a zero-cost surrogate index that eliminates the need for training with real images. Simply inputting the randomly generated Gaussian noise map into the network being evaluated allows for rapid assessment of the neural network architecture's performance. Its advantages are: this invention quickly determines performance by evaluating the rationality of the neural network architecture itself. Using real images introduces the network's ability to fit image features, interfering with the judgment of the neural network architecture itself; simultaneously, the generation and calculation of the random noise map is simple and fast, resulting in high overall evaluation efficiency and lower time costs; moreover, the random noise map activates neurons in every layer of the network, unlike real images which only activate some neurons, thus providing the most complete information from the activation tensor, and the singular value distribution best represents the architecture's capabilities.
[0030] According to the mathematical definition and related theorems of singular values, the magnitude of a singular value is proportional to the variance of the data captured by its corresponding singular vector. Therefore, singular values reflect the importance of each singular vector in representing the data. A high singular value means that the network can capture most of the variance information in the data along that singular vector, which also means that this feature direction is very important. Specifically, as shown below... Figure 2 As shown, the green dots represent the distribution of each value of the activation tensor in the feature space. After singular value decomposition, Figure 2 High singularity on the right and These represent significant variances in the data distribution along their respective directions, containing a large amount of information. Since the singular value ranges differ across activation layers, this invention normalizes the singular values of each layer to a probability value, objectively measuring the number of high singular values through information entropy, thus representing the distribution of singular values. The entropy values of all layers are then summed to serve as a zero-cost proxy metric score for this architecture.
[0031] The following are some specific examples to illustrate this: 1 Experimental conditions To further verify the technical effectiveness of this method, this implementation analyzes ranking consistency and final architecture performance using the CIFAR-10 and CIFAR-100 datasets in the DARTS search space, the CIFAR-10, CIFAR-100, and ImageNet16-120 datasets in the NAS-Bench-201 search space, and the MedMNISTv2 medical dataset. The deep learning framework used is PyTorch, and the performance of this method is verified on a single RTX 4090 GPU.
[0032] 2. Comparison of Experimental Content and Results The experiments in this invention mainly adopt a training-free neural architecture evaluation method based on singular value distribution entropy proposed in this invention. Based on different search spaces, KT and SPR metrics are used to analyze ranking consistency, and finally the architecture performance is obtained.
[0033] Experiment 1: The CIFAR-10 dataset consists of 10 classes of 32x32 pixel color images with 3 channels, a batch size of 64, and random noise with a shape of (64, 3, 32, 32). CIFAR-100 contains 100 classes, with images of 32×32 pixels, 3 channels, a batch size of 64, and random noise with a shape of (64, 3, 32, 32).
[0034] The method of this invention was applied to the CIFAR-10 and CIFAR-100 datasets in the DARTS search space. All zero-cost surrogate metrics employed the Zero-Cost-PT (ZC-PT) method to find the optimal network architecture, as shown in Table 1. Here, TestError represents the error rate, Params represents the number of parameters in the found network architecture, and SearchCost represents the search time cost. Table 1 shows that, compared to other zero-cost surrogate metrics, the TestError of this invention ranks second on the CIFAR-10 dataset, indicating that the found network architecture has high performance; on the CIFAR-100 dataset, the TestError of this invention is optimal, indicating that the found network architecture has the best performance. Furthermore, this invention maintains a low search time cost.
[0035] Table 1 compares the TestError rate, the number of parameters in the Params network architecture, and the SearchCost search time based on the CIFAR-10 and CIFAR-100 datasets. Experiment 2: ImageNet16-120 contains 120 categories. The image size is uniformly adjusted to 16×16 pixels, with 3 channels and a batch size of 64.
[0036] The method of this invention was applied to three benchmark datasets—CIFAR-10, CIFAR-100, and ImageNet16-120—on the NAS-Bench-201 platform. 1000 architectures were randomly sampled, and they were ranked according to the singular value distribution entropy index. The relationship between these rankings and the actual performance of each architecture is as follows: Figure 3 , Figure 4 and Figure 5 As shown in the figure. The results indicate that the higher the singular value distribution entropy of the neural architecture, the higher the corresponding test set accuracy.
[0037] In addition, 3000 architectures were randomly sampled, and three different random seeds were selected to verify the ranking consistency of all zero-cost surrogate index methods, as shown in Table 2. The type refers to the classification of the calculation basis / information source of the zero-cost index: A is based on computational cost (FLOPs); B is based on gradient / parameter information (gradient, Jacobian, NTK, etc.); F is based on activation / feature information (this invention belongs to this category); B+F uses both gradient and activation information.
[0038] As shown in Table 2, compared with existing zero-cost proxy methods, this invention achieves optimal values for both KT and SPR correlation coefficients, demonstrating extremely strong correlation. Specifically, on the CIFAR-10, CIFAR-100, and ImageNet16-120 datasets, the KT (rank correlation coefficient, which measures the consistency between the architecture ranking predicted by the proxy metric and the actual performance ranking; the closer to 1, the more accurate the prediction) correlation coefficient is, the higher the improvement is, from 0.04 to 0.37, and the SPR (Spearman rank correlation coefficient, a ranking consistency index that is more sensitive than KT; the closer to 1, the more accurate the prediction) correlation coefficient is, the higher the improvement is, from 0.04 to 0.46.
[0039] Table 2 Comparison of KT and SPR metrics based on CIFAR-10, CIFAR-100, and ImageNet16-120 datasets. Furthermore, 3000 architectures were randomly sampled, and three different random seeds were selected to verify the performance of all zero-cost proxy metrics methods, as shown in Table 3. In Table 3, ValidACC represents the validation set accuracy, TestACC represents the test set accuracy, and Benchmark represents the theoretically best performance within the search space.
[0040] Table 3 Comparison of ACC accuracy and time based on CIFAR-10, CIFAR-100, and ImageNet16-120 datasets. As shown in Table 3, on the CIFAR-10 and CIFAR-100 datasets, the performance of this invention closely follows the Synflow or Zico methods, with a difference of only 0.13% on CIFAR-10 and 0.77% on CIFAR-100. On the ImageNet16-120 dataset, the performance of this invention is slightly worse than the TENAS method, which has the best accuracy on both the validation and test sets, but the computation time of this invention is significantly reduced.
[0041] Experiment 3: The MedMNISTv2 dataset serves as a benchmark for medical image classification, comprising 12 2D datasets and 6 3D datasets. This dataset covers various imaging modalities, including X-ray, OCT, ultrasound, CT, and microscopy, and various tasks such as multi-class classification, binary classification, multi-label classification, and ordered regression. This experiment selected representative 2D subsets, including: PathMNIST (colon pathology images, 9 classes), DermaMNIST (dermoscopy images, 7 classes), OCTMNIST (retinal OCT images, 4 classes), PneumoniaMNIST (pediatric chest X-rays, pneumonia / normal binary classification), RetinaMNIST (fundus images, diabetic retinopathy grade 5 ordered regression), BreastMNIST (breast ultrasound, benign / malignant binary classification), BloodMNIST (blood cell microscopy images, 8 classes), and organ classification datasets from abdominal CT scans: OrganAMNIST, OrganCMNIST, and OrganSMNIST (corresponding to 11 organ classes for axial, coronal, and sagittal sections, respectively). All subsets of images are uniformly sized to 28*28 pixels, with 3 channels, and the batch size is set to 24.
[0042] The method of this invention was applied to the aforementioned 10 datasets, and the optimal network architecture was searched based on the singular value distribution entropy index. The accuracy results are shown in Table 4 below. In this table, ResNet-18(28), ResNet-50(28), AutoKeras, AutoML, DARTS, SNAS, and HOPNAS represent search methods; AZE (an automated neural architecture search method based on zero-cost proxies) and SynFlow (a classic zero-cost proxies index) represent zero-cost proxies using the MedNAS search method; and AVG represents the average ACC index of the 10 datasets.
[0043] Table 4 Comparison of ACC based on the MedMNIST dataset As shown in Table 4, compared with other search methods and zero-cost proxy metrics, this invention achieved the highest network architecture accuracy on six datasets: Path, Retina, Breast, OrganA, OrganC, and OrganS. Specifically, compared with other zero-cost proxy metrics, this invention achieved an average accuracy of 0.842 on 10 datasets, representing a 1.7% improvement over SynFlow.
[0044] In performance evaluation of medical datasets, in addition to accuracy (ACC), the area under the curve (AUC) is also an important evaluation metric. The AUC results of this invention on the MedMNIST dataset are shown in Table 5.
[0045] Table 5. Comparison of AUC based on the MedMNIST dataset. As shown in Table 5, compared with other methods, this invention achieved the best performance on five datasets: Path, DERMA, Blood, OrganA, and OrganS. Compared with other zero-cost proxy metrics, this invention achieved an average AUC of 0.945 on the above 10 datasets, which is an improvement of 0.005 compared to AZE and an improvement of 0.016 compared to SynFlow.
[0046] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.
Claims
1. A training-free neural architecture evaluation method based on singular value distribution entropy, characterized in that, Includes the following steps: Step 1: Specify the dataset and obtain the image information of the dataset; Step 2: For a given search space, randomly sample neural architectures until a specified number of samples is reached; Step 3: Input random noise that conforms to the image information distribution into the neural architecture to be evaluated, perform a single forward propagation, and obtain the activation tensors output by all activation layers in the neural architecture. Step 4: Perform singular value decomposition on the activation tensor in each activation layer to obtain the corresponding set of singular values; Step 5: For each set of singular values, calculate the information entropy value of its probability distribution; Step 6: Accumulate the information entropy values of all activation layers in a neural architecture to generate a comprehensive score, which serves as the singular value distribution entropy index of the neural architecture to measure its expressive power. Step 7: Repeat steps 3 to 6 for each sampled neural architecture, and select the neural architecture with the highest singular value distribution entropy index score as the optimal network structure.
2. The method according to claim 1, characterized in that, The specific steps for obtaining the image information of the dataset in step 1 are as follows: Obtain image information from the dataset, including the pixel length, width, and number of channels of the image, as well as the batch size of the dataset.
3. The method according to claim 1, characterized in that, Step 3 specifically includes: A random noise tensor conforming to a Gaussian normal distribution is generated based on the image information of the dataset, with dimensions being the batch size, number of channels, length, and width. The random noise tensor is input into the neural architecture to be evaluated, and a single forward propagation is performed. The activation tensor output after random noise passes through each activation layer in the neural architecture is recorded and denoted as . ,in Indicates the activation layer index. To characterize the number of feature vectors representing the total number of input samples in one forward propagation, This is a feature dimension that characterizes the number of neurons in the current layer.
4. The method according to claim 1, characterized in that, The specific steps in step 4 are as follows: The first of the architecture Layer activation tensor Perform singular value decomposition: in It is The matrix, The column vectors are left singular vectors; It is The matrix, The column vectors are right singular vectors; It is A diagonal matrix is represented as: in, It is a diagonal matrix. , Represents a diagonal matrix rank, It also represents the number of non-zero singular values, and the singular values are... Sort by size, conforming to ;No. The set of singular values of a layer can be represented as: in, Represents the first The set of singular values of a layer.
5. The method according to claim 1, characterized in that, The specific steps in step 5 are as follows: The first Singular value set of the layer Calculate the normalized probability of each singular value, using the following formula: in, Representing the Layer The normalized probability of each singular value; then, the singular value distribution entropy of the activation tensor of this layer is calculated based on this probability distribution, with the specific formula as follows: in, As a preset constant, Representing the The sum of the singular value information entropy of the layers.