A method for generating an adaptive mixed-precision quantization network based on self-learning

Abstract

The application relates to the technical field of model quantization, and discloses a generation method of a self-adaptive mixed-precision quantization network based on self-learning, which comprises the following steps: obtaining a teacher bit width set and a student meta-network to be trained, the teacher bit width set comprising multiple candidate teachers with high bit widths, and the student meta-network sharing full-precision weights and supporting multiple bit width configurations; for any bit width configuration, a corresponding target teacher is determined according to the interlayer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher; all bit width configurations of the student meta-network are jointly trained to obtain a self-adaptive bit width student meta-network; based on the interlayer importance evaluation result, a bit width search strategy is used to determine the target bit width of each layer network; and based on the target bit width of each layer network and the self-adaptive bit width student meta-network, a target mixed-precision quantization network is generated, thereby solving the technical problem of low diagnosis accuracy of related models in power equipment state monitoring.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of model quantization, in particular to a generation method of an adaptive mixed-precision quantization network based on self-learning. BACKGROUND

[0002] With the continuous expansion of the scale and the continuous increase of the complexity of the power system, the monitoring, early warning and processing of various power failures and safety hazards become increasingly difficult. Deep learning technology has been widely applied in the field of power system safety monitoring and fault diagnosis due to its powerful data processing and pattern recognition capabilities. For example, by constructing a deep neural network model to analyze the operation data of power equipment, potential equipment failures can be predicted in advance to ensure the reliable operation of the power system. However, the amount of data generated by the power system is huge, and there are a large number of resource-constrained devices in the operating environment, such as edge computing nodes and smart meters distributed throughout the power network, which puts extremely high requirements on the computational complexity and storage requirements of deep learning models. Network model quantization technology aims to reduce the computational complexity and storage requirements of deep neural networks to adapt to resource-constrained devices and scenarios. Traditional quantization methods are trained for a single fixed bit width, and when quantized to other bit widths, the performance will decrease significantly, which limits the flexibility of the model under different precision requirements. In order to support multiple bit widths, multiple training is usually required to train multiple copies of the model into models of different bit widths, which not only consumes time and effort, but also increases storage costs. In recent years, adaptive quantization methods have gradually attracted attention, which allows flexible selection of the bit width of the deep neural network during inference to meet the dynamically changing requirements. However, in existing adaptive bit width quantization methods, high bit width often dominates in training, resulting in poor performance of low bit width quantization, which cannot meet the high precision requirements of power equipment state monitoring. SUMMARY

[0003] The application provides a generation method of an adaptive mixed-precision quantization network based on self-learning, which solves the technical problem of low diagnostic accuracy of related lightweight models in power equipment state monitoring, and achieves the technical effect of improving the precision of power equipment state monitoring.

[0004] In order to achieve the above purpose, the main technical scheme adopted by the application includes:

[0005] In a first aspect, the embodiments of the present application provide a generation method of a self-adaptive mixed-precision quantization network based on self-learning, applied to power equipment state monitoring, the method comprising: obtaining a teacher bit width set and a student meta-network to be trained, the teacher bit width set including multiple high-bit-width candidate teachers, and the student meta-network sharing full-precision weights and supporting multiple bit width configurations; for any bit width configuration, determining a corresponding target teacher according to the inter-layer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher; jointly training all bit width configurations of the student meta-network to obtain a self-adaptive bit width student meta-network, including: for any bit width configuration, determining the training loss between the student meta-network and the target teacher, weighting and summing the training losses under all bit width configurations to obtain a joint loss function, and updating the shared full-precision weights based on the joint loss function; generating an inter-layer importance evaluation result of each layer network based on a normalization parameter ratio, a normalization entropy, a normalization variance, and a normalization activation sparsity; determining the target bit width of each layer network based on real-time equipment state data and the inter-layer importance evaluation result; and generating a target mixed-precision quantization network responding to real-time equipment state based on the target bit width of each layer network and the self-adaptive bit width student meta-network.

[0006] The generation method of a self-adaptive mixed-precision quantization network based on self-learning proposed by the embodiments of the present application is applied to power equipment state monitoring, a method for selecting a target teacher based on inter-layer Manhattan distance and the entropy value of the prediction probability distribution of each candidate teacher is proposed by combining a knowledge distillation training progress adaptive joint loss, the most suitable teacher network is adaptively selected for each bit width, the performance of low-bit-width quantization is significantly improved, and the training process is optimized. The importance of each layer is calculated by using a normalization parameter ratio, a normalization entropy, a normalization variance, and a normalization activation sparsity, an inter-layer importance evaluation scheme is developed, the target bit width of each layer network is determined based on the inter-layer importance evaluation result and real-time equipment state data, a target mixed-precision quantization network responding to real-time equipment state is generated based on the target bit width of each layer network and the self-adaptive bit width student meta-network, and the technical problem of low diagnostic accuracy of related lightweight models in power equipment state monitoring is solved, achieving the technical effect of improving the precision of power equipment state monitoring.

[0007] Optionally, the method further comprises: obtaining historical equipment state data and extracting multi-dimensional features in the historical equipment state data, including device operating state indicators, abnormal feature patterns, and time series change trends; generating a device state-bit width mapping strategy table based on the multi-dimensional features, the device state-bit width mapping strategy table being used to represent the configuration strategy of each layer bit width corresponding to different power equipment states; and matching the real-time equipment state data with the device state feature vectors in the device state-bit width mapping strategy table to determine the target bit width of each layer network.

[0008] Optionally, the method further comprises: analyzing the real-time index and fluctuation characteristics of the historical device state data, evaluating the data real-time strength and abnormal fluctuation frequency based on the real-time index and the fluctuation characteristics; if the data real-time is strong and the abnormal fluctuation frequency is high, the probability is shortened, the round interval is reduced, and the conversion rate is accelerated, otherwise, the round interval is extended, and the conversion rate is slowed down, so that the probability and the power real-time requirement are dynamically adapted.

[0009] Optionally, the training loss is composed of a labeled data supervised loss and a knowledge distillation loss by weighted combination, wherein the labeled data supervised loss is a cross-entropy loss, and the distillation loss is a KL divergence loss, and the network parameters are updated synchronously by gradient accumulation.

[0010] Optionally, the method further comprises: dynamically adjusting the weight coefficients of the cross-entropy loss and the KL divergence loss by using a linear annealing strategy: focusing on the cross-entropy loss to establish a class discrimination boundary in the early stage of training, and gradually increasing the weight coefficient of the KL divergence loss as the training progresses.

[0011] Optionally, the method further comprises: dynamically balancing the weights of the entropy value and the inter-layer Manhattan distance by using a hyperparameter, and linearly adjusting the hyperparameter with the training progress to determine the target teacher corresponding to the current bit width configuration.

[0012] Optionally, the method further comprises: using a quantization function of a downward rounding operator in the training process to learn a quantization step size for the weights and activations of each layer network.

[0013] Optionally, the method further comprises: independently learning a dedicated BN layer for the bit width transition combination of each pair of adjacent layers; quantifying the distribution shift caused by the continuous inter-layer bit width change by calculating the difference between the mean and variance of the activations before and after quantization; and compensating for the distribution shift by dynamically adjusting the scaling parameter and the offset parameter of the BN layer.

[0014] Optionally, based on the inter-layer importance evaluation result, a bit width search strategy is used to determine the target bit width of each layer network, including: arranging the layers in descending order of layer importance according to the inter-layer importance evaluation result, and performing the following operations for each layer: initializing the bit width to the highest quantization bit width supported by the current layer, and iteratively adjusting the bit width downward to a lower bit width by a fixed step under the premise that the accuracy loss does not exceed a set threshold, to determine the target bit width of the current layer.

[0015] Optionally, the method further comprises: performing a diagnosis accuracy test on the current bit width configuration network on the power equipment fault data set to obtain an accuracy loss of the current bit width; and generating the set threshold of the accuracy loss according to the safety level of the power equipment and the fault diagnosis accuracy requirement, and if the adjusted accuracy loss exceeds the set threshold in the process of adjusting the bit width configuration from high bit to low bit, terminating the bit width down-regulation and retaining the bit width configuration of the previous bit.

[0016] In a second aspect, an embodiment of the present application provides a generation system of an adaptive mixed-precision quantization network based on self-learning, applied to power equipment state monitoring, the system comprising: an acquisition module configured to acquire a teacher bit width set and a student meta-network to be trained, the teacher bit width set comprising a plurality of candidate teachers of high bit width, and the student meta-network sharing full-precision weights and supporting multiple bit width configurations; a training module configured to determine a corresponding target teacher according to the inter-layer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher for any bit width configuration; and configured to perform joint training on all bit width configurations of the student meta-network to obtain an adaptive bit width student meta-network, including: determining the training loss between the student meta-network and the target teacher for any bit width configuration, weighting and summing the training losses under all bit width configurations to obtain a joint loss function, and updating the shared full-precision weights based on the joint loss function; a generation module configured to generate an inter-layer importance evaluation result of each layer network based on a normalization parameter ratio, a normalization entropy, a normalization variance, and a normalization activation sparsity; configured to determine a target bit width of each layer network based on real-time device state data and the inter-layer importance evaluation result; and configured to generate a target mixed-precision quantization network responding to real-time device states based on the target bit width of each layer network and the adaptive bit width student meta-network.

[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising: a memory and a processor, which are in communication connection with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the generation method of the adaptive mixed-precision quantization network based on self-learning described above.

[0018] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the generation method of the adaptive mixed-precision quantization network based on self-learning described above.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer instructions, and the computer instructions are used to make a computer execute the generation method of the adaptive mixed-precision quantization network based on self-learning described above. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed to be used in the description of the specific embodiments or prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A schematic diagram of a method for generating an adaptive mixed-precision quantization network based on self-learning provided by an embodiment of the present application is shown.

[0022] Figure 2 A schematic diagram of a system for generating an adaptive mixed-precision quantization network based on self-learning provided by an embodiment of the present application is shown.

[0023] Figure 3 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0025] In modern power systems, it is crucial to ensure the safe and stable supply of electricity. As the scale of power systems continues to expand and the complexity continues to increase, the monitoring, early warning and handling of various power failures and safety hazards become increasingly difficult. Deep learning technology has been widely applied in the field of power system safety monitoring and fault diagnosis due to its powerful data processing and pattern recognition capabilities. For example, by constructing a deep neural network model to analyze the operation data of power equipment, potential equipment failures can be predicted in advance to ensure the reliable operation of the power system. However, the amount of data generated by the power system is huge, and there are a large number of resource-constrained devices in the operating environment, such as edge computing nodes and smart meters distributed throughout the power network, which puts extremely high requirements on the computational complexity and storage requirements of deep learning models.

[0026] Quantization techniques for neural networks aim to reduce the computational complexity and storage requirements of deep neural networks to adapt to resource-constrained devices and scenarios. Traditional quantization methods are trained for a single fixed bit-width, and when quantized to other bit-widths, the performance will decrease significantly, which limits the flexibility of the model under different precision requirements. To support multiple bit-widths, multiple training is usually required, training multiple copies of the model into different bit-width models, which not only consumes time and effort, but also increases storage costs. In recent years, adaptive quantization methods have gradually attracted attention, which allows flexible selection of the bit-width of deep neural networks during inference to meet dynamically changing requirements. In particular, after training, weights and activation can be freely quantized to various precision levels without additional fine-tuning or calibration. However, existing adaptive bit-width quantization methods still have some problems: (1) When different bit-widths share weights for joint training, it is difficult to balance the optimization difficulty between different precisions. High precision often dominates in training, leading to poor performance of low bit-width quantization, which cannot meet the high precision requirements of power equipment state monitoring. (2) Current adaptive bit-width quantization methods support multiple precision switching, but require all layers of the network to use a unified bit-width configuration, and only the same bit-width can be used in the entire network. If mixed precision quantization (i.e., different layers use different bit-widths) is attempted, distribution shifts will occur when the bit-width changes between consecutive layers, which limits its flexibility in real-world scenarios. (3) Existing mixed precision network quantization requires a complex and computationally expensive optimization process to determine the optimal bit-width for each layer, which is not suitable for adaptive bit-width quantization scenarios.

[0027] In recent years, with the wide application of deep learning technology in computer vision and image processing, model optimization technology has gradually become a research hotspot. Among the many optimization techniques, quantization technology has attracted much attention due to its ability to effectively reduce model storage requirements and computational complexity, and improve model running efficiency. In order to further improve the quantization effect, scholars have made various contributions in reducing quantization error, and have adopted different methods and techniques for research from different angles. Mixed precision quantization (MPQ), as an important branch of quantization technology, aims to reduce model size and computational requirements by applying different bit widths to weights and activations, while maintaining or even improving model accuracy. Early research relied on manually determining bit width combinations based on experience, but this approach had limitations. Therefore, mixed precision quantization emerged and developed three main methods: search-based methods combined neural architecture search (NAS), reinforcement learning (RL), or evolutionary algorithms to explore the best bit width for each layer; model feature-based methods analyzed the importance or sensitivity of network layers to guide the quantization process; and learning-based methods treated bit width as a learnable parameter, determining the best bit width for each layer through training and fine-tuning. These methods have made significant progress in optimizing model performance and compression efficiency. In order to maximize the efficiency and deployability of deep neural networks in resource-constrained environments, the combination of quantization technology with other compression techniques has become a research trend. Weight sharing combined with quantization reduces the number of unique parameters in the model through clustering techniques and quantizes shared weights to fewer bits, thereby reducing storage requirements and computational load. Knowledge distillation combined with quantization trains a smaller "student" model to mimic a pre-trained "teacher" model, with the student model using lower-precision weights and activations while retaining most of the teacher model's accuracy. In addition, hybrid compression strategies combine techniques such as pruning, weight sharing, and quantization, with pruning introducing sparsity to ensure that quantization operations are more efficient, resulting in faster inference times and reduced memory usage.

[0028] The above methods have improved the quantization effect and compression efficiency of the model to some extent, but existing research still has some problems. For example, mixed precision quantization methods have high computational cost when determining the best quantization bit width for each layer, and are mostly based on fixed bit width configurations for feature extraction, ignoring the important relationship between layers.

[0029] The embodiment of the present application provides a method for generating an adaptive mixed precision quantization network based on self-learning. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0030] Please refer to Figure 1 ,Figure 1 The schematic diagram of the generation method of the adaptive mixed-precision quantization network based on self-learning provided by the embodiment of the present application is applied to power equipment state monitoring, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S100: obtaining a teacher bit width set and a student meta-network to be trained, the teacher bit width set comprising a plurality of high bit width candidate teachers, and the student meta-network sharing full-precision weights and supporting multiple bit width configurations.

[0032] The teacher bit width set is composed of high bit width candidate teachers of different bit widths, and is used for training and guiding low-precision student meta-networks.

[0033] Step S300: for any bit width configuration, determining a corresponding target teacher according to the inter-layer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher; jointly training all bit width configurations of the student meta-network to obtain an adaptive bit width student meta-network, comprising: for any bit width configuration, determining the training loss between the student meta-network and the target teacher, weighting and summing the training losses under all bit width configurations to obtain a joint loss function, and updating the shared full-precision weights based on the joint loss function.

[0034] Specifically, the Manhattan distance between the candidate teachers and each layer network in the student meta-network is calculated to measure the similarity between the candidate teachers and each layer network. The smaller the similarity, the closer the weight distribution of the two, and the higher the knowledge transfer efficiency. The entropy of the prediction probability distribution output by the candidate teacher can be used to evaluate the prediction confidence of the candidate teacher. The smaller the entropy value, the higher the prediction confidence. The hyperparameter is used to balance the weights of the Manhattan distance and the entropy value of the prediction probability distribution in the teacher selection process, and the hyperparameter is dynamically adjusted according to the training progress to optimize the adaptive bit width network quantization training process, select the target teacher for the student meta-network of the current bit width, so that each layer network in the student meta-network can learn more suitable knowledge in the training process, and provide a basis for subsequent inter-layer importance evaluation.

[0035] Step S500: generating inter-layer importance evaluation results of each layer network based on the normalization parameter ratio, the normalized entropy, the normalized variance, and the normalized activation sparsity; determining the target bit width of each layer network based on real-time device state data and the inter-layer importance evaluation results; and generating a target mixed-precision quantization network responsive to real-time device state based on the target bit width of each layer network and the adaptive bit width student meta-network.

[0036] The layer importance evaluation is evaluated by four evaluation indexes to avoid artificial presetting bias. The four statistical indexes are normalized parameter ratio, normalized entropy, normalized variance, and normalized activation sparsity. The final layer importance score is not a simple weighted average, but is dynamically adjusted by a learnable weight multiplier. The weight multiplier is determined by referring to the importance of the knowledge learned by each layer in the self-learning process. The normalized parameter ratio refers to the ratio of the number of parameters in the layer to the total number of parameters in the model. The normalized entropy is used to evaluate the information richness of the parameters in the layer. The higher the entropy value, the more chaotic the parameter distribution, and the more sensitive the layer (requires high bit width precision). The normalized variance is used to represent the feature diversity of the parameters in the layer. The layer with large variance carries more key features and is sensitive to quantization. The normalized activation sparsity is used to measure the redundancy of the output of the layer. The layer with high sparsity can tolerate low bit width quantization. The normalized essence compresses the values of the four indexes to 0 to 1, which is convenient for weighted summation. The bit width search strategy generates a scheme with the best bit width cost by using a greedy algorithm. Starting from the most important layer, the best bit width precision of each layer is determined by fine-tuning from the maximum possible bit width to a lower bit width under the condition that the accuracy loss is within the threshold range.

[0037] The generation method of the adaptive mixed precision quantization network based on self-learning proposed in the embodiments of the application is applied to power equipment state monitoring. By combining a knowledge distillation training progress adaptive joint loss, a target teacher selection method based on inter-layer Manhattan distance and entropy values of each candidate teacher prediction probability distribution is proposed. The most suitable teacher network is selected for each bit width. The dynamic learning process is used to automatically generate a quantization strategy configuration, which significantly improves the performance of low bit width quantization and optimizes the training process. The importance of each layer is calculated by using normalized parameter ratio, normalized entropy, normalized variance, and normalized activation sparsity. The inter-layer importance evaluation scheme is developed. Based on the inter-layer importance evaluation results and real-time device state data, the target bit width of each layer network is determined. Based on the target bit width of each layer network and the adaptive bit width student meta-network, a target mixed precision quantization network responding to real-time device state is generated. The optimal bit width is allocated to each layer of the neural network, which maximizes the compression efficiency while ensuring accuracy. The technical problem of low diagnostic accuracy of related lightweight models in power equipment state monitoring is solved, and the technical effect of improving the accuracy of power equipment state monitoring is achieved.

[0038] In some embodiments, historical device state data is acquired, and multi-dimensional features in the historical device state data are extracted, including device running state indicators, abnormal feature patterns, and time series change trends; based on the multi-dimensional features, a device state-bit width mapping strategy table is generated, which is used to represent the configuration strategy of each layer bit width corresponding to different power device states; the real-time device state data is matched with the device state feature vectors in the device state-bit width mapping strategy table to determine the target bit width of each layer network.

[0039] In power system safety monitoring and fault diagnosis, real-time analysis of device state data is crucial for fault warning. Based on device running state indicators, abnormal features, and time series change trends of device parameters, through a pre-generated device state-bit width mapping strategy table, the quantization bit width of each layer is adjusted in real time to achieve the goal of reducing cost in normal state and preserving accuracy in abnormal state.

[0040] Specifically, the historical device state data is first analyzed, a multi-dimensional feature library containing device running state, abnormal feature pattern, and time series change trend is constructed, and a device state-bit width mapping strategy table is pre-generated to represent the optimal quantization bit width configuration of each layer network under different running scenarios. For example, in normal operation, low bit width is used to reduce computational overhead; in abnormal fluctuation, high bit width is automatically switched to capture detailed features; in real-time monitoring, the system collects current device data in real time, matches it with the mapping strategy table, and quickly calls the corresponding bit width configuration strategy according to the matching result to dynamically adjust the quantization bit width of each layer, optimize the allocation of computing resources while ensuring the accuracy of fault diagnosis, and achieve fast response to power faults and safety hazards.

[0041] In some embodiments, the real-time indicators and fluctuation characteristics of the historical device state data are analyzed, and the data real-time strength and abnormal fluctuation frequency are evaluated based on the real-time indicators and fluctuation characteristics; if the data real-time is strong and the abnormal fluctuation frequency is high, the probability is shortened and the round interval is reduced to speed up the transformation rate, otherwise, the round interval is extended to slow down the transformation rate, to dynamically adapt the training probability to the power real-time demand.

[0042] In some embodiments, for real-time data such as voltage and current collected by power grid end-to-end devices, as well as device state, fault alarm and other information, due to the strong time series and noise interference of the data, traditional quantization cannot preserve the features. By introducing knowledge distillation, a well-trained high-precision model is used as a teacher to pass the knowledge it has mastered, such as voltage fluctuation trend and stability threshold, to the adaptive bit width quantization student model in the training process, optimizing the training process and improving the adaptability of the model to complex working conditions of the power grid.

[0043] In some embodiments, the training loss is a weighted combination of a labeled data supervised loss and a knowledge distillation loss, wherein the labeled data supervised loss is a cross-entropy loss, and the distillation loss is a KL divergence loss, and the network parameters are updated synchronously by gradient accumulation.

[0044] To minimize the quantization loss, the QAT is defined as an optimization problem, and the optimization objective is to minimize the discretization error and the difference between the model prediction and the true label, and to achieve the optimal quantization representation under the premise of ensuring the prediction accuracy. The optimization objective is defined as follows:

[0045]

[0046] In the above formula, is the training loss; is the original unquantized real-valued weight, which is the variable to be adjusted in the optimization process; is the real-valued weight is the quantized weight obtained after the quantization operation. By adjusting the unquantized real-valued weight , the loss function reaches the minimum, thereby achieving the optimal quantization of the model, reducing the discretization error and the difference between the model prediction and the true label, under the premise of ensuring the prediction accuracy.

[0047] wherein the training loss can be a labeled cross-entropy loss or other loss function, such as a distillation loss. By transferring the knowledge of a large parameter network to supervise the training of a small neural network through knowledge distillation, the performance can be significantly improved, sometimes even exceeding that of the large network. The training loss is defined as a weighted combination of a labeled data supervised loss and a knowledge distillation loss, with the parameter controlling the weight, and the training loss in the equation is defined as:

[0048]

[0049] In the above formula, can be a single distillation loss or a multi-item distillation loss, and the cross-entropy loss between the prediction distribution of the student model and the true label (referred to as ) and the KL divergence loss between the candidate teacher and the student model (referred to as ) are brought into the equation, wherein quantifies the alignment between the prediction of the low-bit network and the true label, and measures the similarity between the low-bit network and the high-bit network:

[0050]

[0051] and respectively, are the soft labels provided by the teacher model and the output of the student model. During training, the weights of the teacher are frozen and it only performs forward propagation. In forward pass, the parameters of the student are quantized while the corresponding full-precision values are kept internally. During backward propagation, the full-precision values kept by the student are updated gradually. After convergence, the student keeps its full-precision weights and the parameters used in quantizer. To achieve bit-width adaptation, a set of bit-widths is given during joint training , the loss of each quantization configuration is calculated synchronously, and the parameter update is performed through gradient accumulation. The total loss is denoted as , and the precision loss of the bit-width is denoted as , where . Therefore, the joint training loss function is obtained:

[0052]

[0053] In the above formula, denotes the probability distribution output by the current model at precision b.

[0054] In some embodiments, a linear annealing strategy is adopted to dynamically adjust the weight coefficients of the cross-entropy loss and the KL divergence loss: in the early stage of training, the cross-entropy loss is focused on to establish the category discrimination boundary, and the weight coefficient of the KL divergence loss is gradually increased as the training progresses.

[0055] In the process of training a deep learning model, dynamically adjusting the weight distribution strategy of the loss function has a significant impact on the optimization of model performance. In the early stage of training, when the model has not fully captured the data distribution characteristics, the cross-entropy loss is preferred as the dominant optimization target, and the preliminary category discrimination boundary is established by minimizing . In this stage, the model parameters are in a period of rapid adjustment, and by minimizing the prediction error, the model can quickly fit the overall distribution characteristics of the training data. When the model enters the late stage of convergence, i.e., close to or reaches the state of convergence, relying solely on the prediction error for optimization may lead the model to fall into a local optimal solution, or even overfitting. Therefore, the weight coefficient of the knowledge divergence loss is gradually increased during the training process. This method not only improves the convergence efficiency of the model, but also effectively balances the fitting ability and generalization performance of the model. Linear annealing is adopted to update , and the total number of training rounds is set to , the fixed weight of the cross-entropy loss is set to , and the linear annealing strategy is used to adjust it, i.e.,

[0056]

[0057] wherein, , is a fixed weight for cross-entropy loss, is the total training round.

[0058] In some embodiments, the weight of the entropy value and the inter-layer Manhattan distance is dynamically balanced by a hyperparameter, and the hyperparameter is linearly adjusted with the training progress to determine the target teacher corresponding to the current bit-width configuration.

[0059] In the training of quantized neural networks, high-precision networks can capture more rich input feature representations due to their larger model capacity, while low-precision networks can achieve model compression advantage by reducing computational precision. However, when selecting a teacher, it is not necessarily true that the more complex the model, the higher the performance. Specifically, the gap in representation ability between the low-precision student and the high-precision teacher will lead to a decrease in knowledge transfer efficiency, which has been verified in various studies. This essentially violates the capacity matching assumption in knowledge distillation, i.e., the representation spaces of the teacher and the student need to maintain a moderate overlap area to achieve effective knowledge transfer. When the precision of the teacher network is much higher than that of the student model, the difference in feature space distribution between the two will significantly reduce the distillation efficiency. The difference in representation ability between the teacher and the student needs to satisfy:

[0060]

[0061] In the above formula, , are the ability indicators of the teacher and the student, respectively, is a threshold value, otherwise the student cannot effectively imitate the teacher.

[0062] By dynamically selecting the optimal teacher network, knowledge is transferred from the existing high-precision teacher branch to the low-precision student branch, while a single network with all bit widths is optimized. Specifically, during the training process, given a set of quantization bit widths: For bit width , by balancing the confidence of the prediction and the distance in the model space, the best teacher is adaptively selected from the teacher bit width set according to the current input.

[0063] Given a higher precision as a candidate teacher for bit width , generally the closer the distance, the higher the similarity between the two networks, and the more suitable it is as a target teacher. The farther the distance, the more difficult it is to effectively learn the knowledge of the teacher network. Although it is usually difficult to measure the distance in the model space, it is particularly easy for adaptive bit width networks due to the shared weights. Inspired by physical distance, Manhattan distance is used to calculate the distance between layers as an important evaluation strategy for teacher selection. For a convolutional layer, let the weight tensors of the teacher network and the student model at a certain convolutional layer be , wherein, is the number of input channels, is the number of output channels, is the size of the convolution kernel, the weight distance between the student filter and the teacher filter is calculated as follows:

[0064]

[0065] wherein, is a specific weight value of the filter of the first channel weight matrix; and are the index numbers of traversing the filter matrix. Based on the above formula, the average distance of all filters in the same layer of the teacher network and the student model is calculated as the weight distance of the layer, and the closer the inter-layer distance is, the higher the similarity of the two networks is, and the more suitable the teacher network is, which is specifically calculated as follows:

[0066]

[0067] For the fully connected layer, let the weight matrix of a certain fully connected layer of the teacher network and the student model be , wherein, is the input feature dimension, is the output unit number, the weight distance between the student neuron weight vector and the teacher neuron weight vector is calculated as follows:

[0068]

[0069] Based on the above formula, the average distance of all neurons in the same fully connected layer of the teacher network and the student model is calculated as the weight distance of the layer, and the closer the inter-layer distance is, the higher the similarity of the two networks is, and the more suitable the teacher network is, which is specifically calculated as follows:

[0070]

[0071] Assuming that the fully connected layer and the convolution layer are equally important, the distance between the two quantified networks is defined as:

[0072]

[0073] wherein, is the number of layers, and student network and teacher network, respectively. The smaller the distance between the two networks, the more highly matched the teacher network is in spatial filtering characteristics with the student network, and the stronger the effectiveness of the multi-scale feature transfer. However, the model spatial distance alone is not enough to select the most suitable teacher. The entropy of the probability distribution of the teacher network after softmax is used to evaluate the prediction confidence of the teacher model. The greater the entropy of the probability distribution of the model output, the more uncertain the prediction. There is a trade-off between prediction confidence and model distance. The network with higher accuracy usually has greater capacity and gives more reliable prediction, and it is farther away from the network with lower accuracy. Therefore, the best teacher of the current batch with the bit width of the current batch is selected in the following way:

[0074]

[0075] wherein, is the probability distribution of the teacher network with the bit width is the probability distribution of the student network with the bit width is a hyperparameter balancing the trade-off, which is dynamically adjusted using a linear annealing strategy, and the specific calculation formula is:

[0076]

[0077] wherein, is the maximum value of the hyperparameter, is the minimum value of the hyperparameter, is the current training round, is the total training round. When , the selection is biased towards the highest accuracy. When ​When very large, it tends to select the closest model. It is found that the preference of student model to teacher also changes during training. Initially, the higher precision model improves faster in performance, and the student model prefers to select higher precision. As training proceeds, the difference between different precisions is alleviated, and it favors closer precision. Therefore, the dynamic teacher selection strategy is more adaptive to the current input and training progress than the manually fixed teacher. The dynamic teacher selection is realized based on the joint evaluation of model space distance and prediction confidence. For each training batch, the target bit width is selected from the teacher bit width set by calculating the total distance between networks accumulated by the weight distance of each layer, combined with the teacher prediction entropy, and the weight of the two is balanced by the hyperparameter to dynamically select the optimal teacher. In the early stage of training, the teacher network with high bit width and high confidence is preferred to quickly guide the low precision network to establish the preliminary performance; as the training proceeds, the model space distance and the prediction confidence between the teacher network and the student model are gradually balanced, and the teacher network closer to the bit width of the student model is selected to realize efficient knowledge transfer; in the later stage of training, more attention is paid to the model space distance, and the teacher network with bit width close to the student model is selected for fine adjustment, so as to realize the minimum bit width requirement while maintaining high accuracy. This dynamic selection method can effectively improve the performance of low precision network, reduce the negative impact of high precision network on it, and realize efficient knowledge transfer and model optimization.

[0078] In some embodiments, the training process employs a quantization function of a floor operator to learn quantization steps for weights and activations of each layer network separately.

[0079] The activation value distribution of power grid data is different under different bit widths. For example, the voltage data fluctuates slightly and is concentrated during normal operation, but the voltage data changes dramatically and the distribution expands during fault. In the embodiments of the application, low-precision quantization is used for slowly changing basic data such as load forecasting, to reduce the calculation and storage overhead; high-precision quantization is used for real-time data such as fault instantaneous current and voltage, to capture data details and realize flexible and efficient mixed-precision quantization.

[0080] First, for a given layer , the quantized representation of the weight and the input activation are denoted as and , respectively, where denotes the bit width. The quantization function for bit width is defined as:

[0081]

[0082] where is a floor operator, This represents a learnable scalar, defining the quantization step size, similar to the LSQ method. For each layer and bit width, two such scalars are learned, one for weights and the other for activation. The rounding function used in the LSQ method is then applied. Replace with rounding down. This allows direct access from... Get It eliminates the need to store full-precision weights, thus significantly reducing model storage requirements; its size is only determined by the given highest-precision quantized network. Size determines it.

[0083] In some embodiments, a dedicated BN layer is independently learned for each pair of adjacent layer bit-width transition combinations; the distribution shift caused by the bit-width change between consecutive layers is quantized by calculating the difference between the activation mean and variance before and after quantization; and the distribution shift is compensated by dynamically adjusting the scaling parameters and offset parameters of the BN layer.

[0084] Building upon the switchable batch normalization layer, a dedicated batch normalization layer (BN layer) is independently learned for each pair of adjacent layer bit-width transition combinations, achieving dynamic alignment of feature distributions. Specifically, assuming there is a set of bit-width candidates... ,save The parameters and internal state copies of each BN layer are denoted as follows: .if It is a layer bit width, It is a layer bit width, learning parameter and .parameter and Still with layers bit width They are correlated because they depend only on the current quantization level and are independent of the layer weights, as shown in the following formula:

[0085]

[0086] In the above formula, The output activation value after processing by the transition batch normalization layer; The original activation values ​​are input to the current transition batch normalization layer; for The mean value in the current batch is used to center the input. for The variance in the current batch is used to measure the dispersion of the distribution of activation values.

[0087] The meta-network is trained using a two-stage approach. The procedure is summarized as follows: Stage 1: In this stage, the network weights remain real-valued, and in each mini-batch, forward propagation is performed for all bit-widths, the training loss for all bit-widths is accumulated based on the knowledge distillation method, and then backpropagation is performed to update the parameters. The network trained by the adaptive bit-width quantization research belongs to this stage. Stage 2: By continuing the training process from the last checkpoint, in this stage, the probability is trained in the same way as described in stage 1 (i.e., using the same bit-width in all layers). At other times, i.e., with probability , the bit-width of each individual layer is randomly selected independently of each other, thus forming a network in which different layers use different bit-widths. As training progresses, the probability is gradually reduced, effectively increasing the chances of training the meta-network with a hierarchical random bit allocation. This process is continued until , where is typically .

[0088] In some embodiments, the inter-layer importance is evaluated by analyzing the degree of influence of each layer on the calculation result, and different quantization precision combinations are tried to generate multiple mixed precision sub-networks, which are then tested using historical power grid fault diagnosis data. According to the fault recognition accuracy, diagnosis speed and other indicators, the mixed precision sub-network with the best performance under the average bit width is finally determined.

[0089] First, the number of parameters in each layer affects the overall model size. Layers with more parameters are prioritized for quantization to reduce model size without affecting accuracy. The normalized parameter ratio is considered, and its calculation formula is:

[0090]

[0091] wherein denotes the normalized parameter ratio of the th layer, which measures the proportion of the parameter quantity of the layer in the total parameter quantity of the model, reflecting the contribution of the layer to the model size. denotes the parameter quantity of the th layer, i.e., the total number of learnable weight parameters in the layer of the neural network. denotes the total parameter quantity of the entire model, i.e., the sum of the parameter quantities of all layers, which is used to normalize the parameter quantities of the layers, making the parameter ratios of different layers comparable.

[0092] Secondly, the entropy of the parameters affects the bit demand, and higher entropy indicates more information, thus requiring more bit-width. The normalized entropy is considered, and its calculation formula is:

[0093]

[0094] wherein, denotes the normalized entropy of the k-th layer, which measures the matching degree between the information richness of the layer parameters and the quantization bit width. The normalized value can reflect the demand of parameter distribution on quantization precision. The higher the value is, the more information the parameter carries, and the higher bit width is needed to preserve information. denotes the zero-order entropy of the k-th layer parameters, which is the information entropy calculated by the distribution probability of parameter values, and is used to quantify the information uncertainty of the layer parameters. denotes the target bit width of the quantization model, which is used to normalize the zero-order entropy, so that the entropy indicators under different bit widths are comparable. The parameter distribution drives the bit precision requirement, which has been fully demonstrated. Most layers in DNN models follow Gaussian distribution, and the variance represents compactness. Considering the normalized variance, the calculation formula is:

[0095]

[0096]

[0097] wherein, denotes the normalized variance of the k-th layer, which is used to measure the dispersion degree of the parameter distribution of the layer. The variance ratio is mapped to a specific range through logarithmic transformation, reflecting the demand of parameter distribution on quantization bit width. The smaller the value is, the more concentrated the parameter distribution is, and the smaller the accuracy loss may be during quantization, which can adapt to lower bit width. The larger the value is, the more dispersed the parameter distribution is, and higher bit width is needed to preserve details. denotes the variance of the k-th layer parameters, which is calculated as the average of the square of the deviation of all parameter values of the layer from the mean value, reflecting the fluctuation degree of parameter values around the mean value. The larger the variance is, the more dispersed the parameter distribution is; the smaller the variance is, the more compact the parameter distribution is. denotes the maximum value of the variance of all layer parameters in the network, wherein k traverses all layers of the network. This value is used to normalize the variances of each layer, so that the variances of different layers are comparable, and ensure that the normalized results fall within a reasonable range. Finally, the activation sparsity indicates the criticality of the layer. Layers with higher or close to zero activation may be less critical. Considering the normalized activation sparsity, the calculation formula is

[0098]

[0099]

[0100] wherein, denotes the normalized variance of the k-th layer, which is used to measure the dispersion degree of the parameter distribution of the layer. The variance ratio is mapped to a specific range through logarithmic transformation, reflecting the demand of parameter distribution on quantization bit width. The smaller the value is, the more concentrated the parameter distribution is, and the smaller the accuracy loss may be during quantization, which can adapt to lower bit width. The larger the value is, the more dispersed the parameter distribution is, and higher bit width is needed to preserve details. ​​​​The normalized activation sparsity of a layer, which measures the proportion of zero or near-zero activations in the layer. This indicator reflects the "criticality" of the layer, and the higher the value, the more invalid (or low-contribution) signals in the activations, which means the layer can be quantized with a lower bit-width to reduce computational overhead without significantly affecting model performance. The number of zero or near-zero activations in the layer. This parameter is used to quantify the absolute number of sparse activations in the layer. The number of zero or near-zero activations in the layer. This parameter is used to quantify the absolute number of sparse activations in the layer. The total number of activations in the layer, i.e., the number of all elements in the output feature map of the layer. This parameter is used to normalize the number of sparse activations, making the sparsity indicators of different layers comparable. The total number of activations in the layer, i.e., the number of all elements in the output feature map of the layer. This parameter is used to normalize the number of sparse activations, making the sparsity indicators of different layers comparable.

[0101] The importance of the final layer combines these components:

[0102]

[0103] where is a weight multiplier used to control the impact on the overall importance score, adjusted according to the optimization goal. Increase the proportion of the parameter weight in the scenario of limited computing resources of edge devices To compress the model, increase the activation sparsity weight of the central control system To accelerate the calculation, increase the entropy weight for transient feature extraction tasks and variance weight To preserve fault features. The optimization goal is to maximize the accuracy of power grid data processing and minimize the inference delay under the constraint of device computing power, and to enhance the robustness of the model.

[0104] In some embodiments, based on the inter-layer importance evaluation result, the target bit-width of each layer network is determined using a bit-width search strategy, including: according to the inter-layer importance evaluation result, arranging the layers in descending order of layer importance, and for each layer, performing the following operations: initializing the bit-width to the highest quantization bit-width supported by the current layer, and iteratively adjusting the bit-width to a lower bit-width by a fixed step size under the premise that the accuracy loss does not exceed a set threshold, to determine the target bit-width of the current layer.

[0105] In some embodiments, the network with the current bit-width configuration is tested for diagnostic accuracy on a power equipment fault dataset, and the accuracy loss of the current bit-width is obtained; according to the safety level of the power equipment and the accuracy requirement of fault diagnosis, the set threshold of accuracy loss is generated, and if the adjusted accuracy loss exceeds the set threshold during the adjustment of the bit-width configuration from high bit to low bit, the bit-width adjustment is terminated and the bit-width configuration of the previous bit is preserved.

[0106] where the layer bit-width selection follows the above layer importance calculation to obtain the inter-layer importance evaluation result of each layer. Given a trained network with a certain bit-width configuration, the bit-width selection algorithm is as follows: Layered adaptive bit-width student meta-network If the average bit width of the mixed precision network is specified Then it must satisfy ,in For the first Layer selection positioning width, For the first The number of parameters in the layer The total bit width cost of the required subnetwork, due to cost Sub-network set The bandwidth is finite, therefore the cost of generating a bit width using a greedy algorithm is... The best candidate is determined through a search process. The optimal bit width precision for each layer is determined by the search process, ensuring that the minimum bit width is achieved while maintaining high accuracy. Layers are ranked by importance, and the search process starts with the most important layer. For each layer, the search begins with the maximum possible bit width, and fine-tuning is performed towards lower layers, ensuring that accuracy loss is within a threshold. Within the range.

[0107] The self-learning-based adaptive mixed-precision quantization network generation method provided in this application effectively overcomes many drawbacks of traditional quantization methods and existing adaptive bit-width quantization methods:

[0108] First, addressing the technical problem that traditional adaptive bit-width network quantization suffers from significant quantization loss at low precision settings, and that the performance of low precision is affected by high precision due to shared weights across all bit widths, this application provides a self-learning-based adaptive mixed-precision quantization network generation method that incorporates knowledge distillation to design a joint loss function that adapts to training progress. Simultaneously, it proposes a target teacher selection method based on model prediction confidence and Manhattan distance between models, utilizing the network's own quantized copy as the teacher to adaptively select the most suitable teacher network for each bit width, significantly improving the performance of low-bit-width quantization and optimizing the training process.

[0109] Secondly, the self-learning-based adaptive mixed-precision quantization network generation method provided in this application adopts a specific quantization method to reduce model storage requirements. At the same time, it proposes a bit-width-sensitive transition batch normalization method. By independently learning a dedicated batch normalization layer for each pair of adjacent layer bit-width transition combinations, it solves the problem of sharp decline in model accuracy and learning process conflicts caused by dynamic changes in bit width due to differences in activation value distribution under different bit widths. It achieves flexible and efficient mixed-precision quantization, enabling the model to run stably under different bit-width configurations.

[0110] In addition, the method for generating an adaptive mixed precision quantization network based on self-learning provided by the embodiments of the present application calculates the importance of each layer by using a plurality of statistical indicators, formulates an inter-layer importance evaluation scheme, and then generates a bit width cost optimal candidate according to the scheme using a greedy algorithm, thereby determining the optimal bit precision of each layer starting from the most important layer while ensuring that the accuracy loss is within a set threshold range. Without additional training, the optimal mixed precision sub-network under a given average precision can be relatively quickly found, which provides strong support for achieving a better trade-off between accuracy and speed in practical applications. In terms of technical effects, taking ResNet18 in the CIFAR-10 dataset as an example, when the target average bit width is 4-bit, the accuracy of the mixed precision network searched by the mechanism reaches 93.38%, which is higher than the 93.34% of the fixed 4-bit precision quantization baseline model. The accuracy of the MobileNet V2 model under the 4-bit average bit width is 91.69%, which is also higher than the 91.61% of the fixed precision quantization baseline. On the ImageNet dataset, the accuracy of the 6-bit average bit width mixed precision network of the ResNet18 model reaches 68.88%, which is 0.09% higher than the single precision 6-bit quantization model, and at the same time, the theoretical calculation amount is reduced by about 33.3%. Compared with the method without inter-layer importance search, the precision loss is reduced by 40%-60% under the same average bit width. For example, in the ResNet18 model of the CIFAR-10, the accuracy of the mixed precision network with random allocation of bit width is 92.16%, which is improved by 1.22 percentage points under the mechanism, fully verifying the significant advantages of the mechanism in balancing the calculation resources and the model performance.

[0111] Please refer to Figure 2 , Figure 2 The schematic diagram of the generation system of the adaptive mixed precision quantization network based on self-learning provided by the embodiments of the present application is as follows: Figure 2As shown, the embodiment of the present application also provides a generation system of an adaptive mixed-precision quantization network based on self-learning, applied to power equipment state monitoring, the system comprising: an acquisition module configured to acquire a teacher bit width set and a student meta-network to be trained, the teacher bit width set comprising a plurality of high bit width candidate teachers, the student meta-network sharing full-precision weights and supporting multiple bit width configurations; a training module configured to determine a corresponding target teacher according to the inter-layer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher for any bit width configuration; to jointly train all bit width configurations of the student meta-network to obtain an adaptive bit width student meta-network, comprising: determining the training loss between the student meta-network and the target teacher for any bit width configuration, weighting and summing the training loss under all bit width configurations to obtain a joint loss function, and updating the shared full-precision weights based on the joint loss function; a generation module configured to generate an inter-layer importance evaluation result of each layer network based on a normalization parameter ratio, a normalization entropy, a normalization variance and a normalization activation sparsity; to determine a target bit width of each layer network based on real-time device state data and the inter-layer importance evaluation result; and to generate a target mixed-precision quantization network responsive to real-time device state based on the target bit width of each layer network and the adaptive bit width student meta-network.

[0112] Further function descriptions of the above-mentioned modules and units are the same as those of the corresponding embodiments, and will not be repeated here.

[0113] The generation system of the adaptive mixed-precision quantization network based on self-learning in the embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit, Application Specific Integrated Circuit) circuits, processors and memories executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0114] Please refer to Figure 3 , Figure 3 A structural schematic diagram of a computer device provided by the embodiment of the present application is shown in FIG. 1. Figure 3As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for the various components to communicate with one another. The various components communicate through the use of the various buses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 10 can process instructions for execution within the computer device, including instructions stored in the memory 20 or elsewhere to implement routines for displaying graphical information, such as a GUI on an external input / output device, such as a display device coupled to the interface 30. In some embodiments, multiple processors and / or multiple buses can be employed as appropriate, as will be appreciated by those skilled in the art. Additionally, various components of the computer device can be used for processing instructions according to the embodiments, as will be appreciated. The computer device can be one of a plurality of computer devices in communication with one another through a network, as is common in an enterprise setup, for example. Figure 3 The processor 10 is taken as an example in the embodiments.

[0115] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0116] The memory 20 stores instructions that are executable by the at least one processor 10, so as to enable the at least one processor 10 to perform the method shown in the embodiments.

[0117] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory that is remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0118] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned kinds of memories.

[0119] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0120] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code stored in a remote storage medium or a non-transitory machine readable storage medium and stored in a local storage medium to be downloaded through a network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, the processor, or the hardware, implements the method shown in the above embodiments.

[0121] The embodiments of the present application provide a computer program product, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method of any of the embodiments of the present application.

[0122] The system or module illustrated in the above embodiments can be implemented by a computer chip or entity, or by a product having certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0123] For the convenience of description, the above apparatus is described as various units in terms of functions to be described respectively. Of course, functions of the units can be implemented in one or more software and / or hardware in the implementation of the present application.

[0124] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0125] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowchart or flowsheet blocks. Figure 1 one or more flowchart or flowsheet blocks.

[0126] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowchart or flowsheet blocks. Figure 1 one or more flowchart or flowsheet blocks.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowchart or flowsheet blocks. Figure 1 one or more flowchart or flowsheet blocks.

[0128] It should also be noted that the term "comprising" or "including" or any other variation thereof is intended to cover the non-exclusive inclusion such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without limitation, an element preceded by "comprises a" or "comprises" does not, without more constraints, foreclose the existence of additional identical elements in the processes, methods, articles, or apparatuses that comprise the element.

[0129] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly explains the difference from other embodiments. In particular, the system embodiments are described simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments.

[0130] The above merely provides an example of the present application, but is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

[0131] Although the embodiments of the present application are described with reference to the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes shall fall within the scope defined by the appended claims.

Claims

1. A method for generating a self-learning based adaptive mixed-precision quantization network, applied to power equipment state monitoring, comprising the steps of: The method comprises: obtaining a teacher bit width set and a student meta-network to be trained, the teacher bit width set comprising multiple high bit width candidate teachers, and the student meta-network sharing full-precision weights and supporting multiple bit width configurations; for any bit width configuration, determining a corresponding target teacher according to the inter-layer Manhattan distance between the student meta-network and each candidate teacher and the entropy value of the prediction probability distribution of each candidate teacher; jointly training all bit width configurations of the student meta-network to obtain an adaptive bit width student meta-network, comprising: for any bit width configuration, determining the training loss between the student meta-network and the target teacher, weighting and summing the training losses under all bit width configurations to obtain a joint loss function, and updating the shared full-precision weights based on the joint loss function; based on the normalized parameter ratio, the normalized entropy, the normalized variance and the normalized activation sparsity, generating an inter-layer importance evaluation result of each layer network; based on real-time device state data and the inter-layer importance evaluation result, determining the target bit width of each layer network; based on the target bit width of each layer network and the adaptive bit width student meta-network, generating a target mixed precision quantization network responsive to real-time device state; The method further comprises: obtaining historical device state data and extracting multi-dimensional features from the historical device state data, including device operating state indicators, abnormal feature patterns and time series change trends; based on the multi-dimensional features, generating a device state-bit width mapping strategy table, which is used to represent the configuration strategy of each layer bit width corresponding to different power device states; matching the real-time device state data with the device state feature vectors in the device state-bit width mapping strategy table to determine the target bit width of each layer network; The method further comprises: according to the inter-layer importance evaluation result, arranging the layers in descending order of layer importance, and performing the following operations for each layer: determining an initialization bit width according to the real-time monitored device state and the highest quantization bit width supported by the current layer, and iteratively adjusting the bit width downward from the initialization bit width to a lower bit width by a fixed step size to determine the target bit width of the current layer, under the premise that the accuracy loss does not exceed a set threshold.

2. The method of claim 1, wherein, The training loss is composed of a labeled data supervised loss and a knowledge distillation loss, wherein the labeled data supervised loss is a cross-entropy loss, and the distillation loss is a KL divergence loss, and the network parameters are updated synchronously through gradient accumulation.

3. The method of claim 2, wherein, The method further comprises: dynamically adjusting the weight coefficients of the cross-entropy loss and the KL divergence loss using a linear annealing strategy: focusing on the cross-entropy loss to establish a class discrimination boundary in the early stage of training, and gradually increasing the weight coefficient of the KL divergence loss as the training progresses.

4. The method of claim 1, wherein, The method further comprises: dynamically balancing the weights of the entropy value and the inter-layer Manhattan distance through a hyperparameter, and linearly adjusting the hyperparameter with the training progress to determine the target teacher corresponding to the current bit width configuration.

5. The method of claim 1, wherein, The method further comprises: using a quantization function of a downward rounding operator in the training process to learn a quantization step size for the weights and activations of each layer network.

6. The method of claim 1, wherein, The method further comprises: independently learning a dedicated BN layer for each pair of adjacent layers of the bit width transition combination; quantifying the distribution offset caused by the continuous inter-layer bit width change by calculating the difference between the activation mean and variance before and after quantization; and compensating for the distribution offset by dynamically adjusting the scaling parameter and offset parameter of the BN layer.

7. The method of claim 1, wherein, The method further comprises: performing a diagnostic accuracy test on the current bit width configuration network on the power equipment fault data set to obtain an accuracy loss of the current bit width; and generating the set threshold of the accuracy loss according to the safety level of the power equipment and the fault diagnosis accuracy requirement, and if the adjusted accuracy loss exceeds the set threshold during the process of adjusting the bit width configuration from high bits to low bits, terminating the bit width reduction and retaining the bit width configuration of the previous bit.

Images