Inspection model training method and device based on small-batch quotient gradient system

Abstract

The invention discloses an inspection model training method and device based on a small-batch quotient gradient system, and belongs to the technical field of power inspection. The method comprises the following steps: S1, constructing a power system inspection data set; s2, training the deep learning network by adopting a small-batch quotient gradient method based on the power system inspection data set to obtain a power system inspection model; s3, using the trained power system inspection model to perform obstacle identification in the intelligent inspection process of the power transmission line of the power system; according to the method based on the small-batch quotient gradient, the thought of small-batch data set training and variable step size integral acceleration training are adopted, and the search direction is fitted in a Limited memory programming mode. The problem that an existing training method is prone to falling into a local minimum point or a global minimum point can be solved, the model precision is improved, and fast and accurate obstacle recognition is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power inspection, and in particular relates to an inspection model training method and device based on a small batch quotient gradient system. Background Art

[0002] Transmission line inspection is one of the daily tasks of the power department. If the inspection work is done effectively, the safety hazards of the transmission line can be discovered in time to avoid major accidents. However, with the development of the power system, the total mileage of transmission lines has increased exponentially, and the task of power line inspection has become more arduous and important, consuming a lot of manpower and material resources, and becoming a burden for the power department. A method that can automatically identify obstacles is needed to replace manual inspection, reduce investment costs, improve inspection efficiency, and achieve the purpose of real-time monitoring of the entire network.

[0003] Gradient descent is the most popular deep network optimization algorithm. Every state-of-the-art deep learning library contains various variants of the gradient descent algorithm. Among the many variants of the gradient descent algorithm, the stochastic gradient descent method (SGD) is the most widely used one in machine learning. SGD obtains gradient estimates from a single training data sample, which can greatly reduce the computational cost of large redundant data sets, and is easy to implement and has low algorithm complexity. However, because SGD cannot change the step size, the gradient estimate converges very slowly. Even when the objective function is a strongly convex function, SGD still cannot achieve linear convergence and will fall into a local optimum. The batch gradient descent method (BGD) uses the entire data set to determine the search direction, which can effectively reduce the number of iterations and converge more accurately in the direction of the extreme value. However, when training for large sample data sets, all samples need to be calculated at each iteration step, resulting in a slow training process and easy memory overflow. Mini-batch gradient descent (MBGD) reduces the computational cost by an order of magnitude through matrix operations and the use of mini-batches, and can produce better solutions than online stochastic gradient. However, it is less effective for training unbalanced data.

[0004] Since the gradient descent algorithm is based on the Taylor series to take a series of orders to iteratively solve the local minimum or global minimum of the network, the error is relatively large. For such problems, existing literature has proved the equivalence of the union of the regular stable equilibrium manifolds of the Quotient Gradient System (QGS) and the entire feasible domain of the OPF problem, and further proved that QGS is completely stable and each trajectory converges to an equilibrium manifold. This provides a theoretical basis for the calculation of the optimal power flow solution of the power system. On this basis, existing literature has proposed a state estimation algorithm based on the dynamic system by constructing a quotient gradient dynamic system, so that the degraded stable equilibrium manifold of the system corresponds to the solution of WLS, overcoming the limitations of the traditional WLS algorithm, with the advantages of strong robustness, strong convergence, and the ability to solve state estimation problems with residual constraints, so that the equality constraints and residual inequality constraints are strictly satisfied, ensuring the rationality of the solution. Existing literature has given a description and visualization of the feasible domain of the cogeneration system through QGS. Its theoretical basis is the relationship between the entire feasible domain of QGS and the union of non-degenerate stable equilibrium manifolds. And starting from any point, the trajectory must converge to an equilibrium manifold QGS, which shows the effectiveness of the trajectory-based method. In order to solve the problem of partial state estimation of distribution networks, existing literature has proposed a solution method for the partial state estimation problem of distribution networks based on QGS. An equivalence relationship is established between the objective of WLS and the DSEM of QGS, which has global convergence. Existing literature has proposed a QGS-based method that can be used to calculate feasible solutions for load demand. It does not require matrix inversion, thereby avoiding the ill-conditioned nature of the Jacobian matrix along the trajectory. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the above-mentioned related art at least to a certain extent.

[0006] To this end, the purpose of the present invention is to provide an inspection model training method and device based on a small batch quotient gradient system, which can solve the problem that the existing training methods are prone to fall into local minimum points or global minimum points, improve model accuracy, and achieve fast and accurate obstacle recognition.

[0007] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: An embodiment of the present invention provides an inspection model training method based on a small batch quotient gradient system, the method comprising: S1. Construct a data set of transmission line obstacle inspection; S2. Based on the transmission line obstacle inspection dataset, a small batch quotient gradient-based method is used to train the deep learning network. By reducing the computational complexity, the memory limitation problem when the data volume is too large is solved. By adding a weight correction function, the overfitting problem of large category training and the underfitting problem of small category training are solved, and the overall performance of the class imbalanced dataset is improved to obtain a better transmission line obstacle inspection model. S3. Use the trained transmission line obstacle inspection model to identify obstacles during the intelligent inspection of power system transmission lines.

[0008] In addition, the inspection model training method based on the small batch quotient gradient system according to the present invention may also have the following additional technical features: In some of the embodiments, the mini-batch quotient gradient based method adopts the idea of ​​mini-batch data set training and variable step size integral accelerated training, and fits the search direction in a limited memory programming manner.

[0009] In some of the embodiments, the solution of the deep learning network model is obtained by tracking the DSEM trajectory of the quotient gradient system.

[0010] In some embodiments, the deep learning network is a convolutional neural network, and its equality constraint function is a generalized equality constraint function that takes into account the loss function of a small batch, and its formula is: ; In the formula, represents the convolutional network loss function, represents the measurement residual constraint, S represents the slack variable, z is the objective function, h () is the loss function, x is the neuron weight, H E (x) represents an equality constraint, H I ( x , S ) represents an inequality constraint.

[0011] In some embodiments, the solution of the generalized equality constraint function is connected to the small batch quotient gradient system in a steady state; the steady state connection is in the form of: ; in, yes The Jacobian matrix of is the weight correction function, x is the neuron weight, S represents the slack variable.

[0012] In some of the embodiments, the specific method of obtaining the deep learning network model solution by tracking the DSEM trajectory of the quotient gradient system is: ; In the formula, x is the neuron weight, yes The Jacobian matrix of is the weight correction function, represents the penalty factor, S represents the slack variable.

[0013] In some implementations, a pseudo-transient continuous method is used to quickly calculate a steady-state solution, specifically including: By referring to the second norm results of the quotient gradient system calculated twice before and after, the training speed can be accelerated by following the correction of the search direction of the quotient gradient system.

[0014] In some implementations, the content of step S1 includes: obtaining raw data of power system inspection, The resnet-50 network is used to extract features from the original data, and the principal component analysis method is used to extract two-dimensional features. Then the data is classified according to the two-dimensional features to obtain the required transmission line obstacle inspection data set.

[0015] In some of the implementations, a weight correction function is added during data classification to solve the problem of large errors caused by underfitting of data; the weight correction function is a weight correction function that takes loss values ​​into consideration.

[0016] In some of the embodiments, the mini-batch size when the mini-batch quotient gradient system is used for model training is 128, and the number of iterations for each mini-batch is 20.

[0017] An embodiment of the present invention also provides an inspection model training device based on a small batch quotient gradient system, comprising a processor and a memory, wherein a software program is stored on the memory, and when the processor runs the software program, the content of the inspection model training method based on a small batch quotient gradient system as described in any one of the above items can be implemented.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: In an embodiment of the present invention, the inspection model training method based on the mini-batch quotient gradient system is provided, and the model training method based on Minibatch QGS is used for training, which effectively solves the problem that when the amount of data is large enough, due to memory limitations, QGS cannot train the data normally, and can greatly improve the test accuracy of the trained model.

[0019] The inspection model training device based on a small batch quotient gradient system of the present invention can realize the content of the inspection model training method based on a small batch quotient gradient system, and thus at least has all the features and advantages of the inspection model training method based on a small batch quotient gradient system, which will not be repeated here. Additional aspects and advantages of the present invention will be partially given in the following description, and part will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of an inspection model training method based on a small batch quotient gradient system disclosed in one embodiment of the present invention; Figure 2 An embodiment of the present invention is disclosed Specific correction diagram of the function; Figure 3 A test error curve diagram of SGD and Minibatch QGS on a test set disclosed in one embodiment of the present invention; Figure 4 A two-dimensional characteristic distribution diagram of a power system inspection data set disclosed in an embodiment of the present invention; Figure 5 A graph showing the proportion of each category of data in a power system inspection data set disclosed in an embodiment of the present invention; Figure 6 This is a test error curve graph of SGD and Minibatch QGS on a test set disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The embodiments of the present invention are described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0023] See also Figure 1As shown, in some embodiments of the present invention, a patrol model training method based on a small batch quotient gradient system is provided to construct a transmission line obstacle patrol data set; based on the transmission line obstacle patrol data set, a patrol model training method based on a small batch quotient gradient system is used to train a deep learning network, and by reducing the computational complexity, the memory limitation problem when the data volume is too large is solved; by adding a weight correction function, the training overfitting problem of large categories and the training underfitting problem of small categories are solved, and the overall performance of the class imbalanced data set is improved to obtain a better transmission line obstacle patrol model; the trained transmission line obstacle patrol model is used to identify obstacles in the process of intelligent patrol of power system transmission lines; the patrol model training method based on the small batch quotient gradient system adopts the idea of ​​small batch data set training and variable step size integral acceleration training, and fits the search direction in a limited memory programming manner.

[0024] Furthermore, QGS was applied to deep network training and some improvements were made. The contents include: 1) QGS is applied to deep learning network training; 2) Adopt the idea of ​​mini-batch and variable step-size integration to accelerate training; 3) Limited memory programming method fits the search direction and trains large networks.

[0025] The mini-batch QGS trained a network with 3.87 million parameters, with an accuracy of 99.7% on the MNIST dataset and 96.3% on the power system inspection dataset.

[0026] The present invention first introduces the objective function of the convolutional neural network; then introduces the concept of mini-batch QGS, and proposes a deep network training method for an unbalanced data set; finally, the deep learning network based on mini-batch QGS is tested.

[0027] (1) Objective function of convolutional neural network and description of power system inspection dataset Convolutional neural network is a deep feedforward neural network model. Compared with other neural network structures, convolutional neural network requires relatively fewer parameters and is widely used. CNN has a multi-layer network structure, and its basic structure mainly includes five parts: input layer, convolution layer, pooling layer, fully connected layer, and output layer. The formula is as follows: ; In the formula, b is the offset, and Indicates l +1 layer convolution input and output, generating feature maps, yes Dimension. Represents the pixel at a certain point on the feature map, K is the number of channels of the feature map, f、 and p are the sizes of the convolution kernels respectively.

[0028] The convolutional layer is also called the feature extraction layer. It is the core of CNN. Its main function is to perform convolution calculations on the input data and complete the process of extracting image features through local calculations of different convolution kernels. After the convolution operation is completed, it enters the pooling layer. Pooling is achieved through pooling kernels. Generally, a continuous range in the image is selected as the pooling area, and the features generated by the same hidden units are clustered and counted. Therefore, these pooling units have translation invariance, which is also the most important feature of the pooling layer.

[0029] The optimized model of CNN is as follows: (4) In the formula, represents the convolutional network loss function, and β represents the measurement residual constraint.

[0030] To convert the constrained formula into an unconstrained formula, slack variables are added to the inequality in (4) and the penalty factor is used. As a measure of the remaining constraints: ; in, S represents the slack variable. In this case, the measurement residuals with higher accuracy are still unsatisfactory. In order to overcome this shortcoming, a method based on QGS will be adopted.

[0031] (2) The concept of mini-batch QGS and the deep network training method for imbalanced datasets Some dynamical systems have many excellent properties, such as the absence of convergence and near-convergence problems. If a nonlinear dynamical system can be constructed so that the local optimal solution to the problem corresponds to a suitable stable equilibrium point of the dynamical system, then the problem is transformed into a solution to the dynamical system. How to construct an effective dynamical system and solve it through its properties is the key to this method. The specific construction method and its properties will be introduced below.

[0032] By adding the mini-batch loss function to Equation (4), the following generalized equality constraint function is generated: (5) in, represents the convolutional network loss function, represents the measurement residual constraint, S represents the slack variable, z is the objective function, h () is the loss function, x is the neuron weight, H E (x) represents an equality constraint, H I ( x , S ) represents an inequality constraint.

[0033] In the proposed approach, a nonlinear dynamical system based on the equality constraint (5) is derived, linking the solution of (5) to the (stable) steady state of the QGS system: (6) in, yes The Jacobian matrix of is the weight correction function, x is the neuron weight, S represents the slack variable.

[0034] Theorem 1: Local Optimum Any stable equilibrium manifold of the quotient gradient system (6) is denoted by , is a function A local minimum point of .

[0035] Definition 3: Degenerate Stable Equilibrium Manifold For a stable equilibrium manifold of the quotient gradient system (6) ,if and ,but It is called a degenerate stable equilibrium manifold.

[0036] According to Theorem 1 and Definition 3, a degenerate stable equilibrium manifold of the quotient gradient system is the function A non-zero local minimum point of .

[0037] because: (7) Also for Therefore, by tracking the DSEM trajectory of QGS, the solution of the deep learning network optimization model can be obtained.

[0038] Therefore, problem (4) can be solved by solving QGS: (8) Balance analysis: In the power system inspection data set, the imbalance of data volume between categories leads to overfitting of large categories or underfitting of small categories. Traditional methods often give data with large errors a smaller weight, which is different in deep network training. Since the power system inspection training set is a data set carefully annotated by humans, the possibility of errors is extremely low. Therefore, when large errors occur, it is mainly because the data is underfitted, and a larger weight will be given to increase its contribution to the training gradient. The weight correction function added in the present invention is as follows: ; The specific correction diagram of the function is as follows: Figure 2 shown.

[0039] The present invention proposes a new loss weight correction function. γ It can reduce the relative loss of classifying large sample examples and focus more on classifying small sample examples.

[0040] Intuitively, this weight function can automatically reduce the weight of large sample examples during training and quickly focus the model on small class examples.

[0041] The mini-batch QGS training proposed in the present invention calculates the gradient on a small part of the data set, which effectively solves the problem that when the data volume is large enough, QGS cannot train the data normally due to memory limitations.

[0042] This has two benefits: 1) Reduce the variance of parameter updates, which can lead to more stable convergence; 2) Compared with the classic batch processing algorithm, the minibatch processing algorithm reduces the computational cost by an order of magnitude and produces better solutions than online learning.

[0043] In order to quickly calculate the steady-state solution, the present invention adopts a pseudo-transient continuation (Pseudo-transient Continuation, PTC for short) technology.

[0044] The present invention corrects the step size by the size of the QGS second norm and refers to the QGS second norm results calculated twice before and after, so that the training speed can follow the correction of the QGS search direction to speed up the training speed.

[0045] ; The present invention adopts the method of limiting memory to find the iteration direction, and approximates the matrix B (V is an m*n matrix, m< <n)。

[0046] ; in, It represents the initial value at the kth iteration. , represents the weight of the kth iteration, , , .

[0047] (3) Testing of deep learning networks based on mini-batch QGS The data sets used to test the performance of deep learning network training algorithms include the standard MNIST data set and the CIFAR-10 data set. In order to further test the performance of the small batch entropy gradient system training method in the field of power systems, the present invention creates a transmission line obstacle inspection data set. The data set contains 43,000 pictures, the image size is 32*32*3, and there are six categories: crane, tower crane, construction machinery, wire foreign matter, dump truck. Among them, the training set has 33,000 pictures and the test set has 10,000 pictures.

[0048] The experiments were conducted on 1 machine; 1 Intel CPU core (2.67 GHz) and a GeForce RTX2080TI GPU. The experiments were conducted using Python and its GPU plugin.

[0049] In the experiment below, a standard metric in machine learning is shown: the objective function evaluated over time on the test data (i.e., test error). The objective function evaluated during training shows a similar trend.

[0050] Computer calculation SGD and Minibatch QGS are used to train the convolutional neural network. The convolutional neural network has two convolutional layers, two pooling layers and one fully connected layer. Therefore, the model has about 3.22*10^7 (3.22 times 10 to the 7th power) parameters, which is a challenge for general optimization methods.

[0051] For the Minibatch QGS algorithm, there are two optimization parameters: the minibatch size and the number of iterations per minibatch. Minibatch was replaced after 20 iterations, and the minibatch size used was 128. A large number of experiments have shown that the above parameters are relatively effective.

[0052] For Minibatch QGS, the present invention uses the above fixed parameters for training; and for SGD, the present invention tries 10 optimization parameter combinations, including changing the mini-batch size in {1, 10, 100, 1000}, taking the number of iterations as a reference, and using values ​​between 0.001-1 for variable step sizes of different strategies.

[0053] Comparing the test errors of SGD and Minibatch QGS on the test set, the best results are summarized as follows Figure 3 The results show that Minibatch converges faster than a well-tuned vanilla SGD. In particular, Minibatch QGS performs better, with a maximum test accuracy of 99.7%.

[0054] The resnet-50 network is used to extract the features of the transmission line obstacle inspection dataset, and the principal component analysis method is used to extract two-dimensional features, such as Figure 4 As shown in the figure, it can be seen that the dispersion and intersection of construction machinery, foreign objects on conductors, and fireworks are very high; the proportion of each category of the data in the power transmission line obstacle inspection data set is statistically analyzed, such as Figure 5 As shown in the figure, the amount of data in each category is very uneven, with the largest data ratio reaching 40% and the smallest only 2%. The reason is that it is difficult to collect samples such as wildfires and kites hanging on power lines. Based on the above analysis, this classification task is very difficult.

[0055] The convolutional neural network is trained using computer calculation SGD and Minibatch QGS. The convolutional neural network has two convolutional layers, two pooling layers, and one fully connected layer. Therefore, the model has about 3.87*10^7 parameters to fit this dataset.

[0056] For the Minibatch QGS algorithm, Minibatch was replaced after 20 iterations, and the size of the minibatch used was 128. The test errors of SGD and Minibatch QGS on the test set were compared, and the best results are summarized in Figure 6 The results show that Minibatch converges faster than a well-tuned vanilla SGD. In particular, Minibatch QGS performs better, with a maximum test accuracy of 96.1%.

[0057] Table 1 Hidden danger category coding Test accuracy crane 98.4% Tower Crane 99.6% Construction machinery 97.3% Foreign matter in wire 94.7 Fireworks 92.2% Tipping bucket 93.5% The results show that Minibatch converges faster than a well-tuned vanilla SGD. In particular, Minibatch QGS performs better, with a maximum test accuracy of 96.1%.

[0058] Parts of the present invention that are not described in detail may refer to the prior art or are known to those skilled in the art.

[0059] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for training an inspection model based on a small batch quotient gradient system, characterized in that: The method comprises: S1. Construct a data set of transmission line obstacle inspection; S2. Based on the transmission line obstacle inspection dataset, a small batch quotient gradient-based method is used to train the deep learning network. By reducing the computational complexity, the memory limitation problem when the data volume is too large is solved. By adding a weight correction function, the overfitting problem of large category training and the underfitting problem of small category training are solved, and the overall performance of the class imbalanced dataset is improved to obtain a better transmission line obstacle inspection model. S3. Use the trained transmission line obstacle inspection model to identify obstacles during the intelligent inspection of power system transmission lines.

2. The inspection model training method based on a small batch quotient gradient system according to claim 1 is characterized in that: The method based on mini-batch quotient gradient adopts the idea of ​​mini-batch data set training and variable step-size integral accelerated training, and fits the search direction in a limited memory programming manner.

3. The inspection model training method based on a small batch quotient gradient system according to claim 2 is characterized in that: By tracking the DSEM trajectory of the quotient gradient system, the solution of the deep learning network model is obtained.

4. The inspection model training method based on a small batch quotient gradient system according to claim 1 is characterized in that: The deep learning network is a convolutional neural network, and its equality constraint function is a generalized equality constraint function that takes into account the loss function of a small batch, and its formula is: ； In the formula, represents the convolutional network loss function, represents the measurement residual constraint, S represents the slack variable, z is the objective function, h () is the loss function, x is the neuron weight, H E (x) represents an equality constraint, H I ( x , S ) represents an inequality constraint.

5. The inspection model training method based on a small batch quotient gradient system according to claim 4 is characterized in that: The solution of the generalized equality constraint function is steadily connected to the small batch quotient gradient system; the steady state connection is in the form of: ； in, yes The Jacobian matrix of is the weight correction function, x is the neuron weight, S represents the slack variable.

6. The inspection model training method based on a small batch quotient gradient system according to claim 3 is characterized in that: The specific method of obtaining the deep learning network model solution by tracking the DSEM trajectory of the quotient gradient system is: ； In the formula, x is the neuron weight, yes The Jacobian matrix of is the weight correction function, represents the penalty factor, S represents the slack variable.

7. The inspection model training method based on a small batch quotient gradient system according to claim 3 is characterized in that: The pseudo-transient continuous method is used to quickly calculate the steady-state solution, including: By referring to the second norm results of the quotient gradient system calculated twice before and after, the training speed can be accelerated by following the correction of the search direction of the quotient gradient system.

8. The inspection model training method based on a small batch quotient gradient system according to claim 1 is characterized in that: Step S1 includes: obtaining the original data of the power system inspection, The resnet-50 network is used to extract features from the original data, and the principal component analysis method is used to extract two-dimensional features. Then the data is classified according to the two-dimensional features to obtain the required transmission line obstacle inspection data set.

9. The inspection model training method based on a small batch quotient gradient system according to claim 1 is characterized in that: When performing data classification, a weight correction function is added to solve the problem of large errors caused by under-fitting of data; the weight correction function is a weight correction function that takes the loss value into consideration.

10. An inspection model training device based on a small batch quotient gradient system, comprising a processor and a memory, wherein a software program is stored in the memory, characterized in that: When the processor runs the software program, it can implement the content of the inspection model training method based on a small batch quotient gradient system as described in any one of claims 1 to 9.

Images