Model generation and real-time construction methods for unsupervised learning models

Abstract

The present invention proposes a model generation and real-time construction method for unsupervised learning models, including: step 1, obtaining the original model; step 2, generating a model block, obtaining a derived model block according to the model block, and constructing a training environment for each derived model block; step 3, training the derived model block and updating the parameters of the derived model block; step 4, evaluating all model blocks to quantify the performance indicators and use value of the model block; step 5, modeling the edge reasoning scenario, constrained optimization problem; step 6, solving the constructed optimization problem; step 7, generating a new model according to the solved model block selection scheme, and scaling the model. The method avoids the influence of instability in unsupervised learning on the generation of derived models, ensures the high precision of derived models; reduces the offline generation cost of derived models in unsupervised learning and the online cost of real-time scaling models in edge reasoning.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a model generation and real-time construction method for an unsupervised learning model. Background Art

[0002] At present, deep learning technology is developing rapidly and has been widely used in many fields, such as video target detection, natural language processing, intelligent recommendation, etc. The bottom layer of deep learning is deep neural network, referred to as "model", which can "reason" the input data to extract effective information from the input data, such as understanding the content in pictures, videos and audio. Models often rely on a large amount of labeled training data for supervised training. The so-called "labeled" means telling the model the effective information in the given training data (such as the category of objects in the given training pictures, the category and position of objects in the given training videos, etc.), and this information often requires a lot of labor costs to manually label. At present, it is difficult to label all valuable data. Therefore, unsupervised training corresponding to supervised training is proposed, that is, to make the training of the model independent of the labels of the training data, or to let the model infer and learn the labels of the training data by itself.

[0003] Typical techniques for unsupervised learning are generative adversarial networks and self-training. Generative adversarial networks usually include two models, namely the generator and the discriminator. The two learn from each other and improve together in the game, getting rid of the dependence on the training data labels; in self-training, the model first infers the training data with existing knowledge and labels it with pseudo-labels. Then the model accepts the training data and pseudo-labels in a supervised learning manner, and then updates the pseudo-labels in turn, and so on, achieving the process of "learning by itself", and also getting rid of the process of manually labeling training data labels. Such practices all have a problem: unstable training, the game and joint learning between the generator and the discriminator in the generative adversarial network can easily lose balance, resulting in the inability to converge training and affect the accuracy of the final model; the self-learning process in self-training is easily affected by the wrong labels from itself, resulting in the model getting worse and worse and unable to continue training.

[0004] In addition, with the explosive growth in the number of edge devices such as smartphones, more and more deep neural networks are being deployed in them to save data transmission costs and alleviate privacy issues. However, in order to improve the performance of DNN, the size and amount of computation of deep neural networks have been increasing in recent years to obtain stronger generalization performance and higher reasoning accuracy. Therefore, the model's reasoning on data requires high computing power and consumes a large amount of running memory and energy, which runs counter to the performance and energy-limited characteristics of edge devices; the scenario of multiple applications in parallel makes resources more scarce and constantly fluctuating, exacerbating this contradiction. The above problems make the research on edge reasoning optimization technology of unsupervised learning models urgent.

[0005] For example, the Chinese invention patent application number CN201911348985.6 discloses a sparse learning method, which adds the scaling factor in each channel in the model as a penalty term to the original loss function, and makes significant changes to the original training process. These practices have a significant interference with the training process of unsupervised learning, making it difficult for the model to converge, so they are not suitable for unsupervised learning models, and there are problems such as a limited number of derived models and high model scaling costs.

[0006] For example, the patent application document with Chinese invention patent application number CN202011341346.X discloses a compression method for generative adversarial networks, which replaces the original operators of each layer with multiple candidate operators with smaller computational complexity, and determines the final retained operators of each layer after training. Although this method solves the problem of low accuracy of derived models, it can only generate one derived model at a time, the offline training cost is high, and the number of derived models is limited; since the derived models are independent of each other, the model scaling cost is high.

[0007] For example, the patent application document with Chinese invention patent application number CN201910890284.9 discloses a quantization compression technology for generative adversarial networks. It reduces the size of the model and the amount of computational effort during runtime through pruning and low-bit quantization technology. This method also has problems such as high offline training cost, limited number of derived models, and high model scaling cost. Summary of the invention

[0008] In order to solve the above-mentioned technical problems existing in the prior art, the present invention provides a model generation and real-time construction method for an unsupervised learning model, which avoids the impact of instability in unsupervised learning on the generation of derived models and ensures the high precision of derived models; reduces the offline generation cost of derived models in unsupervised learning and the online cost of real-time scaling models in edge reasoning; defines the edge reasoning scenario as a single-objective linear programming problem, so that the optimal derived model can be quickly found under various available resource conditions, further optimizing model accuracy and resource utilization.

[0009] The present invention adopts the following technical solutions:

[0010] The present invention provides a model generation and real-time construction method for an unsupervised learning model, comprising:

[0011] Step 1: for unsupervised learning scenarios, define the attributes of the model and the model blocks and the operation methods of the model and the model blocks, and give the unsupervised model the ability to perform analysis and operation to obtain the original model;

[0012] Step 2, generating multiple model blocks from the original model, obtaining derived model blocks according to the model blocks, and building a training environment for each derived model block;

[0013] Step 3, training the derived model block and updating the parameters of the derived model block;

[0014] Step 4, evaluating all model blocks to quantify the performance indicators and usage value of the model blocks, wherein evaluating all model blocks includes evaluating the accuracy loss of derived model blocks;

[0015] Step 5: Select the optimal model block in the current edge inference scenario, with the model block selection scheme as the unknown quantity, the maximization of the model block accuracy as the goal, and the model block operating environment and application scenario as the constraints;

[0016] Step 6, solving the optimal selection scheme of the model block;

[0017] Step 7: scale the current model according to the model block selection scheme obtained by the solution to generate a new model.

[0018] Furthermore, the step 1 of obtaining the original model includes:

[0019] Step 1.1, inherit the model manager abstract class provided in the source code, and implement the missing methods in the model manager abstract class;

[0020] Step 1.2, inherit the model block manager abstract class provided in the source code, supplement and implement the missing methods in the model block manager abstract class, and obtain the original model.

[0021] Furthermore, step 2 constructs a training environment for each derived model block, including:

[0022] Step 2.1: According to the partitioning principle defined in the model block manager, n original model blocks b1, b2, ..., b n ;

[0023] Step 2.2: According to the derived block compression strength and derived block compression method defined in the model block manager, for each original model block bi (1≤i≤n), respectively, with compression strength Compress it to generate n i Derived Model Nuggets The original model block is generated A derived model block;

[0024] Step 2.3, arrange the source of training data for each derived block.

[0025] Furthermore, the step 3 of training the derived model block and updating the parameters of the derived model block includes:

[0026] Step 3.1: Input the training data used to train the original model into the original model, so that the original model completes one inference, and each original model block b in the original model i The input and output data of (1≤i≤n) are transmitted to the corresponding derived model block training environment;

[0027] Step 3.2: The original model block b i The input data (1≤i≤n) is transmitted to each derived model block b i，j (1≤j≤n i ) enables the derived model block to perform feedforward calculation and obtain the derived block b i，j Output data of

[0028] Step 3.3, calculate the original model block b i (1≤i≤n) and the original model block b i Each derived model block b i，j (1≤j≤n i ) output data. If the error value is less than the set threshold, it means that the derived model block has been fully trained and the training is terminated. Otherwise, continue to step 3.4. If all derived model blocks have stopped training, step 3 is terminated.

[0029] Step 3.4, using the back propagation algorithm and the gradient descent method, back propagate the error value and then perform gradient descent to update the parameters of the derived model block so that the error value between the derived model block and the original model block is less than the set threshold.

[0030] Furthermore, the step 4 evaluates all model blocks, including:

[0031] Step 4.1: Evaluate the volume s of the original model and the volume reduction S of the model block i，j (1≤i≤n, 0≤j≤n i ), the volume reduction of the original model block is 0, that is, S i，0 = 0; then the derived model block b i，jThe volume of its original block b i The volume difference is taken as the volume reduction S of the derived model block. i，j ;

[0032] Step 4.2, evaluate the accuracy loss of the derived model block;

[0033] Step 4.3, evaluate the delay reduction T of the model block i，j (1≤i≤n, 0≤j≤n i ).

[0034] Furthermore, the model block of building the edge reasoning scenario in step 5 includes:

[0035] Step 5.1, select the model block, A derived model block, using B i，j ∈{0, 1}(1≤i≤n, 0≤j≤n i ) to indicate whether to select model block b i，j , B i，j When it is 1, it means selection, and when it is 0, it means no selection. A model block is selected at a position in the model, that is,

[0036] Step 5.2, the optimization objective is to minimize the accuracy loss caused by the derived model block, which is expressed as the following formula (1):

[0037]

[0038] In the above formula (1), in order to ensure the normal operation of the application, it is necessary to consider the available running memory s from the operating system max constraints, as well as the maximum allowable delay constraints t from users and application scenarios max , respectively as shown in the following formula (2) and formula (3):

[0039]

[0040]

[0041] In the above formula (2) and formula (3), s is the volume of the original model, S i，j is the volume reduction of the derived model block, t is the inference delay of the original model, and t i is the original model block b i (1≤i≤n) inference delay, T i，j For the derived model block b i，j Relative to the original model block b i The percentage of inference latency reduction; In formula (2): the volume s of the original model minus the volume reduction brought by the derived block That is, get the volume of the current model. The volume of the current model must be smaller than the available memory of the current operating system. max , to ensure the normal operation of the model; In formula (3): the inference delay t of the original model minus the decrease in inference delay brought by the derived model block: That is, obtain the inference delay of the current model. The inference delay of the current model must be less than the maximum allowed delay t max , to meet user experience.

[0042] Furthermore, the model block constructed by solving in step 6 includes:

[0043] Step 6.1, first remove the integer constraints of the unknown quantities in the optimization problem, and change the constraint B i，j ∈{0, 1} is relaxed to 0≤B i，j ≤1, to relax the original integer linear program into a linear program;

[0044] Step 6.2, solve in polynomial time to obtain the optimal real solution to the original problem;

[0045] Step 6.3, use the branch and bound method to find the optimal integer solution based on the optimal real solution.

[0046] Furthermore, in step 7, the new model is generated according to the model block selection scheme obtained by the solution, and the new model is scaled, including:

[0047] Step 7.1, according to B i，j Assemble or switch model blocks, if B i，j Through the first solution optimization, all B will be loaded directly i，j = 1 model block b i，j Assemble into a complete model and end step 7; otherwise go to step 7.2;

[0048] Step 7.2, B i，j Combined solution with the model block obtained in the previous solution Compare them. If there are unequal elements at the same position in the two, it means that the model block at that position needs to be replaced. The model block to be replaced is loaded into the memory, and then it is used to replace the block that needs to be replaced in the model to generate a new model, thus completing the scaling of the model.

[0049] Furthermore, the step 4.2 of evaluating the accuracy loss of the derived model block includes:

[0050] Step 4.21, randomly select n model blocks to form a complete model, repeat this step k times, and generate k different models M1, M2, ..., M k ;

[0051] Step 4.22, derive the model block b i，j Replace k models and get k new models

[0052] Step 4.23, derive the model block b i，j The corresponding original block b i Replace the k models in step 4.22 to get k new models

[0053] Step 4.24, evaluate derived block b i，j The precision loss is expressed as A i，j (1≤i≤n, 0≤j≤ni:

[0054]

[0055] Among them, acc(M) represents the accuracy of model M.

[0056] The superior effects of the present invention are:

[0057] 1. The model generation and real-time construction method for unsupervised learning models described in the present invention splits the original training process based on the complete model into a training process based on model blocks, which is decoupled from the complex and unstable original unsupervised training process, thereby ensuring the convergence of model block training, reducing the accuracy loss of the derived model, and solving the instability problem of the existing model generation method applied to the unsupervised learning model. At the same time, the training based on model blocks does not interfere with each other and is decoupled from each other, supporting parallel training, further reducing the offline training cost, and the decoupled model blocks can generate a variety of available models through different combinations, thereby improving the model generation efficiency;

[0058] 2. The model generation and real-time construction method for unsupervised learning models described in the present invention can accurately calculate the current optimal model block combination scheme in a short time according to the current system available resources, user needs, deployment environment and other factors, and perform fine-grained model scaling, so that the model can use the available resources as efficiently as possible to improve the model accuracy. At the same time, fine-grained model scaling avoids reading in and out the entire model, thereby reducing the energy consumption of model scaling. The combination and switching operations of model blocks are natively supported by the existing deep learning framework, without the need for additional operations, and the switching overhead is low;

[0059] 3. The model generation and real-time construction method for unsupervised learning models described in the present invention is compatible with the reasoning process of the original unsupervised learning model. The reasoning logic of some unsupervised learning models has a certain complexity. The existing model generation method modifies the reasoning logic and introduces additional operations, resulting in increased complexity and difficulty in deployment. The method is free from modifying the original reasoning logic, introducing additional reasoning operations, and relying on special hardware. The model generated by the method is consistent with the use and deployment method of the original unsupervised model, further improving the usability and applicability of the method and solving the deployment problem of the existing model generation method applied to the unsupervised learning model;

[0060] 4. The model generation and real-time construction method for unsupervised learning models described in the present invention utilizes the phenomenon of "combinatorial explosion" to combine a large number of different models through a small number of model blocks. The time cost required for extracting the original model blocks and generating the derived model blocks is negligible. The training logic of the derived model blocks is unified but does not interfere with each other. The training of all derived blocks can be executed in parallel, which further improves the generation efficiency of the model blocks. The combination of the two improves the generation efficiency of the derived models.

[0061] 5. The model generation and real-time construction method for unsupervised learning models described in the present invention only needs to switch part of the model blocks when scaling the model, which greatly reduces the energy consumption of model scaling and improves the applicability of the model on energy-constrained edge devices. At the same time, the method described in the present invention defines the edge reasoning scenario as a classic linear programming problem. In different systems and different available resources, the optimization algorithm can quickly and accurately find the best model in a large number of available derivative models, greatly improving resource utilization;

[0062] 6. The model generation and real-time construction method for unsupervised learning models described in the present invention is applicable to deep neural networks with common structures and heterogeneous edge devices, and has low adaptation cost. When adapted to common unsupervised learning methods, it only needs to re-implement the model manager and model block manager, which improves the development and deployment efficiency and enhances the applicability of unsupervised learning models in edge reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a schematic diagram of a model generation and real-time construction method for an unsupervised learning model described in an embodiment of the present invention;

[0064] Figure 2 is a flowchart of step 1 of the method described in an embodiment of the present invention;

[0065] Figure 3 is a flowchart of step 2 of the method described in an embodiment of the present invention;

[0066] Figure 4is a flowchart of step 3 of the method described in an embodiment of the present invention;

[0067] Figure 5 is a flowchart of step 4 of the method described in an embodiment of the present invention;

[0068] Figure 6 is a flowchart of step 5 of the method described in an embodiment of the present invention;

[0069] Figure 7 is a flowchart of step 6 of the method described in an embodiment of the present invention;

[0070] Figure 8 is a flow chart of step 7 of the method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the specific implementation methods of the specification. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0072] Example

[0073] like Figure 1 As shown in the figure, the model generation and real-time construction method for unsupervised learning models includes two stages: offline training and online reasoning. In the offline training stage, block-granularity-based model training and compression are performed, and the original unsupervised learning model is divided into multiple blocks. Each block generates multiple sub-blocks with different compression degrees. The original model sub-block is used to train each derived sub-block, so that the training of each block is decoupled from the original training process, thereby avoiding the influence of unstable training of unsupervised learning, and hundreds or thousands of models can be derived based on the trained sub-blocks; in the online reasoning stage, based on the accuracy of each training sub-block, the online reasoning is used to train each derived sub-block. The linear programming algorithm is used to quickly solve the sub-block composition with the highest accuracy, generate the optimal model, and the switching granularity is the sub-block. Compared with switching the entire model, switching the sub-block will bring extremely small switching overhead. For unsupervised learning methods, it can effectively decouple the compression model from the original training process, effectively stabilize the compression model training process, and train hundreds of derivable models within a limited offline training time. The optimal model can be quickly solved based on linear programming, and the model is switched based on the block granularity. While providing a high-precision available model, the online switching cost is greatly reduced. Specifically, the method includes:

[0074] Step 1: For unsupervised learning scenarios, define the attributes of the model and the attributes of the model block and the operation methods of the model and the model block to give the unsupervised model the ability to analyze and operate, and then obtain the original model that has been trained unsupervised. For example, the attributes of the model may include the model name, the attributes of the model block may include the model block identifier, the basis for dividing the model block, the operation method of the model is the reasoning logic of the model in the unsupervised scenario, and the operation method of the model block is the method of switching the model block, such as Figure 2 As shown:

[0075] Step 1.1, inherit the model manager abstract class provided in the source code, and implement the missing methods in the abstract class, such as the model's reasoning logic;

[0076] Step 1.2, inherit the model block manager abstract class provided in the source code, and implement the missing methods in the abstract class, such as the block division principle and the compression strength of the derived block;

[0077] Step 2, generate multiple model blocks from the original model, such as Figure 3 As shown:

[0078] Step 2.1: According to the partitioning principle defined in the model block manager, n original model blocks b1, b2, ..., b n ;

[0079] Step 2.2: According to the derived block compression strength and derived block compression method defined in the model block manager, for each original model block b i (1≤i≤n), respectively, with compression strength Compress it to generate n i Derived Model Nuggets Final Generation A derived model block;

[0080] Step 2.3, building a training environment for each derived model block, that is, arranging the source of training data for each derived block;

[0081] Step 3: Train the derived model block and update the parameters of the derived model block, such as Figure 4 As shown:

[0082] Step 3.1: Input the training data used to train the original model into the original model, so that the original model completes one inference, and each original model block b in the original model i The input data and output data of (1≤i≤n) are transmitted to the corresponding derived block training environment;

[0083] Step 3.2: The original model block b iThe input data (1≤i≤n) is transmitted to each derived model block b i，j (1≤j≤n i ) enables the derived model block to perform feedforward calculation and obtain the derived block b i，j Output data of

[0084] Step 3.3, calculate the original model block b i (1≤i≤n) and each of its derived blocks b i，j (1≤j≤ni) The error value between the output data. If the error value is less than the set threshold, it means that the derived model block has been fully trained and its training is terminated. Otherwise, continue with step 3.4. If all derived blocks have stopped training, step 3 is terminated.

[0085] Step 3.4, using the back propagation algorithm and the gradient descent method, back propagate the error value and then perform gradient descent to update the parameters of the derived model block so that it develops in the direction of smaller and smaller error values, thereby making the derived model block more and more similar in function to the original model block;

[0086] Step 4: Evaluate all model blocks to quantify the performance indicators and usage value of the model blocks. The evaluation of all model blocks includes evaluating the accuracy loss of derived model blocks, such as Figure 5 As shown:

[0087] Step 4.1: Evaluate the volume s of the original model and the volume reduction S of the model block i，j (1≤i≤n, 0≤j≤ni, the volume reduction of the original model block is 0, that is, Si,0=0; then the volume of the derived model block bi,j is compared with that of its original block b i The volume difference is taken as the volume reduction S of the derived model block. i，j ;

[0088] Step 4.2, evaluate the accuracy loss of the model block;

[0089] Step 4.3, evaluate the delay reduction T of the model block i，j (1≤i≤n, 0≤j≤n i ):

[0090] Step 4.31, transfer the model block to the edge device. Since the inference delay of the model and the model block is related to the device, the delay evaluation needs to be performed on the target edge device;

[0091] Step 4.32, evaluate the inference delay t of the original model and the original model block b i The inference delay t of (1≤i≤n) i ;

[0092] Step 4.33, evaluate the derived model nugget bi，j The inference delay t i，j It should be noted that the inference delay is not only related to the device, but also to the real-time available resources. The delay reduction brought by the derived block is t i -t i，j It fluctuates with the environment and is not suitable as an evaluation metric. However, the percentage of latency reduction However, it can remain relatively stable in various environments. This indicator is used to evaluate the delay reduction of derived blocks, that is,

[0093] Step 5: Model the edge reasoning scenario and construct an optimization problem, such as Figure 6 As shown:

[0094] Step 5.1, clarify the unknown quantity of the optimization problem, that is, the choice of model blocks, for a total of model blocks, using B i，j ∈{0, 1}(1≤i≤n, 0≤j≤n i ) to indicate whether to select model block b i，j , when it is 1, it means selection, and when it is 0, it means no selection. At the same time, only one model block can be selected for a position in the model.

[0095] Step 5.2, clarify the goals and constraints of the optimization problem. In edge reasoning, the available memory constraint from the system is inevitable, but the model accuracy and reasoning latency requirements will be traded off according to the user or application scenario. For example, accuracy is used as the optimization goal and latency is used as a constraint, or latency is used as the optimization goal and accuracy is used as a constraint. In this step, the former (accuracy is used as the optimization goal and latency is used as a constraint) is taken as an example. The goal of the optimization problem is model accuracy, that is, the smaller the accuracy loss caused by the derived model block, the better:

[0096]

[0097] The constraints of the optimization problem are derived from the available system memory s max The constraints and the maximum allowable delay constraints from users and application scenarios t max :

[0098]

[0099]

[0100] Where s is the volume of the original model; S i，j is the volume reduction of the derived model block; t is the inference delay of the original model; t i is the original model block b i (1≤i≤n) inference delay; Ti，j For the derived model block b i，j Relative to the original model block b i The percentage of inference latency reduction is: the volume s of the original model minus the volume reduction brought by the derived block That is, get the volume of the current model. The volume of the current model must be smaller than the available memory of the current operating system. max , avoid the scenario of insufficient available memory to ensure the normal operation of the model. In the above formula: the inference delay t of the original model, minus the reduction in inference delay brought by the derived block That is, obtain the inference delay of the current model. The inference delay of the current model must be less than the maximum allowed delay t max , to ensure a good user experience;

[0101] Step 6: Solve the optimization problem constructed in the previous step. Since the unknown quantity B i，j It is either 0 or 1, and the constraints are all linear. Therefore, this problem belongs to the classic 01 integer programming problem and can be solved using an open source optimization problem solver. An optional solution idea is as follows: Figure 7 As shown:

[0102] Step 6.1, the optimization problem is an integer linear programming problem, which cannot be solved in polynomial time, so the integer constraint is removed (constraint B i，j ∈{0,1} is relaxed to 0≤B i，j ≤1), relax the original integer linear programming problem into a linear programming problem;

[0103] Step 6.2, the linear programming problem can be solved in polynomial time to obtain a relaxed solution to the original problem;

[0104] Step 6.3, use the branch and bound method to iteratively search for the optimal integer solution based on the relaxed solution. During the iteration process, if the gap between the current optimal solution and the relaxed solution is less than the predetermined threshold, the current optimal solution is considered to be good enough and the solution is exited early; otherwise, continue this step. The branch and bound method is a continuous iterative process. The current integer solution obtained in this process will gradually approach the optimal integer solution, but the time of this process cannot be estimated. In order to ensure the operating efficiency, during the iteration process, if the gap between the current integer solution and the optimal integer solution is less than the predetermined threshold, the current integer solution is considered to be good enough and the iteration is exited early to obtain B. i，j Otherwise, continue to iterate;

[0105] Step 7, according to B i，j Assemble or switch model blocks to complete the scaling of the model, such as Figure 8 As shown:

[0106] Step 7.1, if you get B for the first timei，j , then all B will be loaded directly i，j = 1 model block b i，j And assemble them into a complete model, and step 7 ends; otherwise, go to step 7.2;

[0107] Step 7.2, B i，j Combined solution with the model block obtained in the previous solution For comparison:

[0108]

[0109] B * The meaning is to select the model block b 1，2 , b 2，1 , b 3，4 , b 4，3 Load into the model framework (i.e. the block composition of the current model), B means to select model block b 1，2 , b 2，4 , b 3，4 , b 4，5 Load into the model framework (i.e., the block composition of the target model). Obviously, the system only needs to reload the model block b 2，4 , b 4，5 , and then the previous model block b 2，1 , b 4，3 Just replace it, thus generating a new model and completing the scaling of the model.

[0110] In the above embodiments, model is the abbreviation of deep neural network; model block represents a collection of multiple consecutive network layers in the model, and multiple model blocks can be considered to constitute a model; derived model / derived block represents a model generated based on the original model, which has the same function as the original model, but has different size, computational complexity and accuracy (for example, model compression technology can be used to compress the original model into a derived model with lower accuracy but smaller size, and the same applies to derived blocks, that is, model blocks generated based on the original model blocks); edge device refers to a terminal device on the edge side of the network close to the data source, such as a smart phone, an Internet of Things device, etc.; edge reasoning means using a model to process data on an edge device; model scaling means that when using a model for reasoning on an edge device, if the available resources increase or decrease, the model should also be adjusted to a larger or smaller model through switching or reorganization.

[0111] In the above embodiment, the classic anomaly detection method GANomaly is used based on the generative adversarial network in the unsupervised learning technology. By learning normal samples, it has the ability to identify abnormal samples. The GANomaly model has a simple structure and contains a total of 12 convolutional layers or deconvolution layers. It is small in size and has low computational complexity. It is suitable for edge devices with low performance. In the specific implementation, the Raspberry Pi 4B embedded device with a CPU of ARM Cortex-A72@1.5GHz and 2GB of available memory is used as the target edge device, and the GANomaly model is deployed therein, representing an anomaly detection application running in an embedded device.

[0112] In step 2 of the above embodiment, the GANomaly model is divided into model blocks under the guidance of the model block manager. Starting from the top layer of the network, every two convolutional layers or deconvolutional layers are used as a block, and a total of 6 original model blocks are divided. A compression ratio of [0.125, 0.25, 0.375, 0.5, 0.625] is set for each original model block, and compression is performed using the classic filter pruning technology, that is, 5 derived model blocks can be generated on the basis of each original block, and a total of 6×(5+1)=36 model blocks can be generated. When running online, 6^(5+1)=46656 available models can be combined. After the derived block is generated, a training environment is established for it, that is, a data transmission channel is established between the original model block in the original GANomaly model and its derived model block, for example, the input data of the first original model block is transmitted to its derived block as a channel for input data, and the output data of the first original block is transmitted to its derived block as a channel for error value calculation.

[0113] In step 3 of the above embodiment, the anomaly detection dataset Coil-100 used for the original GANomaly model training is input into the original model in batches for reasoning, and the data generated during the reasoning process is transmitted along the transmission channel to the corresponding derived block for feedforward calculation and error value calculation, and then the back propagation algorithm and the gradient descent method are used to adjust the parameters of the derived block so that the derived block develops in the direction of smaller and smaller error values, that is, the function of the derived block is closer and closer to its original block, until the training reaches the maximum number of times, or the error value is less than the set threshold. In the specific implementation, the maximum number of training times is set to 20, and the error value threshold is set to one thousandth of the current error value, that is, if the error value of the two previous and subsequent trainings decreases by less than one thousandth, it is considered that the training of the derived block has converged.

[0114] In step 4 of the above embodiment, in step 4.1, the volume index of 36 model blocks is evaluated; in step 4.2, k=3 is set, that is, 3 models are used to test the average accuracy gap between a derived block and its original block, which is used as the accuracy index of the derived block; in step 4.3, the generated 36 model blocks and the existing evaluation results are transmitted to the edge device Raspberry Pi 4B, and then the inference delay of the original model, the original model block and the derived model block is evaluated, and the delay reduction percentage of the derived model block relative to the original model block is calculated as its delay evaluation index.

[0115] In step 4.2 of the above embodiment, the model block cannot infer the input data alone, and the model block itself has no concept of accuracy. A method for evaluating the accuracy of the model block based on comparison is proposed to evaluate the accuracy of the derived block b. i，j Precision loss A i，j (1≤i≤n, 0≤j≤n i ) are as follows:

[0116] Step 4.21, randomly select n model blocks to form a complete model, repeat this step k times, and generate k different models M1, M2, ..., M k ;

[0117] Step 4.22, derive block b i，j Replace these k models and get k new models

[0118] Step 4.23, derive block b i，j The corresponding original block b i Replace the k models in step 4.22 to get k new models

[0119] Step 4.24, Where acc(M) represents the accuracy of model M.

[0120] In step 5 of the above embodiment, the GANomaly anomaly detection application based on the method is first started on the Raspberry Pi 4B, and the available memory status of the system is continuously monitored, wherein the detection accuracy of the anomaly detection application is set as the optimization target, that is, the higher the accuracy, the better, and the detection delay is used as a constraint, that is, the detection delay does not exceed a certain threshold (set to 100 milliseconds). Once the available memory is monitored to change, the method will reconstruct the optimization problem according to steps 5.1 and 5.2, and solve the optimization problem according to step 6 to obtain the current optimal model block combination solution.

[0121] When the available memory of Raspberry Pi 4B is 12MB, the optimal block combination solution obtained by the method is B1, that is, the optimal block is b 1，5 , b2，5 , b 3，0 , b 4，0 , b 5，2 , b 6，0 , and the accuracy of the corresponding model is 0.9887; when the available memory is reduced from 12MB to 10MB, the optimal block combination solution solved by the method is B2:

[0122]

[0123] That is, the optimal block is b 1，5 , b 2，5 , b 3，0 , b 4，4 , b 5，2 , b 6，3 , and the corresponding model accuracy is 0.9830. By comparing the two, it is easy to find that when the available memory is reduced from 12MB to 10MB, the method only needs to load the model block b 4，4 , b 6，3 , and then the model block b in the current model 4，0 , b 6，0 The total size of the read-in and read-out is only 6.6MB. The filter pruning method needs to switch the entire model in this scenario, which brings 13MB of switching load. At the same time, the model accuracy is reduced from 0.9805 to 0.9714. The method is applied to the GANomaly anomaly detection application on the Raspberry Pi 4B, which can reduce 49.2% of the online scaling energy consumption compared to the filter pruning technology, and avoid more accuracy loss.

[0124] Through testing on a variety of unsupervised learning models and edge devices, the method performs well in unsupervised learning scenarios. In scenarios where optimization accuracy, available memory, and latency are constraints, the method can bring an average 12.26% accuracy improvement compared to model compression and nested model technology (14.96% accuracy improvement when memory is scarce), and an average 17.84% accuracy improvement compared to the computational scaling model. The gap between the inference delay of the assembled optimal model and the delay constraint is reduced by 28.81%, which means that the method can better utilize available resources, thereby further improving model accuracy. In terms of the efficiency of derived model generation, the method of the present invention can bring thousands of times the number of derived models compared to model compression and computational scaling model technology within a similar training time; in terms of online model scaling, the method can bring an average of 68.16% and 83.03% energy savings on CPU edge devices and GPU edge devices.

[0125] The present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which are within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A model generation and real-time construction method for an unsupervised learning model, characterized in that: include: Step 1: For unsupervised learning scenarios, define the attributes of the model and the model blocks, as well as the operation methods of the model and the model blocks, to give the unsupervised model the ability to perform analysis and operation to obtain the original model: Step 1.1, inherit the model manager abstract class provided in the source code, and implement the missing methods in the model manager abstract class; Step 1.2, inherit the model block manager abstract class provided in the source code, supplement and implement the missing methods in the model block manager abstract class, and obtain the original model; Step 2: Generate multiple model blocks from the original model, obtain derived model blocks based on the model blocks, and build a training environment for each derived model block: Step 2.1: According to the partitioning principle defined in the model block manager, n original model blocks b1, b2, ..., b n ; Step 2.2: According to the derived block compression strength and derived block compression method defined in the model block manager, for each original model block b i ,1≤i≤n, respectively, with compression strength Compress it to generate n i Derived Model Nuggets n original model blocks are generated A derived model block; Step 2.3, arrange the source of training data for each derived block; Step 3: Train the derived model block and update the parameters of the derived model block: Step 3.1: Input the training data used to train the original model into the original model, so that the original model completes one inference, and each original model block b in the original model i ,1≤i≤n input data and output data are transmitted to the corresponding derived model block training environment; Step 3.2: The original model block b i ,1≤i≤n input data is transmitted to each derived model block b i,j ,1≤j≤n i Make the derived model block perform feedforward calculation to obtain the derived block b i,j Output data of Step 3.3, calculate the original model block b i , 1≤i≤n and each of its derived model blocks b i,j ,1≤j≤n i If the error value is less than the set threshold, it means that the derived model block has been fully trained and the training is terminated. Otherwise, continue to step 3.

4. If all derived blocks have stopped training, step 3 is terminated. Step 3.4, using the back propagation algorithm and the gradient descent method, back propagate the error value and then perform gradient descent to update the parameters of the derived model block, so that the error value becomes smaller and smaller, thereby making the function of the derived model block closer and closer to the original model block; Step 4: Evaluate all model blocks to quantify the performance indicators and usage value of the model blocks. The evaluation of all model blocks includes evaluating the accuracy loss of derived model blocks: Step 4.1: Evaluate the volume s of the original model and the volume reduction S of the model block i,j ,1≤i≤n,0≤j≤n i , the volume reduction of the original model block is 0, that is, S i,0 = 0; then the derived model block b i,j The volume of its original block b i The volume difference is taken as the volume reduction S of the derived model block. i,j ; Step 4.2, evaluate the accuracy loss of the derived model block: Step 4.21, randomly select n model blocks to form a complete model, repeat this step k times, and generate k different models M1, M2, ..., M k ; Step 4.22, derive the model block b i,j Replace k models and get k new models Step 4.23, derive the model block b i,j The corresponding original block b i Replace the k models in step 4.22 to get k new models Step 4.24, evaluate derived block b i,j The precision loss is expressed as A i,j ,1≤i≤n,0≤j≤n i : Among them, acc(M) represents the accuracy of model M; Step 4.3, evaluate the delay reduction T of the model block i,j ,1≤i≤n,0≤j≤n i : Step 4.31, transfer the model block to the edge device. Since the inference delay of the model and the model block is related to the device, the delay evaluation needs to be performed on the target edge device; Step 4.32, evaluate the inference delay t of the original model and the original model block b i Inference delay t for 1≤i≤n i ; Step 4.33, evaluate the derived model nugget b i,j The inference delay t i,j It should be noted that the inference delay is not only related to the device, but also to the real-time available resources. The delay reduction brought by the derived block is t i -t i,j It fluctuates with the environment and is not suitable as an evaluation metric. However, the percentage of latency reduction However, it can remain relatively stable in various environments. This indicator is used to evaluate the delay reduction of derived blocks, that is, Step 5: Select the optimal model block in the edge inference scenario, with the model block selection scheme as the unknown quantity, the maximum model accuracy as the goal, and the model block operating environment and application scenario as the constraint factors. Step 5 includes: Step 5.1, select the model block, A derived model block, using B i,j ∈{0,1},1≤i≤n,1≤j≤n i To indicate whether to select model block b i,j , B i,j When it is 1, it means selection, and when it is 0, it means no selection. A model block is selected at a position in the model, that is, Step 5.2, the optimization objective is to minimize the accuracy loss caused by the derived model block. The optimization objective is expressed as the following formula (1): In the above formula (1), A i,j Represents the derived model block b i,j To ensure the normal operation of the application, the available operating memory from the operating system needs to be considered. max constraints, as well as the maximum allowable delay constraints t from users and application scenarios max , respectively as shown in the following formula (2) and formula (3): In the above formula (2) and formula (3), s is the volume of the original model; S i,j is the volume reduction of the derived model block; t is the inference delay of the original model; t i is the original model block b i ,1≤i≤n inference delay; T i,j For the derived model block b i,j Relative to the original model block b i Percentage decrease in inference latency; Step 6, find the optimal selection solution for the model block: Step 6.1, first remove the integer constraints of the unknowns in the optimization problem and relax the original integer linear programming problem into a linear programming problem; Step 6.2, solve in polynomial time to obtain the optimal real solution to the original problem; Step 6.3, use the branch and bound method to find the optimal integer solution based on the optimal real solution; Step 7: Scale the current model according to the model block selection scheme obtained to generate a new model: Step 7.1, according to B i,j Assemble or switch model blocks, if B i,j is obtained by solving the optimization problem for the first time, then all B i,j = 1 model block b i,j Assemble into a complete model and end step 7; otherwise go to step 7.2; Step 7.2, B i,j Combined solution with the model block obtained last time For comparison, if there are unequal elements at the same position in the two model block combinations, it means that the model block at that position needs to be replaced. The model block to be replaced is loaded into the memory, and then the model block loaded into the memory is used to replace the block that needs to be replaced in the model to generate a new model, thus completing the scaling of the model.

Images