A multi-model merging and compression method
By independently training and compressing multiple deep learning models and sharing the same underlying network, the problem of redundant structure of multiple models is solved, achieving more efficient operation and saving computing resources.
Patent Information
- Application Number
- CN201911253771.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-09
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-12-09
AI Technical Summary
The prior art is difficult to effectively optimize the overall redundant structure of multiple deep learning models, resulting in increased storage and computing requirements, limiting product deployment and cost.
By independently training and compression of each model, separate the underlying and high-level layers, connect and train the mapping layer using the mapping layer, freeze the underlying and high-level layers, train only the mapping layer to share the same underlying network, and select the optimal result for deployment.
It reduces the overall redundant structure of multiple models, improves operating efficiency, saves computing resources, and improves product operating performance.
Smart Images

Figure CN113033763B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a model compression method, and in particular to a multi-model merging and compression method in the field of deep learning. Background Art
[0002] Model compression is an important step in the deployment and productization of deep learning algorithms. Usually, deep learning models with large parameters (large models) are used when training algorithms. Large models are used because they have strong learning capabilities and are easy to train.
[0003] However, during the algorithm deployment and productization phase, the large number of parameters in the large model becomes redundant, and further use will cause many problems:
[0004] (1) The large model has a large number of parameters, which requires a lot of storage and calculation, reducing the efficiency of the algorithm and increasing the power consumption of the product;
[0005] (2) Large models require large amounts of computation and place high demands on the computing power of chips, which limits the choice of chips for deployment and may increase product costs.
[0006] (3) Each model is often just a functional module, and the final product often contains many models. If the scale of a single model is very large, it will inevitably limit the number of modules included in the product.
[0007] Therefore, when deploying an algorithm, the model is often compressed without sacrificing performance. Common compression methods include pruning, distillation, and quantization.
[0008] Traditional model compression mainly operates on a single model, reducing the redundancy of each model. However, for products that contain many algorithm models, there are still redundant structures in the multi-model as a whole that can be eliminated. Traditional model compression targets a single model and cannot optimize this redundant structure as a whole.
[0009] The application number is 201210358480.X. The Chinese invention discloses a multi-model compression method for sending longitude and latitude position data of general aircraft by Beidou, which includes step one, determining the accuracy of longitude and latitude and altitude data according to actual conditions and truncating the data; step two, predicting and encoding the truncated data in step one based on one or more hypothetical models; step three, performing Huffman encoding on the multiple groups of predictive encoding data obtained in step two; step four, generating a data header and a compressed data body of the compressed data for the multiple groups of Huffman encoding obtained in step three, and selecting the one with the smallest number of bytes as the final compression result; step five, decompressing the data. Summary of the invention
[0010] In order to solve the deficiencies of the prior art, an object of the present invention is to provide a multi-model merging and compression method that can reduce the redundant structure of multiple models in a product as a whole.
[0011] In order to achieve the above object, the present invention provides a multi-model merging and compression method, comprising the following steps:
[0012] Step S1: Train and compress each model independently;
[0013] Step S2: Separate the bottom layer and the top layer of each model;
[0014] Step S3: Connect the bottom layer and the upper layer through the mapping layer;
[0015] Step S4: training the mapping layer;
[0016] Step S5: After the training is completed, the trained new model is evaluated and the best result is selected.
[0017] Advantageously, step S6 is also included, that is, deploying the optimal result on the final product.
[0018] In step S1, each model is trained and compressed independently, and the results of training and compression are used as input for multi-model merging and compression.
[0019] Each model uses ResNet-18 as the backbone of the network.
[0020] In step S2, the bottom layer L and the high layer H of each model are respectively peeled off, and the bottom layer is the first five convolutional layers.
[0021] In the step S3, an additional convolutional layer is used as the mapping layer M of each model, and the bottom layer L, the mapping layer M and the high layer H are combined accordingly.
[0022] In step S4, the new network constructed in step S3 is trained on the original task of each model. During training, the bottom layer L and the high layer H are frozen, and only the mapping layer M is trained.
[0023] In step S4, when training the mapping layer M, the weights of the mapping layer are initialized multiple times for training so as to be used for evaluation and final selection in step S5.
[0024] In step S5, after the training of step S4 is completed, the bottom layer of each model is used as the bottom layer of the common bottom layer new network, and compared with the corresponding original network, the best result is selected, and the bottom layer of the model corresponding to the best result is used as the common bottom layer of the fusion model.
[0025] Preferably, in step S1, the number of models is 3, each model is trained and compressed independently, and the results of training and compression are used as input for multi-model merging and compression.
[0026] Among them, in the step S2, the bottom layer and high layer of the three models are separated respectively, the bottom layer is called La, Lb and Lc respectively, and the high layer is called Ha, Hb and Hc respectively, and the bottom layer is the first five convolutional layers.
[0027] Among them, in the step S3, additional convolutional layers are used as mapping layers of each model, which are called Ma, Mb and Mc respectively. The bottom layer L, the mapping layer M and the high layer H are combined respectively, and a common bottom layer La is selected. Ha is the corresponding high layer itself, so no mapping layer is required; Hb and Hc do not correspond to the bottom layer La, so mapping layers are added to connect them respectively to obtain two new networks La-Mb-Hb and La-Mc-Hc; similarly, when Lb or Lc is used as the common bottom layer, the corresponding new network is constructed accordingly.
[0028] In step S5, the performance of the mapping layer is evaluated, including the following steps:
[0029] Step S51, on the test data set of any one of the three tasks corresponding to the three models, use k trained mapping layers to perform forward reasoning and evaluation, where K>1. Taking the detection task as an example, the evaluation index is generally mIOU, and the mapping layer M* with the highest evaluation index score is naturally used as the mapping layer of this network;
[0030] Step S52, the same operation is performed to select the mapping layer with the best performance in the remaining network;
[0031] Step S53, the final deployment solution will be generated from three options, which are:
[0032] When La is the bottom layer, La-Ha, La-Mb*-Hb and La-Mc*-Hc;
[0033] When Lb is the bottom layer, Lb-Ma*-Ha, Lb-Hb and Lb-Mc*-Hc;
[0034] When Lc is the bottom layer, Lc-Ma*-Ha, Lc-Mb*-Hb and Lc-Hc;
[0035] Among the three options, the solution that is closest to the overall performance of the network before fusion is selected based on the actual reasoning performance, and then deployed on the final product.
[0036] The principle of the present invention is to allow multiple models to share the same underlying network. The advantage is that it reduces the redundant structure of multiple models in the product as a whole, improves the overall operating efficiency of multiple models, and can save a lot of underlying time-consuming calculations, thereby improving the operating efficiency of the product. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Attached Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION
[0038] like Figure 1 As shown, a multi-model merging and compression method provided by the present invention comprises the following steps:
[0039] Step S1: Train and compress each model independently;
[0040] Step S2: Separate the bottom layer and the top layer of each model;
[0041] Step S3: Connect the bottom layer and the upper layer through the mapping layer;
[0042] Step S4: training the mapping layer;
[0043] Step S5: After the training is completed, the trained new model is evaluated and the best result is selected.
[0044] In addition, step S6 is also included, that is, deploying the optimal result on the final product.
[0045] In step S1, each model is trained and compressed independently, and the results of training and compression are used as input for multi-model merging and compression. Model compression refers to a class of methods that reduce the size and amount of calculation of a model while retaining the accuracy of the original model as much as possible. Common methods include pruning, distillation, and quantization. Usually, model compression is aimed at compressing a single deep learning model.
[0046] Model pruning is a model compression method that introduces sparsity to the dense connections of deep neural networks and reduces the number of non-zero weights by directly setting "unimportant" weights to zero.
[0047] Model distillation is to transfer the "knowledge" of the generalization ability of a trained complex model to a network with a simpler structure, or to learn the "knowledge" of a complex model through a simple network.
[0048] Model quantization is to store floating-point numbers that were previously expressed in 32 bits or 64 bits in the form of 1 bit or 2 bits, which takes up less memory space.
[0049] As we all know, the bottom layer of the deep learning model is functionally used to extract low-dimensional information of the input data. Taking computer vision as an example, the bottom layer of the model can extract low-dimensional features of the image, such as geometric features such as lines and corners. Although the weights of the bottom layers of each network obtained by separate training will not be consistent, their functions are very similar, and the bottom layer network is usually more time-consuming due to its high resolution. Therefore, the principle of the present invention is to allow multiple models to share the same bottom layer network to improve the overall operating efficiency of multiple models.
[0050] In step S2, each model uses ResNet-18 as the backbone of the network. The deep residual network (Residual Network, abbreviated as ResNet) can be said to be the most groundbreaking work in the field of computer vision and deep learning in the past few years. ResNet makes it possible to train hundreds or even thousands of layers, and still show excellent performance in this case. Due to its powerful representation ability, in addition to image classification, many computer vision applications including object detection and face recognition have achieved performance improvements. The 18 in ResNet-18 specifies 18 layers with weights, including convolutional layers and fully connected layers, excluding pooling layers and BN layers.
[0051] In the step S2, the bottom layer L and the high layer H of each model are peeled off respectively, the bottom layer is the first five convolutional layers. For example, taking the number of models as 3, namely 3 models a, b, and c, the bottom layer and the high layer of the 3 models are peeled off respectively, the bottom layers are called La, Lb, and Lc, and the high layers are called Ha, Hb, and Hc, respectively.
[0052] In the step S3, an additional convolutional layer is used as the mapping layer M of each model, and the bottom layer L, the mapping layer M and the high layer H are respectively combined. In the case of three models, the mapping layers are called Ma, Mb and Mc, respectively, and a common bottom layer La is selected. Ha is the corresponding high layer, so no mapping layer is needed; Hb and Hc do not correspond to the bottom layer La, and then the mapping layers are added and connected respectively to obtain two new networks La-Mb-Hb and La-Mc-Hc; similarly, when Lb or Lc is used as the common bottom layer, the corresponding new network is constructed, and the following new networks are obtained respectively: Lb-Ma-Ha; Lb-Mc-Hc; Lc-Ma-Ha; Lc-Mb-Hb.
[0053] In step S4, the new network constructed in step S3 is trained on the original task of each model. During training, the bottom layer L and the high layer H are frozen, and only the mapping layer M is trained. In the case of 3 models, the following 6 new networks are trained respectively:
[0054] (1) La-Mb-Hb, freeze the bottom layer La and the upper layer Hb, and only train the mapping layer Mb;
[0055] (2) La-Mc-Hc, freeze the bottom layer La and the upper layer Hc, and only train the mapping layer Mc;
[0056] (3) Lb-Ma-Ha, freeze the bottom layer Lb and the top layer Ha, and only train the mapping layer Ma;
[0057] (4) Lb-Mc-Hc, freeze the bottom layer Lb and the high layer Hc, and only train the mapping layer Mc;
[0058] (5) Lc-Ma-Ha, freeze the bottom layer Lc and the top layer Ha, and only train the mapping layer Ma;
[0059] (6) Lc-Mb-Hb, freeze the bottom layer Lc and the high layer Hb, and only train the mapping layer Mb;
[0060] Among them, La refers to the bottom layer of model a, Ha refers to the high layer of model a, and Ma refers to the mapping layer of model a; Lb refers to the bottom layer of model b, Hb refers to the high layer of model b, and Mb refers to the mapping layer of model b; Lc refers to the bottom layer of model c, Hc refers to the high layer of model c, and Mc refers to the mapping layer of model c.
[0061] In addition, in step S4, when training the mapping layer M, the weights of the mapping layer are initialized multiple times for training so as to be used for evaluation and final selection in step S5.
[0062] In step S5, the performance of the mapping layer is evaluated, including the following steps:
[0063] Step S51, on the test data set of any one of the three tasks corresponding to the three models, for example, on the test data set of task b, k trained mapping layers are used for forward reasoning and evaluation, where K>1. Taking the detection task as an example, the evaluation index is generally mIOU, and the mapping layer M* with the highest evaluation index score is naturally used as the mapping layer of this network;
[0064] Step S52, the same operation is performed to select the mapping layer with the best performance in the remaining network;
[0065] Step S53, the final deployment solution will be generated from three options, which are:
[0066] When La is the bottom layer, La-Ha, La-Mb*-Hb and La-Mc*-Hc;
[0067] When Lb is the bottom layer, Lb-Ma*-Ha, Lb-Hb and Lb-Mc*-Hc;
[0068] When Lc is the bottom layer, Lc-Ma*-Ha, Lc-Mb*-Hb and Lc-Hc;
[0069] Among them, Ma* refers to the mapping layer of the highest-scoring a model, Mb* refers to the mapping layer of the highest-scoring b model, and Mc* refers to the mapping layer of the highest-scoring c model.
[0070] Among the three options, the solution that is closest to the overall performance of the network before fusion is selected based on the actual reasoning performance, and then deployed on the final product.
[0071] In step S5, after the training of step S4 is completed, the bottom layer of each model is used as the bottom layer of the common bottom layer new network, and compared with the corresponding original network, the best result is selected, and the bottom layer of the model corresponding to the best result is used as the common bottom layer of the fusion model.
[0072] In another embodiment, the number of models is 4, namely, 4 models a, b, c, and d. The multi-model compression method according to this embodiment is as follows:
[0073] (1) Each model is trained and compressed independently. The training and compression results are used as the input for multi-model merging and compression. Each model uses ResNet-18 as the backbone of the network. Common model compression methods include pruning, distillation, and quantization.
[0074] (2) Separate the bottom layer L and the top layer H of each model, where the bottom layer is the first five convolutional layers, the bottom layers are called La, Lb, Lc and Ld, and the top layers are called Ha, Hb, Hc and Hd;
[0075] (3) Use an additional convolutional layer as the mapping layer M of each model, and combine the bottom layer L, mapping layer M and high layer H. In the case of 4 models, the mapping layers are called Ma, Mb, Mc and Md respectively. A common bottom layer La is selected. Ha is the corresponding high layer, so no mapping layer is needed. Hb, Hc and Hd do not correspond to the bottom layer La, so they are connected with mapping layers to obtain three new networks: La-Mb-Hb, La-Mc-Hc and La-Md-Hd. Similarly, when Lb, Lc or Ld is used as the common bottom layer, the corresponding new networks are constructed, and the following new networks are obtained respectively:
[0076] 1) Lb-Ma-Ha, Lb-Mc-Hc, Lb-Md-Hd;
[0077] 2) Lc-Ma-Ha, Lc-Mb-Hb, Lc-Md-Hd;
[0078] 3) Ld-Ma-Ha, Ld-Mb-Hb, Ld-Mc-Hc;
[0079] (4) Train the new network constructed in step S3 on the original task of each model. During training, freeze the bottom layer L and the upper layer H, and only train the mapping layer M. In the case of 4 models, train the following 12 new networks respectively:
[0080] 1) La-Mb-Hb, freeze the bottom layer La and the upper layer Hb, and only train the mapping layer Mb;
[0081] 2) La-Mc-Hc, freeze the bottom layer La and the high layer Hc, and only train the mapping layer Mc;
[0082] 3) La-Md-Hd, freeze the bottom layer La and the high layer Hd, and only train the mapping layer Md;
[0083] 4) Lb-Ma-Ha, freeze the bottom layer Lb and the high layer Ha, and only train the mapping layer Ma;
[0084] 5) Lb-Mc-Hc, freeze the bottom layer Lb and the high layer Hc, and only train the mapping layer Mc;
[0085] 6) Lb-Md-Hd, freeze the bottom layer Lb and the high layer Hd, and only train the mapping layer Md;
[0086] 7) Lc-Ma-Ha, freeze the bottom layer Lc and the high layer Ha, and only train the mapping layer Ma;
[0087] 8) Lc-Mb-Hb, freeze the bottom layer Lc and the high layer Hb, and only train the mapping layer Mb;
[0088] 9) Lc-Md-Hd, freeze the bottom layer Lc and the high layer Hd, and only train the mapping layer Md;
[0089] 10) Ld-Ma-Ha, freeze the bottom layer Ld and the top layer Ha, and only train the mapping layer Ma;
[0090] 11) Ld-Mb-Hb, freeze the bottom layer Ld and the high layer Hb, and only train the mapping layer Mb;
[0091] 12) Ld-Mc-Hc, freeze the bottom layer Ld and the high layer Hc, and only train the mapping layer Mc;
[0092] Among them, La refers to the bottom layer of model a, Ha refers to the high layer of model a, and Ma refers to the mapping layer of model a; Lb refers to the bottom layer of model b, Hb refers to the high layer of model b, and Mb refers to the mapping layer of model b; Lc refers to the bottom layer of model c, Hc refers to the high layer of model c, and Mc refers to the mapping layer of model c; Ld refers to the bottom layer of model d, Hd refers to the high layer of model d, and Md refers to the mapping layer of model d.
[0093] When training the mapping layer M, the weights of the mapping layer are initialized multiple times for training so as to be used for the evaluation and final selection in step S5.
[0094] (5) In step S5, the performance of the mapping layer is evaluated, including the following steps:
[0095] Step S51, on the test data set of any one of the four tasks corresponding to the four models, for example, on the test data set of task b, use k trained mapping layers to perform forward reasoning and evaluation, where K>1. Taking the detection task as an example, the evaluation index is generally mIOU, and the mapping layer M* with the highest evaluation index score is naturally used as the mapping layer of this network;
[0096] Step S52, the same operation is performed to select the mapping layer with the best performance in the remaining network;
[0097] Step S53, the final deployment solution will be generated from four options, which are:
[0098] 1) When La is the bottom layer, La-Ha, La-Mb*-Hb, La-Mc*-Hc and La-Md*-Hd;
[0099] 2) When Lb is the bottom layer, Lb-Ma*-Ha, Lb-Hb, Lb-Mc*-Hc and Lb-Md*-Hd;
[0100] 3) When Lc is the bottom layer, Lc-Ma*-Ha, Lc-Mb*-Hb, Lc-Hc and Lc-Md*-Hd;
[0101] 4) When Ld is the bottom layer, Ld-Ma*-Ha, Ld-Mb*-Hb, Ld-Mc*-Hc and Ld-Hd;
[0102] Among them, Ma* refers to the mapping layer of the model a with the highest score, Mb* refers to the mapping layer of the model b with the highest score, Mc* refers to the mapping layer of the model c with the highest score, and Md* refers to the mapping layer of the model d with the highest score.
[0103] Among the four options, the solution that is closest to the overall performance of the network before fusion is selected based on the actual reasoning performance, and then deployed on the final product.
[0104] In another embodiment, the number of models is 5, namely, 5 models a, b, c, d, and e. The multi-model compression method according to this embodiment is as follows:
[0105] (1) Each model is trained and compressed independently. The training and compression results are used as the input for multi-model merging and compression. Each model uses ResNet-18 as the backbone of the network. Common model compression methods include pruning, distillation, and quantization.
[0106] (2) Separate the bottom layer L and the high layer H of each model, where the bottom layer is the first five convolutional layers, the bottom layers are called La, Lb, Lc, Ld and Le, and the high layers are called Ha, Hb, Hc, Hd and He;
[0107] (3) Use an additional convolutional layer as the mapping layer M of each model, and combine the bottom layer L, mapping layer M and high layer H. In the case of 5 models, the mapping layers are called Ma, Mb, Mc, Md and Me respectively. A common bottom layer La is selected. Ha is the corresponding high layer, so no mapping layer is needed. Hb, Hc, Hd and He do not correspond to the bottom layer La, so they are connected with mapping layers to obtain four new networks: La-Mb-Hb, La-Mc-Hc, La-Md-Hd and La-Me-He. Similarly, when Lb, Lc, Ld or Le is used as the common bottom layer, the corresponding new networks are constructed, and the following new networks are obtained respectively:
[0108] 1) Lb-Ma-Ha, Lb-Mc-Hc, Lb-Md-Hd, Lb-Me-He;
[0109] 2) Lc-Ma-Ha, Lc-Mb-Hb, Lc-Md-Hd, Lc-Me-He;
[0110] 3) Ld-Ma-Ha, Ld-Mb-Hb, Ld-Mc-Hc, Ld-Me-He;
[0111] 4) Le-Ma-Ha, Le-Mb-Hb, Le-Mc-Hc, Le-Md-Hd;
[0112] (4) Train the new network constructed in step S3 on the original task of each model. During training, freeze the bottom layer L and the upper layer H, and only train the mapping layer M. In the case of 5 models, train the following 20 new networks respectively:
[0113] 1) La-Mb-Hb, freeze the bottom layer La and the upper layer Hb, and only train the mapping layer Mb;
[0114] 2) La-Mc-Hc, freeze the bottom layer La and the high layer Hc, and only train the mapping layer Mc;
[0115] 3) La-Md-Hd, freeze the bottom layer La and the high layer Hd, and only train the mapping layer Md;
[0116] 4) La-Me-He, freeze the bottom layer La and the top layer He, and only train the mapping layer Me;
[0117] 5) Lb-Ma-Ha, freeze the bottom layer Lb and the high layer Ha, and only train the mapping layer Ma;
[0118] 6) Lb-Mc-Hc, freeze the bottom layer Lb and the high layer Hc, and only train the mapping layer Mc;
[0119] 7) Lb-Md-Hd, freeze the bottom layer Lb and the high layer Hd, and only train the mapping layer Md;
[0120] 8) Lb-Me-He, freeze the bottom layer Lb and the high layer He, and only train the mapping layer Me;
[0121] 9) Lc-Ma-Ha, freeze the bottom layer Lc and the top layer Ha, and only train the mapping layer Ma;
[0122] 10) Lc-Mb-Hb, freeze the bottom layer Lc and the high layer Hb, and only train the mapping layer Mb;
[0123] 11) Lc-Md-Hd, freeze the bottom layer Lc and the high layer Hd, and only train the mapping layer Md;
[0124] 12) Lc-Me-He, freeze the bottom layer Lc and the high layer He, and only train the mapping layer Me;
[0125] 13) Ld-Ma-Ha, freeze the bottom layer Ld and the top layer Ha, and only train the mapping layer Ma;
[0126] 14) Ld-Mb-Hb, freeze the bottom layer Ld and the high layer Hb, and only train the mapping layer Mb;
[0127] 15) Ld-Mc-Hc, freeze the bottom layer Ld and the high layer Hc, and only train the mapping layer Mc;
[0128] 16) Ld-Me-He, freeze the bottom layer Ld and the top layer He, and only train the mapping layer Me;
[0129] 17) Le-Ma-Ha, freeze the bottom layer Le and the top layer Ha, and only train the mapping layer Ma;
[0130] 18) Le-Mb-Hb, freeze the bottom layer Le and the upper layer Hb, and only train the mapping layer Mb;
[0131] 19) Le-Mc-Hc, freeze the bottom layer Le and the high layer Hc, and only train the mapping layer Mc;
[0132] 20) Le-Md-Hd, freeze the bottom layer Le and the high layer Hd, and only train the mapping layer Md;
[0133] Among them, La refers to the bottom layer of model a, Ha refers to the high layer of model a, and Ma refers to the mapping layer of model a; Lb refers to the bottom layer of model b, Hb refers to the high layer of model b, and Mb refers to the mapping layer of model b; Lc refers to the bottom layer of model c, Hc refers to the high layer of model c, and Mc refers to the mapping layer of model c; Ld refers to the bottom layer of model d, Hd refers to the high layer of model d, and Md refers to the mapping layer of model d; Le refers to the bottom layer of model e, He refers to the high layer of model e, and Me refers to the mapping layer of model e.
[0134] When training the mapping layer M, the weights of the mapping layer are initialized multiple times for training so as to be used for the evaluation and final selection in step S5.
[0135] (5) In step S5, the performance of the mapping layer is evaluated, including the following steps:
[0136] Step S51, on the test data set of any one of the five tasks corresponding to the five models, for example, on the test data set of task b, use k trained mapping layers to perform forward reasoning and evaluation, where K>1. Taking the detection task as an example, the evaluation index is generally mIOU, and the mapping layer M* with the highest evaluation index score is naturally used as the mapping layer of this network;
[0137] Step S52, the same operation is performed to select the mapping layer with the best performance in the remaining network;
[0138] Step S53, the final deployment solution will be generated from five options, which are:
[0139] 1) When La is the bottom layer, La-Ha, La-Mb*-Hb, La-Mc*-Hc, La-Md*-Hd and La-Me*-He;
[0140] 2) When Lb is the bottom layer, Lb-Ma*-Ha, Lb-Hb, Lb-Mc*-Hc, Lb-Md*-Hd and Lb-Me*-He;
[0141] 3) When Lc is the bottom layer, Lc-Ma*-Ha, Lc-Mb*-Hb, Lc-Hc, Lc-Md*-Hd and Lc-Me*-He;
[0142] 4) When Ld is the bottom layer, Ld-Ma*-Ha, Ld-Mb*-Hb, Ld-Mc*-Hc, Ld-Hd and Ld-Me*-He;
[0143] 5) When Le is the bottom layer, Le-Ma*-Ha, Le-Mb*-Hb, Le-Mc*-Hc, Le-Md*-Hd and Le-He;
[0144] Among them, Ma* refers to the mapping layer of the model a with the highest score, Mb* refers to the mapping layer of the model b with the highest score, Mc* refers to the mapping layer of the model c with the highest score, Md* refers to the mapping layer of the model d with the highest score, and Me* refers to the mapping layer of the model e with the highest score.
[0145] Among the five options, the solution that is closest to the overall performance of the network before fusion is selected based on the actual reasoning performance, and then deployed on the final product.
[0146] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the design concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the design concept of the present invention on the basis of the prior art should be within the scope of the present invention and / or the protection scope determined by the claims.
Claims
1. A multi-model merging and compression method, comprising the following steps: Step S1: Train and compress each model independently; Train and compress each model independently, and use the training and compression results as input for multi-model merging and compression; Step S2: Separate the bottom layer and the high layer of each model; each model uses ResNet-18 as the backbone of the network; in the step S2, the bottom layer L and the high layer H of each model are respectively peeled off, and the bottom layer is the first five convolutional layers; Step S3: Connect the bottom layer and the upper layer through the mapping layer; In the step S3, an additional convolutional layer is used as the mapping layer M of each model, and the bottom layer L, the mapping layer M and the high layer H are respectively combined accordingly; Step S4: training the mapping layer; In step S4, the new network constructed in step S3 is trained on the original task of each model. During training, the bottom layer L and the upper layer H are frozen, and only the mapping layer M is trained; In the step S4, when training the mapping layer M, the weights of the mapping layer are initialized multiple times for training so as to be used for evaluation and final selection in step S5; Step S5: After the training is completed, the trained new model is evaluated and the best result is selected; In step S5, after the training in step S4 is completed, the bottom layer of each model is used as the bottom layer of the common bottom layer new network, and compared with the corresponding original network, the best result is selected, and the bottom layer of the model corresponding to the best result is used as the common bottom layer of the fusion model; Step S6: Deploy the optimal result on the final product.
2. A multi-model merging and compression method according to claim 1, in, In step S1, the number of models is 3, and each model is trained and compressed independently, and the results of training and compression are used as input for multi-model merging and compression.
3. A multi-model merging and compression method according to claim 2, in, In step S2, the bottom layers and high layers of the three models are separated respectively, the bottom layers are called La, Lb and Lc respectively, and the high layers are called Ha, Hb and Hc respectively, and the bottom layers are the first five convolutional layers.
4. A multi-model merging and compression method according to claim 3, in, In step S3, additional convolutional layers are used as mapping layers of each model, which are called Ma, Mb and Mc respectively. The bottom layer L, the mapping layer M and the high layer H are respectively combined, and a common bottom layer La is selected. Ha is the high layer corresponding to itself, so no mapping layer is required; Hb and Hc do not correspond to the bottom layer La, so they are connected with mapping layers respectively to obtain two new networks La-Mb-Hb and La-Mc-Hc; Similarly, when Lb or Lc is used as the common bottom layer, a corresponding new network is constructed accordingly.
5. According to the multi-model merging compression method of claim 4, in step S5, evaluating the performance of the mapping layer comprises the following steps: Step S51, on the test data set of any one of the three tasks corresponding to the three models, use k trained mapping layers to perform forward reasoning and evaluation respectively, in, If K>1, taking the detection task as an example, the evaluation index is mIOU, and the mapping layer M* with the highest evaluation index score is naturally used as the mapping layer of this network; Step S52, the same operation is performed to select the mapping layer with the best performance in the remaining network; Step S53, the final deployment solution will be generated from three options, which are: When La is the bottom layer, La-Ha, La-Mb*-Hb and La-Mc*-Hc; When Lb is the bottom layer, Lb-Ma*-Ha, Lb-Hb and Lb-Mc*-Hc; When Lc is the bottom layer, Lc-Ma*-Ha, Lc-Mb*-Hb and Lc-Hc; Among the three options, the solution that is closest to the overall performance of the network before fusion is selected based on the actual reasoning performance, and then deployed on the final product.
