Terminal model processing method, device and equipment

Abstract

The present application provides a terminal model processing method, device and electronic device, which are applicable to the field of model miniaturization technology in the field of artificial intelligence. The method includes: dividing the model layer in the terminal model to be processed into multiple original layer sets, and performing multiple lightweight processing on each original layer set to obtain a corresponding quantized layer set. Using preset sample data, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the model converges and the lightweight training of the model is completed. Then, based on the model obtained by lightweight training, the data to be processed is processed to obtain the corresponding processing results. Since the amount of data of the weight parameters of the selected layer set is smaller than the original model to be processed, and the amount of calculation when the model is running is also greatly reduced, the size of the model can be reduced and computing resources can be saved while ensuring the performance of the model.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to terminal model processing methods, devices and equipment. Background Art

[0002] With the continuous advancement of artificial intelligence (AI) technology, the scenarios for using AI to improve the intelligence of electronic devices are becoming more and more abundant. For example, by building AI applications containing terminal models such as neural network models (hereinafter referred to as models) in terminal devices, the functions of terminal devices can be greatly enriched, making terminal devices more intelligent.

[0003] In order to improve the performance of the model, more weight parameters are often trained to represent the model, which causes the storage space occupied by the model when it is stored and the computing resources occupied when it is running to increase rapidly, affecting the processing performance of the terminal device. Especially for terminal devices with limited computing resources such as mobile phones, it may even cause AI applications to be unable to be used normally. Summary of the invention

[0004] In view of this, embodiments of the present application provide a terminal model processing method, apparatus, and device, which can reduce the memory footprint and computing resources of the terminal model and improve the processing performance of the terminal device.

[0005] A first aspect of an embodiment of the present application provides a terminal model processing method, including:

[0006] The model layers in the terminal model to be processed are divided into multiple original layer sets, and various lightweight processing is performed on each original layer set to obtain a corresponding quantized layer set. Using preset sample data, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the model converges and the lightweight training of the model is completed. Then, based on the model obtained by lightweight training, the data to be processed is processed to obtain the corresponding processing results. Among them, the lightweight processing includes at least two of the following processing: quantizing the weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between the model layers in the layer set.

[0007] Since the amount of data of the selected layer set weight parameters is smaller than the original model to be processed and the amount of calculation during model operation is also greatly reduced, the storage space occupied by electronic devices and the computing resources occupied during model operation are reduced while ensuring the model performance.

[0008] In a first possible implementation manner of the first aspect, reducing the number of weight parameters in a layer set includes:

[0009] Reduce the size of the convolution kernels included in the layer set, or reduce the number of convolution kernels.

[0010] By reducing the size and number of convolution kernels, lightweight operations on layer sets can be better achieved.

[0011] In a second possible implementation manner of the first aspect, as an optional manner of lightweighting the first layer set in the first possible implementation manner, the method includes:

[0012] The network structure of the first layer set is optimized based on the depth-wise separable convolution technology.

[0013] By optimizing the network structure of the first layer set through the deep separable convolution technology, the network structure redundancy of the model layer in the first layer set can be reduced, so that the number of weight parameters and the amount of calculation in the first layer set can be reduced.

[0014] In a third possible implementation manner of the first aspect, before performing iterative training on the weight parameters of the second layer set, the method further includes:

[0015] A preset activation probability corresponding to each second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameter in the second layer set.

[0016] In each training process of the second layer set, including:

[0017] According to the corresponding preset activation probability, the second layer set to be activated is selected, and the weight parameters of the selected second layer set are updated.

[0018] By setting the activation probability, the training of each second-layer set is controlled to ensure that the second-layer set of low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0019] In a fourth possible implementation manner of the first aspect, before performing iterative training on weight parameters of the first layer set and the second layer set, the method further includes:

[0020] The preset activation probability corresponding to each first layer set and second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set.

[0021] In each training process of the first layer set and the second layer set, it includes:

[0022] According to the corresponding preset activation probability, a layer set to be activated is selected from the first layer set and the second layer set, and the weight parameters of the selected layer set are updated.

[0023] By setting the activation probability, the training of each first-layer set and second-layer set can be controlled to ensure that the layer set with low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0024] In a fifth possible implementation manner of the first aspect, the iterative training process of the weight parameters of each layer set includes:

[0025] If the terminal model to be processed does not meet the preset convergence condition after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, at least one performance indicator parameter of the terminal model to be processed after the training is completed is obtained.

[0026] Using preset sample data and at least one performance indicator parameter, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the terminal model to be processed meets the preset convergence conditions and the lightweight processing is completed.

[0027] In an embodiment of the present application, by iteratively updating the weight parameters of the layer set in a manner of feedback indicator parameters each time training, the training effect of the model can be improved, while the speed at which the model meets the convergence conditions can be accelerated, thereby improving the model training efficiency.

[0028] In a sixth possible implementation manner of the first aspect, the iterative training process of the weight parameters of each layer set includes:

[0029] A differentiable neural architecture search method based on gradient descent is used to iteratively train the weight parameters of the model.

[0030] Since differentiable neural structure search can realize the search for the best architecture in the search space, on the basis of setting the search nodes as the layer sets in each level, the model can be trained through the differentiable neural structure search training method, and the layer sets in each level can be selected during the model training process, so that when the model meets the convergence conditions, a model composed of the selected layer sets can be obtained.

[0031] In a seventh possible implementation manner of the first aspect, the iterative training process of the weight parameters of each layer set includes:

[0032] If the terminal model to be processed meets the preset convergence conditions after iterative training, each second-layer set is used as a path node, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed terminal model.

[0033] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0034] In an eighth possible implementation manner of the first aspect, the iterative training process of the weight parameters of each layer set includes:

[0035] If the terminal model to be processed meets the preset convergence conditions after iterative training, the path search of the layer set is performed on the terminal model to be processed with each first layer set and the second layer set as the path node, and the layer set path that meets the preset performance index requirements is obtained, and the model composed of each path node in the layer set path is used as the processed terminal model.

[0036] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0037] A second aspect of an embodiment of the present application provides a terminal model training method, including:

[0038] The model layers in the terminal model to be processed are divided into multiple original layer sets, and various lightweight processes are performed on each original layer set to obtain a corresponding quantized layer set. The weight parameters of the second layer set are iteratively trained using preset sample data, or the weight parameters of the first layer set and the second layer set are iteratively trained until the model converges to obtain a trained terminal model. Among them, the lightweight processing includes at least two of the following processing: quantizing the weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between the model layers in the layer set.

[0039] Since the amount of data of the selected layer set weight parameters is smaller than the original model to be processed and the amount of calculation during model operation is also greatly reduced, the storage space occupied by electronic devices and the computing resources occupied during model operation are reduced while ensuring the model performance.

[0040] In a first possible implementation manner of the second aspect, reducing the number of weight parameters in the layer set includes:

[0041] Reduce the size of the convolution kernels included in the layer set, or reduce the number of convolution kernels.

[0042] By reducing the size and number of convolution kernels, lightweight operations on layer sets can be better achieved.

[0043] In a second possible implementation manner of the second aspect, as an optional manner of lightweighting the first layer set in the first possible implementation manner, the method includes:

[0044] The network structure of the first layer set is optimized based on the depth-wise separable convolution technology.

[0045] By optimizing the network structure of the first layer set through the deep separable convolution technology, the network structure redundancy of the model layer in the first layer set can be reduced, so that the number of weight parameters and the amount of calculation in the first layer set can be reduced.

[0046] In a third possible implementation manner of the second aspect, before performing iterative training on the weight parameters of the second layer set, the method further includes:

[0047] A preset activation probability corresponding to each second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameter in the second layer set.

[0048] In each training process of the second layer set, including:

[0049] According to the corresponding preset activation probability, the second layer set to be activated is selected, and the weight parameters of the selected second layer set are updated.

[0050] By setting the activation probability, the training of each second-layer set is controlled to ensure that the second-layer set of low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0051] In a fourth possible implementation manner of the second aspect, before performing iterative training on weight parameters of the first layer set and the second layer set, the method further includes:

[0052] The preset activation probability corresponding to each first layer set and second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set.

[0053] In each training process of the first layer set and the second layer set, it includes:

[0054] According to the corresponding preset activation probability, a layer set to be activated is selected from the first layer set and the second layer set, and the weight parameters of the selected layer set are updated.

[0055] By setting the activation probability, the training of each first-layer set and second-layer set can be controlled to ensure that the layer set with low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0056] In a fifth possible implementation manner of the second aspect, the iterative training process of the weight parameters of each layer set includes:

[0057] If the terminal model to be processed does not meet the preset convergence condition after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, at least one performance indicator parameter of the terminal model to be processed after the training is completed is obtained.

[0058] Using preset sample data and at least one performance indicator parameter, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the terminal model to be processed meets the preset convergence conditions, thereby obtaining a trained terminal model.

[0059] In an embodiment of the present application, by iteratively updating the weight parameters of the layer set in a manner of feedback indicator parameters each time training, the training effect of the model can be improved, while the speed at which the model meets the convergence conditions can be accelerated, thereby improving the model training efficiency.

[0060] In a sixth possible implementation manner of the second aspect, the iterative training process of the weight parameters of each layer set includes:

[0061] A differentiable neural architecture search method based on gradient descent is used to iteratively train the weight parameters of the model.

[0062] Since differentiable neural structure search can realize the search for the best architecture in the search space, on the basis of setting the search nodes as the layer sets in each level, the model can be trained through the differentiable neural structure search training method, and the layer sets in each level can be selected during the model training process, so that when the model meets the convergence conditions, a model composed of the selected layer sets can be obtained.

[0063] In a seventh possible implementation manner of the second aspect, the iterative training process of the weight parameters of each layer set includes:

[0064] If the terminal model to be processed meets the preset convergence conditions after iterative training, each second-layer set is used as a path node, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed terminal model.

[0065] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0066] In an eighth possible implementation manner of the second aspect, the iterative training process of the weight parameters of each layer set includes:

[0067] If the terminal model to be processed meets the preset convergence conditions after iterative training, the path search of the layer set is performed on the terminal model to be processed with each first layer set and the second layer set as the path node, and the layer set path that meets the preset performance index requirements is obtained, and the model composed of each path node in the layer set path is used as the processed terminal model.

[0068] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0069] A third aspect of an embodiment of the present application provides a terminal model training method, including:

[0070] Construct an initial model including multiple layer groups to be trained; wherein each layer group to be trained includes multiple layer sets, each layer set includes one or more model layers, and the layer sets included in each layer group to be trained have at least one of the following differences: the number of bytes occupied by weight parameters in the layer set, the number of weight parameters in the layer set, and the network structure between model layers in the layer set;

[0071] The weight parameters of the initial model are initialized, and the weight parameters of each layer set are iteratively trained until the initial model meets the preset convergence conditions to obtain a trained terminal model.

[0072] Since the amount of data in the layer set selected after mixed precision training is small, the size of the final model and the computing resources occupied during operation are also small. The storage space occupied by electronic devices and the computing resources occupied during model operation are reduced, which improves the performance of electronic devices.

[0073] In a first possible implementation manner of the third aspect, reducing the number of weight parameters in a layer set includes:

[0074] Reduce the size of the convolution kernels included in the layer set, or reduce the number of convolution kernels.

[0075] By reducing the size and number of convolution kernels, lightweight operations on layer sets can be better achieved.

[0076] In a second possible implementation manner of the third aspect, before performing iterative training on the weight parameters of the second layer set, the method further includes:

[0077] Obtaining a preset activation probability corresponding to each of the layer sets, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameter in the layer set;

[0078] In the process of iteratively training the weight parameters of each layer set, each training process includes:

[0079] According to the corresponding preset activation probability, the layer set to be activated is selected, and the weight parameters of the selected layer set are updated.

[0080] By setting the activation probability, the training of each layer set can be controlled to ensure that the layer set with low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0081] In a fourth possible implementation manner of the third aspect, the iterative training process of the weight parameters of each layer set includes:

[0082] If the initial model does not meet the preset convergence condition after a single training of the layer set is completed, at least one performance indicator parameter of the initial model after the training is completed is obtained.

[0083] The weight parameters of the layer set are iteratively trained using preset sample data and at least one performance indicator parameter until the initial model meets the preset convergence condition, thereby obtaining a trained terminal model.

[0084] In an embodiment of the present application, by iteratively updating the weight parameters of the layer set in a manner of feedback indicator parameters each time training, the training effect of the model can be improved, while the speed at which the model meets the convergence conditions can be accelerated, thereby improving the model training efficiency.

[0085] In a fifth possible implementation manner of the third aspect, the iterative training process of the weight parameters of each layer set includes:

[0086] A differentiable neural architecture search method based on gradient descent is used to iteratively train the weight parameters of the model.

[0087] Since differentiable neural structure search can realize the search for the best architecture in the search space, on the basis of setting the search nodes as the layer sets in each level, the model can be trained through the differentiable neural structure search training method, and the layer sets in each level can be selected during the model training process, so that when the model meets the convergence conditions, a model composed of the selected layer sets can be obtained.

[0088] In a seventh possible implementation manner of the third aspect, the iterative training process of the weight parameters of each layer set includes:

[0089] If the initial model after iterative training meets the preset convergence conditions, each layer set is used as a path node, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed terminal model.

[0090] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0091] A fourth aspect of the embodiments of the present application provides an image classification model training method, comprising:

[0092] The hidden layer of the image classification model to be processed is divided into at least one first layer set, wherein each first layer set contains at least one first model layer, and the first model layer is a hidden layer containing a convolution operator, and the convolution operator is used for image feature analysis. Each first layer set is lightweight processed to obtain a corresponding second layer set. The weight parameters of the second layer set are iteratively trained using a preset sample image, or the weight parameters of the first layer set and the second layer set are iteratively trained until the image classification model to be processed meets the preset convergence conditions, thereby obtaining a trained image classification model. The lightweight processing includes at least two of the following processing: quantizing the weight parameters contained in the convolution operator in the layer set, reducing the number of weight parameters contained in the convolution operator in the layer set, and adjusting the network structure between the model layers in the layer set.

[0093] Since the amount of data of the weight parameters in the convolution operator in the selected layer set is smaller than that of the original image classification model to be trained, and the amount of computation during model operation is also greatly reduced, the storage space occupied by electronic devices and the computing resources occupied during model operation are reduced while ensuring the performance of the image classification model.

[0094] In a first possible implementation manner of the fourth aspect,

[0095] The number of weight parameters contained in the convolution operator in the reduction layer set includes:

[0096] Reduce the size of the convolution kernel contained in the convolution operator, or reduce the number of convolution kernels in the convolution operator.

[0097] By reducing the size and number of convolution kernels, lightweight operations on layer sets can be better achieved.

[0098] By lightweight processing the first layer set, the lightweight operation of the model can be better achieved.

[0099] In a second possible implementation manner of the fourth aspect, as an optional manner of lightweighting the first layer set in the first possible implementation manner, the method includes:

[0100] The network structure of the first layer set is optimized based on the depth-wise separable convolution technology.

[0101] By optimizing the network structure of the first layer set through the deep separable convolution technology, the network structure redundancy of the model layer in the first layer set can be reduced, so that the number of weight parameters and the amount of calculation in the first layer set can be reduced.

[0102] In a third possible implementation manner of the fourth aspect, before performing iterative training on the weight parameters of the second layer set, the method further includes:

[0103] A preset activation probability corresponding to each second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters of the convolution operator in the second layer set.

[0104] In each training process of the second layer set, including:

[0105] According to the corresponding preset activation probability, the second layer set to be activated is selected, and the weight parameters of the selected second layer set are updated.

[0106] By setting the activation probability, the training of each second-layer set is controlled to ensure that the second-layer set of low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0107] In a fourth possible implementation manner of the fourth aspect, before performing iterative training on weight parameters of the first layer set and the second layer set, the method further includes:

[0108] The preset activation probability corresponding to each first layer set and second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters of the convolution operators in the first layer set and the second layer set.

[0109] In each training process of the first layer set and the second layer set, it includes:

[0110] According to the corresponding preset activation probability, a layer set to be activated is selected from the first layer set and the second layer set, and the weight parameters of the selected layer set are updated.

[0111] By setting the activation probability, the training of each first-layer set and second-layer set can be controlled to ensure that the layer set with low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0112] In a fifth possible implementation manner of the fourth aspect, the iterative training process of the weight parameters of each layer set includes:

[0113] If the image classification model to be processed does not meet the preset convergence conditions after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, at least one performance indicator parameter of the image classification model to be processed after the training is completed is obtained.

[0114] Using preset sample data and at least one performance indicator parameter, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the image classification model to be processed meets the preset convergence conditions and the lightweight processing is completed.

[0115] In an embodiment of the present application, by iteratively updating the weight parameters of the layer set in a manner of feedback indicator parameters each time training, the training effect of the model can be improved, while the speed at which the model meets the convergence conditions can be accelerated, thereby improving the model training efficiency.

[0116] In a sixth possible implementation manner of the fourth aspect, the iterative training process of the weight parameters of each layer set includes:

[0117] A differentiable neural architecture search method based on gradient descent is used to iteratively train the weight parameters of the model.

[0118] Since differentiable neural structure search can realize the search for the best architecture in the search space, on the basis of setting the search nodes as the layer sets in each level, the model can be trained through the differentiable neural structure search training method, and the layer sets in each level can be selected during the model training process, so that when the model meets the convergence conditions, a model composed of the selected layer sets can be obtained.

[0119] In a seventh possible implementation manner of the fourth aspect, the iterative training process of the weight parameters of each layer set includes:

[0120] If the image classification model to be processed meets the preset convergence conditions after iterative training, each second layer set is used as a path node, and a layer set path search is performed on the image classification model to be processed to obtain a layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed image classification model.

[0121] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0122] In an eighth possible implementation manner of the fourth aspect, the iterative training process of the weight parameters of each layer set includes:

[0123] If the image classification model to be processed meets the preset convergence conditions after iterative training, the path search of the layer set is performed on the image classification model to be processed with each first layer set and the second layer set as the path node to obtain the layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed image classification model.

[0124] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0125] A fifth aspect of the embodiments of the present application provides a speech recognition model training method, comprising:

[0126] The hidden layer of the speech recognition model to be processed is divided into at least one first layer set, wherein each first layer set contains at least one first model layer, and the first model layer is a hidden layer containing a preset operator, and the preset operator is a time series operator or a convolution operator, the time series operator is used to process time series data, and the convolution operator is used for speech feature analysis. Each first layer set is lightweight processed to obtain a corresponding second layer set. The weight parameters of the second layer set are iteratively trained using preset sample speech, or the weight parameters of the first layer set and the second layer set are iteratively trained until the speech recognition model to be processed meets the preset convergence conditions, thereby obtaining a trained speech recognition model. The lightweight processing includes at least two of the following processing: quantizing the weight parameters contained in the preset operator in the layer set, reducing the number of weight parameters contained in the preset operator in the layer set, and adjusting the network structure between the model layers in the layer set.

[0127] Since the amount of data of the weight parameters in the preset operator in the selected layer set is smaller than that of the original speech recognition model to be trained, and the amount of calculation during model operation is also greatly reduced, while ensuring the performance of the speech recognition model, the storage space occupied by the electronic device and the computing resources occupied during model operation are reduced.

[0128] In a first possible implementation manner of the fifth aspect,

[0129] The reducing the number of weight parameters contained in the preset operators in the layer set includes:

[0130] Reduce the size of the convolution kernel contained in the preset operator, or reduce the number of convolution kernels in the preset operator

[0131] By reducing the size and number of convolution kernels, lightweight operations on layer sets can be better achieved.

[0132] In a second possible implementation manner of the fifth aspect, as an optional manner of lightweighting the first layer set in the first possible implementation manner, the method includes:

[0133] The network structure of the first layer set is optimized based on the depth-wise separable convolution technology.

[0134] By optimizing the network structure of the first layer set through the deep separable convolution technology, the network structure redundancy of the model layer in the first layer set can be reduced, so that the number of weight parameters and the amount of calculation in the first layer set can be reduced.

[0135] In a third possible implementation manner of the fifth aspect, before performing iterative training on the weight parameters of the second layer set, the method further includes:

[0136] The preset activation probability corresponding to each second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters of the preset operators in the second layer set.

[0137] In each training process of the second layer set, including:

[0138] According to the corresponding preset activation probability, the second layer set to be activated is selected, and the weight parameters of the selected second layer set are updated.

[0139] By setting the activation probability, the training of each second-layer set is controlled to ensure that the second-layer set of low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0140] In a fourth possible implementation manner of the fifth aspect, before performing iterative training on weight parameters of the first layer set and the second layer set, the method further includes:

[0141] The preset activation probability corresponding to each first layer set and second layer set is obtained, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters of the preset operators in the first layer set and the second layer set.

[0142] In each training process of the first layer set and the second layer set, it includes:

[0143] According to the corresponding preset activation probability, a layer set to be activated is selected from the first layer set and the second layer set, and the weight parameters of the selected layer set are updated.

[0144] By setting the activation probability, the training of each first-layer set and second-layer set can be controlled to ensure that the layer set with low-precision weight parameters can be fully trained, so that the overall model training effect is better.

[0145] In a fifth possible implementation manner of the fifth aspect, the iterative training process of the weight parameters of each layer set includes:

[0146] If the speech recognition model to be processed does not meet the preset convergence conditions after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, obtain at least one performance indicator parameter of the speech recognition model to be processed after the training is completed.

[0147] Using preset sample data and at least one performance indicator parameter, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the speech recognition model to be processed meets the preset convergence conditions and the lightweight processing is completed.

[0148] In an embodiment of the present application, by iteratively updating the weight parameters of the layer set in a manner of feedback indicator parameters each time training, the training effect of the model can be improved, while the speed at which the model meets the convergence conditions can be accelerated, thereby improving the model training efficiency.

[0149] In a sixth possible implementation manner of the fifth aspect, the iterative training process of the weight parameters of each layer set includes:

[0150] A differentiable neural architecture search method based on gradient descent is used to iteratively train the weight parameters of the model.

[0151] Since differentiable neural structure search can realize the search for the best architecture in the search space, on the basis of setting the search nodes as the layer sets in each level, the model can be trained through the differentiable neural structure search training method, and the layer sets in each level can be selected during the model training process, so that when the model meets the convergence conditions, a model composed of the selected layer sets can be obtained.

[0152] In a seventh possible implementation manner of the fifth aspect, the iterative training process of the weight parameters of each layer set includes:

[0153] If the speech recognition model to be processed meets the preset convergence conditions after iterative training, each second-layer set is used as a path node, and a layer set path search is performed on the speech recognition model to be processed to obtain a layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed speech recognition model.

[0154] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0155] In an eighth possible implementation manner of the fifth aspect, the iterative training process of the weight parameters of each layer set includes:

[0156] If the speech recognition model to be processed meets the preset convergence conditions after iterative training, the path search of the layer set is performed on the speech recognition model to be processed with each first layer set and the second layer set as the path node to obtain the layer set path that meets the preset performance index requirements, and the model composed of each path node in the layer set path is used as the processed speech recognition model.

[0157] By using layer sets as search nodes and performing path search on layer sets for models that meet convergence conditions, it is possible to obtain layer sets that are suitable for retention when the model meets the convergence conditions, thereby achieving lightweight training of the model.

[0158] A sixth aspect of an embodiment of the present application provides a terminal model processing device, including:

[0159] The acquisition module is used to obtain the terminal model to be processed.

[0160] The model lightweight module is used to perform lightweight processing on the terminal model to be processed to obtain the processed terminal model.

[0161] The data processing module is used to obtain the data to be processed and process the data to be processed through the processed terminal model to obtain the corresponding processing results.

[0162] Model lightweight module, including:

[0163] The layer set division module is used for dividing the model layers of the terminal model to be processed into at least one first layer set.

[0164] A lightweight submodule is used to perform lightweight processing on each of the first layer sets to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing the weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between model layers in the layer set.

[0165] The iterative training module is used to iteratively train the weight parameters of the second layer set using preset sample data, or iteratively train the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence conditions and completes the lightweight processing.

[0166] In a first possible implementation manner of the sixth aspect, the lightweight submodule includes:

[0167] The reduction module is used to reduce the size of the convolution kernels contained in the layer set, or reduce the number of convolution kernels.

[0168] In a second possible implementation manner of the sixth aspect, the layer set lightweight module includes:

[0169] The network structure of the first layer set is optimized based on the depth-wise separable convolution technology.

[0170] In a third possible implementation manner of the sixth aspect, the terminal model processing device further includes:

[0171] The probability acquisition module is used to obtain the preset activation probability corresponding to each second-layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameter in the second-layer set.

[0172] Correspondingly, the iterative training module includes:

[0173] The layer set activation module is used to select the second layer set to be activated according to the corresponding preset activation probability, and update the weight parameters of the selected second layer set.

[0174] In a fourth possible implementation manner of the sixth aspect, the terminal model processing device further includes:

[0175] The probability acquisition module is used to obtain the preset activation probability corresponding to each first layer set and the second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set.

[0176] Correspondingly, the iterative training module includes:

[0177] The layer set activation module is used to select the layer set to be activated from the first layer set and the second layer set according to the corresponding preset activation probability, and update the weight parameters of the selected layer set.

[0178] In a fifth possible implementation manner of the sixth aspect, the iterative training module includes:

[0179] The parameter feedback module is used to obtain at least one performance indicator parameter of the terminal model to be processed after the single training of the second layer set is completed, if the terminal model to be processed does not meet the preset convergence condition after the single training of the first layer set and the second layer set is completed.

[0180] The iterative training submodule is used to iteratively train the weight parameters of the second layer set using preset sample data and at least one performance indicator parameter, or to iteratively train the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence conditions and completes the lightweight processing.

[0181] In a sixth possible implementation manner of the sixth aspect, the iterative training module includes:

[0182] The path search module is used to search the path of the layer set of the terminal model to be processed if the terminal model to be processed meets the preset convergence conditions after iterative training, taking each second layer set as the path node, to obtain the layer set path that meets the preset performance index requirements, and use the model composed of each path node in the layer set path as the processed terminal model.

[0183] In a seventh possible implementation manner of the sixth aspect, the iterative training module includes:

[0184] The path search module is used to search the path of the layer set of the terminal model to be processed if the terminal model to be processed meets the preset convergence conditions after iterative training, taking each first layer set and the second layer set as the path node, to obtain the layer set path that meets the preset performance index requirements, and use the model composed of each path node in the layer set path as the processed terminal model.

[0185] It can be understood that the sixth aspect is the device content corresponding to the first aspect, and its beneficial effects can be found in the relevant description of the first aspect, which will not be repeated here.

[0186] The seventh aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the electronic device implements the steps of the terminal model processing method as described in any one of the first aspects above, or implements the steps of the terminal model training method as described in any one of the second aspects above, or implements the steps of the image classification model training method as described in any one of the third aspects above, or implements the steps of the speech recognition model training method as described in any one of the fourth aspects above.

[0187] An eighth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising: storing a computer program, characterized in that when the computer program is executed by a processor, the electronic device implements the steps of the terminal model processing method as described in any one of the first aspects above, or implements the steps of the terminal model training method as described in any one of the second aspects above, or implements the steps of the image classification model training method as described in any one of the third aspects above, or implements the steps of the speech recognition model training method as described in any one of the fourth aspects above.

[0188] The ninth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the terminal model processing method described in any one of the first aspects above, or implements the steps of the terminal model training method described in any one of the second aspects above, or implements the steps of the image classification model training method described in any one of the third aspects above, or implements the steps of the speech recognition model training method described in any one of the fourth aspects above.

[0189] It can be understood that the beneficial effects of the sixth to ninth aspects mentioned above can be found in the relevant descriptions in the first, second, third and fourth aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0190] Figure 1 It is a flowchart of a neural network model training method provided in one embodiment of the present application;

[0191] Figure 2 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0192] Figure 3A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0193] Figure 3B It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0194] Figure 4A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0195] Figure 4B It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0196] Figure 4C It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0197] Figure 5A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0198] Figure 5BIt is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0199] Figure 5C It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0200] Figure 5D It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0201] Figure 5E It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0202] Figure 6 It is a flowchart of a neural network model training method provided in one embodiment of the present application;

[0203] Figure 7 It is a flowchart of a neural network model training method provided in one embodiment of the present application;

[0204] Figure 8 It is a structural schematic diagram of a neural network model training device provided in an embodiment of the present application;

[0205] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0206] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0207] The terminal model processing method, terminal model training, image classification model training and speech recognition model training methods provided in the embodiments of the present application can be applied to the same electronic device or different electronic devices. The specific device type of the electronic device is not limited here, including but not limited to servers, as well as terminal devices such as mobile phones, computers and wearable devices, which need to be determined by the actual application scenario.

[0208] In order to facilitate the understanding of the present application, the embodiments of the present application are briefly described here:

[0209] When building a terminal model, in order to improve the performance of the terminal model, technicians often stack more model layers and train more weight parameters. As the number of model layers and weight parameters increases, the size of the model becomes more and more bloated, and the amount of calculation during operation increases greatly, which causes the storage space occupied when the model is stored and the computing resources occupied when it is running to grow rapidly, reducing the processing performance of the electronic devices that store and run the model. Among them, the specific model type of the terminal model is not limited here and can be determined by the technician according to the actual scenario. For example, it includes but is not limited to neural network models, decision tree models, and random forest models in the field of artificial intelligence.

[0210] In order to reduce the volume of the model and reduce the computing resources occupied by the model when it is running, the lightweight processing of the model is realized. The embodiment of the present application first divides the model to be processed into layer sets, and performs multiple lightweight processing on the layer sets as units to obtain layer sets with smaller volume and less computing resources. After obtaining the quantized layer sets corresponding to each layer set, these layer sets are then subjected to mixed iterative training of single precision or different precision weight parameters until the model converges. Since the model after iterative training meets the performance convergence conditions, and there will be certain differences in the data processing effects corresponding to the combination of different layer sets in the model. Therefore, suitable layer sets can be selected from the layer sets with smaller volume and less computing resources for combination, thereby ensuring the performance of the final model. At the same time, since the amount of data of the selected layer set weight parameters is smaller than the original model to be processed, and the amount of calculation when the model is running is also greatly reduced, the volume of the model is reduced and computing resources are saved while ensuring the performance of the model.

[0211] The model to be processed can be a trained model or an untrained initial model. The essence of lightweight processing of the untrained initial model is to train the weight parameters of the initial model. During the training process, the weight parameters are quantized and mixed with weight parameters of different storage precisions to reduce the number of weight parameters and storage precision of the trained model. The essence of lightweight processing of the trained model is to retrain the model to achieve compression processing of the model, reduce the number of weight parameters and reduce the storage precision of weight parameters.

[0212] At the same time, according to different actual scenarios, the two operations of model lightweighting and data processing using the lightweight model in the embodiment of the present application can be performed in the same electronic device or in different electronic devices. For example, after completing the lightweight operation of the model, electronic device A can process the data to be processed based on the lightweight model. At this time, since the model is smaller in size, it occupies fewer computing resources during operation, thereby saving the local storage space of electronic device A and reducing the occupation of processor resources of electronic device A when the model is running, thereby improving the processing performance of electronic device A. It is also possible that after electronic device A completes the lightweight operation of the model, the lightweight model is stored in electronic device B, and electronic device B uses the lightweight model to process data. At this time, the local storage space of electronic device B can be saved, and the occupation of processor resources of electronic device B when the model is running can be reduced, thereby improving the processing performance of electronic device B. Among them, electronic device A and electronic device B are two different devices, and both can be servers or terminal devices.

[0213] In order to illustrate the technical solution described in the present application, the terminal model is taken as a neural network model as an example and explained through a specific embodiment.

[0214] Figure 1 The following is a flowchart of the implementation of the neural network model training method provided in Example 1 of the present application, which is described in detail as follows:

[0215] S101, the electronic device obtains preset sample data and a neural network model to be processed.

[0216] The preset sample data is used to train the model to be processed, and the preset sample data can be stored in the electronic device by the technician according to the actual scene requirements. The data type of the preset sample data and the method of obtaining the model to be processed need to be determined in combination with the model type and training situation of the model to be processed in the actual application scenario, and are not limited here. The model types include but are not limited to image classification models, natural language processing models, machine translation models, speech recognition models, and video processing models.

[0217] For example, when the embodiment of the present application is applied in an image classification scenario, the model to be processed is the image classification model, and the preset sample data is the sample image that can be classified. Depending on whether a trained image classification model is stored in the electronic device, S101 includes at least the following situations:

[0218] 1. If a trained image classification model is stored in the electronic device, the electronic device can read the trained image classification model and use it as a model to be processed.

[0219] 2. If the electronic device does not store a trained image classification model, the electronic device can build a model for image classification by itself, initialize the weight parameters of the model, and then use the initialized model as the model to be processed.

[0220] For another example, when the embodiment of the present application is combined with a speech recognition scenario application, the model to be processed is the speech recognition model, and the preset sample data is the sample speech. Depending on whether a trained speech recognition model is stored in the electronic device, S101 may also include the above-mentioned situation 1 and situation 2 (only the image classification model needs to be replaced with the speech recognition model). Other application scenarios can also refer to the above-mentioned image classification scenario to determine the preset sample data and operate S101, which will not be described in detail here.

[0221] S102, the electronic device divides the model layers of the neural network model to be processed into at least one first layer set.

[0222] For a neural network model, it is generally composed of an input layer, an output layer, and multiple hidden layers connected in series. In an embodiment of the present application, a model layer refers to a hidden layer of a neural network model. These hidden layers can be divided into model layers with weight parameters (such as convolutional layers, fully connected layers, and batch normalization layers, etc.), and model layers without weight parameters (such as incentive layers). Neural network model training mainly refers to updating and training the weight parameters in each model layer so that the neural network model after training can meet the preset convergence conditions.

[0223] On the basis of each model layer of the neural network model, different model layers may have different network structures, such as serial structure, pyramid structure and residual network structure. There will be certain differences in the connection relationship and parameter update logic between model layers in different network structures, which will lead to large differences in the number of weight parameters finally retained. Therefore, in order to achieve the optimization of the model to be processed. After obtaining the model to be processed, the embodiment of the present application will divide the model layer of the model to be processed into multiple original layer sets, wherein each original layer set contains one or more model layers. At this time, the specific network structure between the model layers in each original layer set and the operator situation contained in the original layer set, such as the number of convolution kernels contained in the operator, the size of each convolution kernel and the number of weight parameters contained in the operator, can be known, as the basis for lightweighting the original layer set. Among them, the embodiment of the present application does not limit the specific original layer set division method, which can be set by the technician according to actual needs. However, it is necessary to meet the requirement that at least one model layer with weight parameters is contained in a single original layer set to ensure that the original layer set can be optimized for weight parameters.

[0224] As an optional embodiment of the present application, in order to ensure that the quantization layer set can normally update and use weight parameters, the original layer set division rules can be set to: include one or more model layer branches, at least one of which contains a convolutional layer or a fully connected layer.

[0225] In an embodiment of the present application, the original layer set obtained after the model layer is divided is uniformly named as the first layer set, and the quantized layer set obtained after the weight parameter quantization processing of the original layer set is uniformly named as the second layer set.

[0226] As an embodiment of the present application, refer to Figure 2 Part (a) in the figure is a neural network model A that contains 9 model layers. Figure 2 In part (b), in the embodiment of the present application, the 9 model layers of the neural network model A are divided into a first layer set A, a first layer set B and a first layer set C. The first layer set A includes 2 convolutional layers, 1 excitation layer and 1 pooling layer, the first layer set B includes 1 convolutional layer and 1 pooling layer, and the first layer set C includes 1 convolutional layer, 1 pooling layer and 1 fully connected layer.

[0227] S103, the electronic device performs lightweight processing on each first layer set to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing the weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between model layers in the layer set.

[0228] In order to improve the performance of the model, when setting the model weight parameters, data types with higher precision and larger number of bytes are often selected as the data storage type of the weight parameters, such as full-precision floating point and half-precision floating point. On the one hand, the volume of the model will be larger. On the other hand, weight parameters with too high precision will also lead to a sharp increase in computing resources occupied when the model is running. In order to ensure the performance of the model while lightweighting the model, the embodiment of the present application will perform lightweight processing on a layer set basis, thereby obtaining the original layer set that has not been lightweight and the corresponding lightweight quantized layer set. The model is then trained based on these quantized layer sets to ensure the performance of the final model. Among them, lightweighting refers to reducing the volume of the layer set and reducing the amount of calculation of the weight parameters in the layer set. In the embodiment of the present application, three optional lightweight methods are provided, including:

[0229] Method 1: Quantize the weight parameters in the layer set.

[0230] Method 2: Reduce the number of weight parameters in the layer collection.

[0231] Method 3: Optimize the network structure between model layers in the layer set.

[0232] For method 1, the quantization processing of the weight parameters refers to reducing the number of bytes or bits occupied when the weight parameters are stored, so as to reduce the storage space occupied by the weight parameters. For example, for a weight parameter that originally occupies 4 bytes, the number of bytes it occupies can be reduced to 2 bytes, or reduced to less than 1 byte, such as 2 bits and 1 bit, by means of a linear quantization method. The embodiment of the present application does not limit the method, number of times of quantization of the weight parameters, and the number of bytes occupied by the target for each quantization, which can be selected or set by technical personnel according to actual needs. Among them, for the quantization method, it includes but is not limited to a linear quantization method, an exponential quantization method, and a binary quantization method. For the number of quantization times, it can be set to any integer greater than or equal to 1. For the target number of bytes occupied by quantization, it can be set to be less than the actual number of bytes occupied by the weight parameters in the original layer set. For the case where the number of quantization times is greater than 1, it can be set so that the target number of bytes occupied for each quantization is different to ensure the difference of each quantization and ensure the effect of quantization.

[0233] As an embodiment of the present application, it is assumed that the neural network model A after layer set division includes the first layer set A, the first layer set B and the first layer set C, each of which occupies 4 bytes of weight parameters. At the same time, the weight parameters of each first layer set are quantized once, and the target occupies 1 byte, and the result is Figure 3A The corresponding neural network model A. At this time, the neural network model A includes the second layer set a1, the first layer set b1 and the first layer set c1, each of which occupies 1 byte of weight parameters.

[0234] exist Figure 3A On the basis of , assuming that the weight parameters of each first-layer set are quantized once, and the target occupied byte number is 2 bits (that is, two quantization operations with different target occupied byte numbers are performed in total), we can get Figure 3B The corresponding neural network model A. At this time, the neural network model A also includes the second layer set a2, the first layer set b2 and the first layer set c2, each of which occupies 2 bits of bytes.

[0235] For method 2, reducing the number of weight parameters in the layer set can reduce the volume of the layer set. The embodiments of the present application do not place too many restrictions on the specific method of reducing the number of weight parameters, which can be selected or set by technical personnel according to actual needs. For example, in some embodiments, the number of convolution kernels in the layer set can be reduced randomly or according to certain preset rules. In other embodiments, the size of the convolution kernel in the layer set can also be reduced to reduce the number of weight parameters included. In some other embodiments, the number of convolution kernels in the layer set can also be reduced, and the size of the remaining convolution kernels can also be reduced at the same time.

[0236] Among them, the specific reduction rules for the convolution kernel size can be set by the technical staff according to actual needs, and are not limited here. As an optional embodiment of the present application, it can be set to reduce the size of each convolution kernel in the layer set by n units. For example, when n=2, the original 7×7 convolution kernel will be reduced to 5×5. As another optional embodiment of the present application, it can also be set that the size of each convolution kernel in the layer set is randomly reduced to a size greater than or equal to 1×1. For example, two convolution kernels of the original size of 7×7 may have one randomly reduced to 5×5 and the other randomly reduced to 3×3. At the same time, the method of reducing the number of convolution kernels and the number of channels is not limited here, and can be set by the technical staff according to actual needs. Including but not limited to, randomly selecting some convolution kernels or channels for deletion.

[0237] Regarding method 3, considering that the network structure redundancy of the model layer in the traditional neural network model is high, the traditional neural network model is large in size and consumes more computing resources during operation. In order to achieve the lightweight of the layer set, in the embodiment of the present application, the network structure of each original layer set can be optimized to reduce the network structure redundancy of the original layer set, so that the number of weight parameters and the amount of calculation in the original layer set can be reduced.

[0238] Among them, the specific network structure optimization method is not limited here and can be selected or set by the technical personnel according to the actual situation. Including but not limited to some optimization methods based on the depth separable convolution technology, such as using the FPN algorithm to optimize the original layer set, or adjusting the network structure of the original layer set to some lightweight network structures, such as mobilenet and resnet type model layer network structures. After being processed by the depth separable convolution technology, the depth separable convolution operator will be optimized, so that the obtained quantization layer set does not contain the depth separable convolution operator, thereby making the number of weight parameters contained in the operator in the quantization layer set and the corresponding amount of calculation smaller than the original layer set.

[0239] refer to Figure 4A , Figure 4B and Figure 4C , are several optional lightweight network structures provided in the embodiments of the present application. In practical applications, the network structure of the original layer set can be adjusted to any one of them to optimize the network structure of the original layer set.

[0240] In order to ensure the lightweight effect of the layer set, based on the above 3 optional lightweight methods, the embodiment of the present application will select 2 or 3 of them as lightweight solutions for the original layer set. Therefore, the embodiment of the present application actually includes: method 1 + method 2, method 1 + method 3, method 2 + method 3, and method 1 + method 2 + method 3, a total of 4 available lightweight solutions. In the actual application process, the technicians can select any one of the lightweight solutions according to actual needs to achieve lightweight processing of the original layer set.

[0241] Take an example to illustrate. Assume that layer set a0, layer set b0 and layer set c0 are the original layer sets of model A. The weight parameter precision in each original layer set is 4 bytes, and the convolution kernel size is 7×7. Method 1+Method 2+Method 3 are selected as the lightweight solution for the original layer set. In method 1, two target occupancy bytes are set, which are 1 byte and 2 bits respectively. In method 2, two size reduction units are set, which are 2 units and 4 units respectively. In method 3, two optimized network architectures for replacement are set, which are Figure 4B and Figure 4C The network structure in . At this time, a total of 2×2×2=8 specific lightweight solutions can be obtained.

[0242] After processing layer set a0, layer set b0 and layer set c0 respectively using 8 lightweight schemes, a model A containing 3×8=24 quantization layer sets can be obtained. Referring to FIG5 , the parameter configuration of each quantization layer set is as follows:

[0243] Layer sets a1, b1, and c1: weight parameter precision is 1 byte, convolution kernel size is 5×5, and the space between model layers is Figure 4B The network structure in .

[0244] Layer sets a2, b2, and c2: weight parameter precision is 1 byte, convolution kernel size is 5×5, and the distance between model layers is Figure 4C The network structure in .

[0245] Layer sets a3, b3, and c3: weight parameter precision is 1 byte, convolution kernel size is 3×3, and the distance between model layers is Figure 4B The network structure in .

[0246] Layer sets a4, b4, and c4: weight parameter precision is 1 byte, convolution kernel size is 3×3, and the distance between model layers is Figure 4C The network structure in .

[0247] Layer sets a5, b5, and c5: weight parameter precision is 2 bits, convolution kernel size is 5×5, and the distance between model layers is Figure 4B The network structure in .

[0248] Layer sets a6, b6, and c6: weight parameter precision is 2 bits, convolution kernel size is 5×5, and the distance between model layers is Figure 4C The network structure in .

[0249] Layer sets a7, b7, and c7: weight parameter precision is 2 bits, convolution kernel size is 3×3, and the distance between model layers is Figure 4B The network structure in .

[0250] Layer sets a8, b8, and c8: weight parameter precision is 2 bits, convolution kernel size is 3×3, and the distance between model layers is Figure 4C The network structure in .

[0251] S104, the electronic device uses preset sample data to iteratively train the weight parameters of the second layer set, or iteratively trains the weight parameters of the first layer set and the second layer set, until the neural network model to be processed meets the preset convergence conditions and the training is completed.

[0252] In order to ensure model performance while lightweighting the model, after the above lightweighting operation on each original layer set, the embodiment of the present application will start training the model, that is, update the weight parameter values ​​in each layer set of the model until the performance index of the model meets the convergence condition requirements.

[0253] Among them, in actual applications, the functions of each layer set in the model will be different. Under the condition of meeting the model performance, different layer sets may also have different requirements for the accuracy, quantity and model layer network structure of weight parameters. For example, when the model is an image classification model, the function of some layer sets is to extract image features, but the accuracy of weight parameters required for different image feature extractions may be different. Therefore, if the weight parameters of all layer sets of the model are set to the same accuracy, on the one hand, if the accuracy is set to a higher level, such as maintaining the original full-precision floating point type, although the performance of the model can be guaranteed, it will also cause the model to be larger; on the other hand, if the accuracy is set to a lower level, such as 1 bit, at this time, although the model size will be reduced, it will cause the model to have a larger error and performance will be difficult to guarantee.

[0254] In order to take into account both the lightweight effect and performance of the model, the embodiment of the present application can adopt two methods for training the model:

[0255] 1. Keep the original layer set and the quantized layer set at the same time, and perform mixed training of multiple precision weight parameters.

[0256] 2. Only retain the set of quantized layers for single-precision or mixed-precision weight parameter training.

[0257] For case 1, an example is given to illustrate. Figure 5A , Figure 5B .exist Figure 5A Based on the corresponding instance, if training is performed according to the above method 1, then Figure 5A The corresponding model A is trained. Figure 5B Yes Figure 5A The model A is obtained by discarding the original layer set in . If the training is performed according to the above method 2, then Figure 5B The corresponding model A is trained.

[0258] Figure 5C This is a model that performs only one light quantization on each original layer set, so the accuracy of each quantized layer set is consistent. If the original layer set is discarded according to the above method 2, we can get Figure 5D The corresponding model. Figure 5D The corresponding model is trained.

[0259] For case 2, it can be further divided into two cases:

[0260] a. In S103, each original layer set is quantized only once, and each original layer set corresponds to only one quantized layer set. Figure 5C and Figure 5D The corresponding instance.

[0261] b. In S103, each original layer set is quantized multiple times. At this time, each original layer set corresponds to multiple quantized layer sets. Figure 5A and Figure 5B In the corresponding example, each original layer set is quantized 8 times.

[0262] For case a, since the embodiment of the present application only retains a set of quantization layers for training, at this time, only the weight parameter iterative update training will be performed on each set of quantization layers with single precision weight parameters.

[0263] For case b, at this time, the weight parameters of each quantization layer set with different weight parameter precisions are iteratively updated and trained to achieve mixed precision training of the quantization layer set.

[0264] In an embodiment of the present application, the quantization times of the original layer set can be set to be greater than 1 so that only situation b will occur in situation 2, thereby realizing the weight parameter training of the layer set with mixed precision.

[0265] In the embodiment of the present application, the abandonment of the original layer set means not to perform iterative update training on the weight parameters of the original layer set. The specific abandonment method is not limited here and can be set by the technicians, including but not limited to deleting the original layer set in the model, or setting the activation probability of all the original layer sets to 0.

[0266] The embodiments of the present application do not impose too many restrictions on the specific model training method, which can be selected or set by technicians according to actual needs. For example, the model training method includes but is not limited to the gradient descent method, the Newton method, the conjugate gradient method and the quasi-Newton method.

[0267] At the same time, the embodiments of the present application do not impose too many restrictions on the specific convergence conditions, and the technical personnel can select or set them according to the type of actual model and the needs of the application scenario. However, it should be understood that in order to ensure the actual application scenario's demand for model performance, the convergence condition should include at least one performance indicator of the model, such as the accuracy, operating efficiency and power consumption of the model. Among them, the specific performance indicators and the indicator range corresponding to the indicators can also be set by the technical personnel according to the actual application scenario requirements. For example, assuming that the model is a face recognition model, it is applied to the face identity authentication scenario of the mobile payment software. At this time, the face recognition accuracy and recognition time (i.e., operating efficiency) of the model are required to be high. The technical personnel can set the face recognition accuracy and recognition time to the convergence conditions, and set the corresponding indicator range according to the actual demand for face recognition accuracy and recognition time. For example, the indicator range can be set to: the face recognition accuracy is set to be greater than 95%, and the recognition time is less than 1 second. In addition, in addition to the performance of the model itself, lightweight indicators such as the volume of the model and the computing resources occupied during operation can also be included in the convergence conditions to ensure the lightweight effect of the model.

[0268] In addition, in order to prevent unreasonable settings of model performance indicators and lightweight indicators, which may cause the model to converge too slowly or even fail to converge normally, making it impossible for model training to be completed normally. While setting model performance indicators and lightweight indicators, you can also set some indicators such as the maximum number of iterations and the maximum training time to the convergence conditions, or you can also set some indicator change trend ranges to the convergence conditions to ensure that the model can complete training normally.

[0269] about Figure 1 A few remarks on the illustrated embodiment:

[0270] 1. The number of training times for different layer sets can vary to a certain extent.

[0271] In actual applications, it is found that for layer sets with higher precision of weight parameters, such as full-precision floating-point weight parameters, the weight parameters can carry more information, so the training effect of the layer sets in actual training is often better. On the contrary, for layer sets with lower precision of weight parameters, such as Boolean weight parameters, the weight parameters can carry less information, so the training effect of the layer sets in actual training is often poor. It is for this reason that when mixed training is performed on layer sets with different precision of weight parameters, if all layer sets are trained the same number of times, the training effects of each layer set will be very different. In particular, for quantized layer sets with lower precision of weight parameters, their weight parameters cannot be fully trained, which makes the training effect of the quantized layer set poor.

[0272] In order to avoid large differences in the training effects of each layer set in the model, resulting in poor overall model training effect. Figure 1 In the illustrated embodiment, when a lightweight layer set is selected by quantizing the weight parameters. In the embodiment of the present application, the number of training times corresponding to each layer set can also be adjusted. The principle of adjustment is that the higher the weight parameter precision, the fewer the corresponding training times, and the lower the weight parameter, the more the corresponding training times. The ratio of the number of training times corresponding to the precision of each weight parameter is not limited here, and can be set by technical personnel according to actual needs, as long as the number of training times is negatively correlated with the precision of the weight parameter. For example, in some optional embodiments, assuming that the unit of precision is set to the number of occupied bytes, the ratio of the number of training times corresponding to 4 bytes, 3 bytes, 2 bytes and 1 byte can be set to: 1:2:4:8. In other optional embodiments, assuming that the unit of precision is set to occupied bits, the ratio of the number of training times corresponding to 32 bits, 16 bits, 8 bits, 4 bits, 2 bits and 1 bit can be set to: 1:2:4:8:16:32.

[0273] As an optional embodiment of the present application, in order to implement the above-mentioned adjustment of the number of training times for each layer set, the specific operations include:

[0274] Before iteratively training the weight parameters of the layer set, the preset activation probability corresponding to each layer set is obtained, wherein the preset activation probability is negatively correlated with the accuracy of the weight parameters in the layer set.

[0275] In the process of iteratively training the weight parameters of the layer set in S104, each training process of the layer set includes:

[0276] The set of layers to be activated is selected according to the corresponding preset activation probability, and the weight parameters of the selected set of layers are updated.

[0277] In order to control the ratio of the number of training times for each layer set, in an embodiment of the present application, an activation probability is set for each layer set to be trained. S104 During the model training process, each time when the weight parameters of the layer set need to be updated, the layer set that needs to be activated at that time will be determined according to the activation probability, and the weight parameters will only be updated for the layer set that is determined to be activated at that time. Since the training of the model is a process of iteratively updating the weight parameters of each layer set, the number of iterations is often large. Therefore, by controlling the activation probability to select the activated layer set each time, it is possible to effectively control the ratio of the number of training times for each layer set of the model during the overall model training process. Among them, the activation probability value corresponding to the accuracy of each specific weight parameter can be set by the technician according to the actual training times ratio requirement, which is not limited here.

[0278] Take an example to illustrate, refer to Figure 5A Assume that the corresponding relationship between the precision of each weight parameter and the activation probability is: 4 bytes correspond to 8% total activation probability, 1 byte corresponds to 40% total activation probability, and 2 bits correspond to 52% total activation probability. On the basis of the above conditions, for case 1 in S104, the original layer set and the quantized layer set are retained to perform mixed training of weight parameters with multiple precisions. Each time the weight parameters of the layer set to be trained need to be updated, not all the layer sets to be trained are trained and updated, but the layer set that needs to be activated and updated at that time is first selected according to the activation probability, and then the selected layer set is trained.

[0279] For example, for layer set a0, layer set a1, layer set a2, layer set a3, layer set a4, layer set a5, layer set a6, layer set a7 and layer set a8, the corresponding activation probabilities are 8%, 10%, 10%, 10%, 10%, 13%, 13%, 13% and 13% respectively. During the iterative training process, each time the weight parameters of each layer set need to be updated, a layer set to be activated will be selected from layer set a0 to layer set a8 based on these activation probabilities, and only the weight parameters of the selected layer set will be updated. Similarly, the same activation selection operation will be performed for layer set b0 to layer set b8, layer set c0 to layer set c8. For example, assuming that layer set a0, layer set b2 and layer set c8 are selected this time, refer to Figure 5E , where the gray layer set is the layer set that is not activated this time. During this training, only the weight parameter update training will be performed on layer set a0, layer set b2 and layer combination c8, and the other 24 layer sets will not be updated.

[0280] Based on the embodiment of the present application, if you want to implement situation 2 in S104 to abandon the original layer set, only retain the quantized layer set for single-precision or mixed-precision weight parameter training. As a way to abandon each original layer set, the activation probability of all original layer sets can be set to 0. Since the layer set will not be updated and trained by the weight parameter when the activation probability is 0, each original layer set ultimately cannot meet the convergence requirements of the model and is then abandoned.

[0281] As another implementation method for case 2 in S104, all original layer sets can also be deleted, and the layer set selection and weight parameter update training for each training can be implemented according to the activation probability corresponding to the remaining quantized layer sets. For details, please refer to the above description of the layer set selection and weight parameter update based on the activation probability in case 1 in S104, which will not be repeated here. However, it should be noted that for case a in case 2 of S104 where only one quantization is performed, since the precision of the weight parameters of all quantized layer sets is the same, the activation probability does not need to be set at this time.

[0282] 2. The set of layers that the model ultimately retains can be selected during model training or after the model converges.

[0283] In the process of training the model in S104, except for case a: each original layer set is quantized once, and only the weight parameters of the quantized layer set are iteratively updated during training, each original layer set in the model will correspond to multiple layer sets to be trained. Figure 5A In the example shown, for layer set a0, there are 9 layer sets to be trained, from layer set a0 to layer set a8. Figure 5B In the example shown, for layer set a0, there are 8 layer sets to be trained, from layer set a1 to the second layer set a8. In the model that is finally trained, for each of the multiple layer sets to be trained corresponding to the original layer set, only one of the layer sets needs to be retained to realize the function of the model. Therefore, it is necessary to select the layer set that actually needs to be retained for the model in order to complete the final determination of the model structure.

[0284] In order to select each layer set in the model, two optional selection methods are provided in the embodiment of the present application as follows:

[0285] (1) Establish a hierarchical search space, each of which contains multiple sets of layers to be trained. Use the gradient descent-based Differentiable Neural Architecture Search (DNAS) method to iteratively train the model weight parameters. The selection of layer sets is completed during the iterative training process.

[0286] Corresponding to the case 1 in S104, each layer contains one original layer set and one or more quantized layer sets corresponding to the original layer set. Figure 5A Corresponding to the example, the neural network model A in this example can be divided into 3 levels. The first level contains 9 layer sets to be trained, from layer set a0 to layer set a8. The second level contains 9 layer sets to be trained, from layer set b0 to layer set b8. The third level contains 9 layer sets to be trained, from layer set c0 to layer set c8.

[0287] Corresponding to case b in S104, each layer contains multiple quantized layer sets corresponding to one original layer set, but does not contain the original layer set itself (the original layer set has been discarded). Figure 5B In the example shown, the neural network model A in this example can be divided into 3 levels. The first level includes 8 layer sets to be trained, from layer set a1 to layer set a8. The second level includes 8 layer sets to be trained, from layer set b1 to layer set b8. The third level includes 8 layer sets to be trained, from layer set c1 to layer set c8.

[0288] (2) First, train the model until it meets the convergence condition. Then, use each layer set in the model as a path node to perform path search and determine one of the layer set paths as the final path. Finally, a model consisting of each layer set in the final path is obtained.

[0289] For selection method (1), DNAS can realize the search for the best architecture in the search space. On the basis of setting the search node as the layer set in each level, the model is trained through the DNAS training method, and the layer set in each level can be selected during the model training process, so that when the model meets the convergence condition, a model composed of the selected layer set can be obtained.

[0290] As an optional embodiment of the present application, during the iterative training of the weight parameters of each layer set, after each training, it will be judged whether the model meets the convergence conditions. If not, the next weight parameters will be started for each layer set. Among them, in order to ensure the performance and lightweight effect of the obtained model, in the process of iterative model training using DNAS. Before starting a new training of each layer set, the performance index parameters corresponding to the model obtained in the current training will be calculated first, and these calculated performance index parameters will be fed back to the model to start the next training. The types of performance index parameters calculated specifically are not limited here, and can be selected or set by technical personnel according to actual scene requirements, including but not limited to any one or more of performance indicators such as model accuracy, operating efficiency and power consumption. As another optional embodiment of the present application, while calculating the performance index parameters, lightweight index parameters can also be calculated, such as the volume of the model and the computing resources occupied by the model when it is running. And these lightweight index parameters are fed back to the model together with the above-mentioned performance index parameters to start the next training.

[0291] By iteratively updating the weight parameters of the layer set by feeding back the indicator parameters during each training, the training effect of the model can be improved, while speeding up the speed at which the model meets the convergence conditions and improving the efficiency of model training.

[0292] When the selected model training method cannot automatically search for the optimal model architecture, when the model meets the convergence conditions, it only determines the values ​​of each weight parameter in each layer set in the model. At this time, the model still contains a large number of layer sets that need to be selected. In order to select the layer set in this case, the selection method (2) will determine the appropriate layer set by path search, as detailed below:

[0293] Since in case 1 and case b of S104, each original layer set will correspond to multiple layer sets to be trained. Assume that there are n original layer sets in total, and each original layer set corresponds to m layer sets to be trained. Taking each layer set as a path node, according to the permutations and combinations, it can be known that m layers can be combined at this time. n There are three kinds of layer collection paths, each of which corresponds to a complete model structure.

[0294] On this basis, the embodiment of the present application will calculate the index parameters for each layer set path, and select a layer set path that meets the index parameters as the final path. The screening method is not limited here, and can be selected or set by technical personnel according to actual needs, and will not be repeated here. For example, it can be m n Traverse the set of paths in each layer and select the final path with the best index parameters. nThe index parameters are calculated in turn for each layer set path until a path that meets the index parameter requirements is found and used as the final path. n The traversal of the path of the seed layer set has a smaller workload. Among them, the specific parameter types contained in the indicator parameters can be selected or set by the technicians according to actual needs. There are no excessive restrictions here, but it must include at least one performance indicator parameter, such as any one or more of model accuracy, operating efficiency and power consumption to ensure the performance of the model. At the same time, it can also include lightweight indicator parameters, such as one or more of the model volume and computing resources occupied during operation, to ensure the lightweight effect of the model.

[0295] refer to Figure 5A Corresponding example, assuming that the neural network model A meets the convergence conditions after training, there are 3 original layer sets in the neural network model A, and each original layer set corresponds to 9 layer sets to be trained. According to the permutations and combinations, there are 9×9×9=729 layer set paths. In the embodiment of the present application, the final path that meets the index parameter requirements can be screened out by calculating the index parameters of each layer set path. For example, assuming that by traversing 729 layer set paths, it is determined that the path: layer set a0-layer set b2-layer combination c8 is the final path with the best index parameters. Reference Figure 5E At this time, the embodiment of the present application will only retain the set of path node layers in the final path, and thus obtain the neural network model A that has been finally trained.

[0296] From the above description of the selection methods (1) and (2), it can be seen that the embodiment of the present application combines the lightweight operation of the model into the model training process by performing mixed precision training of weight parameters on each layer set and screening the layer set. In theory, a model composed of layer sets with multiple different weight parameter precisions can be obtained. Compared with the prior art of directly compressing the trained model and then performing performance verification on the compressed model. On the one hand, the embodiment of the present application can ensure the performance and lightweight effect of the model. On the other hand, a usable model with more flexible weight parameter precision can be obtained, which prevents the situation where there is no solution after model compression caused by direct compression of the model (that is, it is impossible to compress the model while ensuring the performance of the model). On the other hand, by performing model structure search in units of layer sets, the speed is faster and more efficient than the traditional model structure search in units of model layers. At the same time, combining the lightweight operation with the model training process can also greatly save the time used for model lightweighting and improve the efficiency of model lightweighting.

[0297] As an embodiment of the present application, after the lightweight training of the model is completed in the above-mentioned embodiments, the trained model can be stored locally in the electronic device, or it can also be stored in other devices. When the trained model is stored locally in the electronic device, the execution subject of the model lightweight and the model data processing is the same electronic device. At this time, the occupancy of the local storage space of the electronic device by the model can be reduced, and the occupancy of the computing resources of the electronic device when the model is running can be reduced. When the trained model is stored in other devices by sending model data or manually copying the model by a technician, the execution subject of the model data processing is the device that stores the model. At this time, the local storage space occupied by the model of the device that stores the model can be reduced, and the occupancy of the computing resources of the device that stores the model when the model is running can be reduced.

[0298] A terminal model training method provided in an embodiment of the present application is described in detail as follows:

[0299] Construct an initial model containing multiple layer groups to be trained. Each layer group to be trained contains multiple layer sets, each layer set contains one or more model layers, and the layer sets contained in each layer group to be trained have at least one of the following differences: the number of bytes occupied by weight parameters in the layer set, the number of weight parameters in the layer set, and the network structure between model layers in the layer set.

[0300] The weight parameters of the initial model are initialized, and the weight parameters of each layer set are iteratively trained until the initial model meets the preset convergence conditions to obtain a trained terminal model.

[0301] exist Figure 1 In the illustrated embodiment, the scenario is: there is already a model, and lightweight training is performed on the basis of the existing model, thereby reducing the size of the model and the computing resources occupied during operation. However, in actual applications, scenarios where there is no existing model are often encountered. In order to cope with such scenarios, the embodiment of the present application will build and train an initial model. In order to obtain a model that is smaller in size and occupies less computing resources than the model obtained by the traditional model building and training method.

[0302] The principles of the embodiments of this application are Figure 1 The embodiments shown are basically the same, so the concept description of layer set and model layer, as well as the specific training process description, can be referred to Figure 1 The relevant description of the embodiment shown is not repeated here. Figure 1 The differences of the embodiments shown are described below:

[0303] In the process of building the initial model, the embodiment of the present application will distinguish the various layer sets in the same layer group to be trained from three aspects: the storage accuracy and quantity of the weight parameters and the network architecture between the model layers. Among them, the relevant descriptions of the size accuracy and quantity of the weight parameters, as well as the description of the network structure of the model layers in the layer set, can be referred to Figure 1 The contents of the embodiments shown are not described in detail here. Figure 5A The example shown. Figure 1 In the embodiment shown, Figure 5A The model A is obtained by performing lightweight processing on the original layer set a0, the original layer set b0 and the original layer set c0 in the model A. However, in the embodiment of the present application, since there is no existing model, there is no need for the concepts of "original layer set" and "quantized layer set". Therefore, for the embodiment of the present application, the original layer set can be omitted and the following can be constructed: Figure 5A The initial model shown in Figure 1. At this time, the initial model contains 3 layers to be trained, and each layer group to be trained contains 9 layers. Figure 1 Corresponding to the lightweight method of layer sets 3 in the illustrated embodiment, the embodiment of the present application will also realize differentiated processing of each layer set from three aspects: the number of weight parameters of the layer set, the storage accuracy and the network structure of the model layer within the layer set.

[0304] At the same time, when constructing the initial model, the embodiment of the present application creates multiple different layer sets for each layer group to be trained. Therefore, when the embodiment of the present application performs model training, each layer group to be trained is a mixed training of layer sets of multiple precisions.

[0305] In the traditional model building and training process, there is no concept of layer set, and mixed training of multiple layer sets with different precisions will not be performed. Therefore, compared with the traditional model building and training method, the embodiment of the present application can train a model with more flexible and practical weight parameter storage precision and model layer network architecture, and the model is smaller in size and occupies less computing resources, which can greatly save the storage space of electronic devices and improve the performance of electronic devices.

[0306] It should be understood that Figure 1 The extended or refined embodiment scheme contents corresponding to the illustrated embodiment. Such as adjusting the ratio of the number of training times and setting the activation probability of each layer set, as well as selecting and retaining the model layer set, etc. can also be combined with the embodiments of the present application for application without exceeding the protection scope of the present application. The principles and details of the specific application can be parameterized. Figure 1 The corresponding related descriptions in the illustrated embodiment are not repeated here.

[0307] Figure 6The following is a flowchart of the image classification model training method provided in the second embodiment of the present application, which is described in detail as follows:

[0308] S601, the electronic device obtains a preset sample image and an image classification model to be processed.

[0309] The principle of S601 is basically the same as that of S101. For the related principle description and operation instructions, please refer to the instructions in S101 and will not be repeated here.

[0310] S602, the electronic device divides the hidden layer of the image classification model to be processed into at least one first layer set, wherein each first layer set includes at least one first model layer, and the first model layer is a hidden layer including a convolution operator, and the convolution operator is used for image feature analysis.

[0311] The principle of S602 is basically the same as that of S102. For the principle description and operation description of the same parts, please refer to the description in S102, which will not be repeated here. The differences between S602 and S102 are described as follows:

[0312] Image processing technology can be divided into three levels, namely: image processing in a narrow sense, image analysis, and image understanding. Image processing in a narrow sense refers to the processing and analysis of images at the pixel level, which can obtain relevant features at the pixel level of the image. For example, the pixel value, size, and resolution of each pixel of the image. Image analysis refers to the detection of the target object in the image and the local or overall feature analysis of the target object to obtain objective information of the target object. For example, in an image containing a cat, the cat's legs, tail, ears and other local features, as well as the cat's size and color and other overall features. Image understanding refers to further analyzing and understanding the features of the target object based on image analysis to obtain the corresponding abstract features. For example, in the above example, for an image containing a cat, image understanding can obtain some abstract features of the cat, such as the cat has 4 legs and 1 tail.

[0313] In the neural network model, the process of image classification is to gradually extract the image from pixel-level features to objective information features of the target object, and then to abstract features, and to judge the image type based on the final feature result vector. In practical applications, the convolution operator is mainly used to perform convolution kernel weight convolution operations on the input data (including the image itself and the data obtained after processing the image), thereby realizing the extraction of these image features. Therefore, in order to ensure the lightweight effect of the model in the embodiment of the present application, when dividing the original layer set, the embodiment of the present application needs to ensure that each original layer set contains at least one model layer with a convolution operator. At the same time, since the convolution operator already has a weight parameter, that is, each original layer set in the embodiment of the present application will inevitably meet the requirement of "but it must meet the requirement that a single original layer set contains at least one model layer with a weight parameter to ensure that the original layer set can be optimized for weight parameters" in S102, so in actual operation, this requirement of dividing the original layer set in S102 can no longer be considered.

[0314] S603, the electronic device performs lightweight processing on each first layer set to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing the weight parameters included in the convolution operator in the layer set, reducing the number of weight parameters included in the convolution operator in the layer set, and adjusting the network structure between model layers in the layer set.

[0315] Among them, the principle of S603 is basically the same as that of S103, and the relevant principle description and operation description can refer to the description in S103, which will not be repeated here. However, the difference from S103 is that, since the image classification model is mainly performed by the convolution operator for image feature extraction operations, the volume of the final model and the computing resources occupied during operation are closely related to the weight parameters in the convolution operator. Based on this principle, when quantizing the weight parameters of the original layer set, the embodiment of the present application can only quantize the weight parameters of the convolution operator therein, so as to reduce the workload of model lightweighting while ensuring the lightweight effect of the model. When reducing the weight parameters, it is also possible to only reduce the number of weight parameters in the convolution operator, such as reducing the size or number of convolution kernels in the convolution operator. As an optional embodiment of the present application, all weight parameters in the original layer set can also be quantized at the same time, which is the same as the operation of quantizing the weight parameters in S103.

[0316] S604, the electronic device uses the preset sample images to iteratively train the weight parameters of the second layer set, or iteratively trains the weight parameters of the first layer set and the second layer set, until the image classification model to be processed meets the preset convergence conditions, thereby obtaining a trained image classification model.

[0317] The principle of S604 is basically the same as that of S104. For the related principle description and operation description, please refer to the description in S104, which will not be repeated here.

[0318] It should be understood that Figure 1 The extended or refined embodiment scheme contents corresponding to the illustrated embodiment. Such as lightweighting of the original layer set, adjusting the ratio of training times and setting the activation probability of each layer set, and selecting and retaining the model layer set. It can also be combined with the embodiments of the present application for application without exceeding the protection scope of the present application. The principles and details of the specific application can be parameterized. Figure 1 The corresponding related descriptions in the illustrated embodiment are not repeated here.

[0319] At the same time, based on the embodiment of the present application, the image classification model trained by the embodiment of the present application can be set in the terminal device. Since the image classification model has been lightweight, it occupies less storage space of the terminal device. At the same time, when the actual model is run for image classification processing, more computing resources can be saved, thereby ensuring the normal use of AI applications in terminal devices.

[0320] It should be noted that, corresponding to the above three levels of image processing, if the pixel-level features, the target object objective information features, and then the abstract features are defined as the feature levels rising in sequence. The embodiment of the present application extracts features from the image in units of layer sets. At this time, the layer set closer to the input layer has lower-level image features, and the layer set closer to the output layer has higher-level image features, thereby realizing the sequential analysis and extraction of image features from low-level to high-level.

[0321] Since the extraction of features at different levels may have certain differences in the accuracy and operation relationship of weight parameters, the appropriate network structure corresponding to different layer sets in practical applications, as well as the appropriate accuracy of weight parameters corresponding to convolution operators, may have certain differences. In an embodiment of the present application, the network structure of the layer set is optimized, and the layer sets with different weight parameter accuracies are mixed for training. The weight parameter accuracy and network structure of the layer set suitable for the model can be selected to the greatest extent. During the training process, by adjusting the proportion of the activation probability of each layer set, it can be ensured that each layer set can be fully trained, thereby ensuring the accurate extraction of image features, and then ensuring the performance of the model for image classification. At the same time, the convergence judgment is made by combining the model performance index and the lightweight index, and the model structure search is performed with the layer set as the node. While ensuring the performance and lightweight degree of the model, the model layer set can be quickly selected, which greatly improves the efficiency of model training.

[0322] It should also be noted that for other models in the field of image processing, such as face verification models and scene recognition models, you can also refer to Figure 6 The method of the embodiment shown in the figure performs lightweight model training without exceeding the protection scope of the present application. Since the principles and operations of the training are basically the same, no further explanation is given here. Figure 6 Description of the illustrated embodiment.

[0323] Figure 7 The following is a flowchart of the implementation of the speech recognition model training method provided in the third embodiment of the present application, which is described in detail as follows:

[0324] S701, the electronic device obtains a preset sample voice and a voice recognition model to be processed.

[0325] The principle of S701 is basically the same as that of S101. For the related principle description and operation instructions, please refer to the instructions in S101 and will not be repeated here.

[0326] S702, the electronic device divides the hidden layer of the speech recognition model to be processed into at least one first layer set, wherein each first layer set contains at least one first model layer, the first model layer is a hidden layer containing a preset operator, the preset operator is a time series operator or a convolution operator, the time series operator is used to process time series data, and the convolution operator is used for speech feature analysis.

[0327] The principle of S702 is basically the same as that of S102 and S602. For the principle description and operation description of the same parts, please refer to the description of S102 and S602, which will not be repeated here. The differences between S702 and S602 are described as follows:

[0328] and Figure 6 Similar to the image processing in the illustrated embodiment, speech recognition is also a process of gradually extracting different features of speech and obtaining the final speech recognition result. However, the difference from image processing is that speech data is data with a certain temporal sequence, that is, there is a certain temporal relationship between the front and back of speech data. For example, according to the grammatical rules of Chinese, there is actually a certain order relationship between names, prepositions and verbs. When users speak Chinese, they often comply with the grammatical rules, which leads to a strong logical connection between different words. Therefore, when analyzing speech data, in order to improve the effect of the analysis, the temporal sequence of the speech data can be analyzed.

[0329] In order to realize the recognition of speech data with a time series relationship, the embodiment of the present application requires that each original layer set must contain at least one model layer of a convolution operator or a time series operator, and may also contain a convolution operator and a time series operator at the same time. Among them, whether to use a convolution operator or a time series operator for speech feature analysis can be determined by a technician according to actual needs, and there is no limitation here. At the same time, the embodiment of the present application does not limit the specific type of time series operator, and it needs to be determined according to the type of model actually used. For example, for a recurrent neural network (RNN) model, the time series operator can be an RNN operator in the model. For a gated recurrent unit (GRU) model, the time series operator can be a GRU operator in the model. For a long short-term memory network (LSTM) model, the time series operator can be an LSTM operator in the model. For a transformer model, the time series operator can be a transformer operator.

[0330] S703, the electronic device performs lightweight processing on each first layer set to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing the weight parameters included in the preset operators in the layer set, reducing the number of weight parameters included in the preset operators in the layer set, and adjusting the network structure between model layers in the layer set.

[0331] Among them, the principle of S703 is basically the same as that of S103, and the relevant principle description and operation description can refer to the description in S103, which will not be repeated here. However, the difference from S103 is that when the time series operator is selected for speech feature analysis. Since the speech recognition model is mainly performed by the time series operator for speech feature extraction operations, the volume of the final model and the computing resources occupied during operation are greatly related to the situation of the weight parameters in the time series operator. Based on this principle, when the weight parameters of the original layer set are quantized in the embodiment of the present application, only the weight parameters of the time series operator can be quantized to reduce the workload of the model lightweighting on the basis of ensuring the lightweight effect of the model. When reducing the weight parameters, only the number of weight parameters in the time series operator can be reduced, such as reducing the size or number of convolution kernels in the time series operator. As an optional embodiment of the present application, all weight parameters in the original layer set can also be quantized at the same time, which is the same as the operation of quantizing the weight parameters in S103.

[0332] S704, the electronic device uses the preset sample speech to iteratively train the weight parameters of the second layer set, or iteratively trains the weight parameters of the first layer set and the second layer set, until the speech recognition model to be processed meets the preset convergence conditions, thereby obtaining a trained speech recognition model.

[0333] The principle of S704 is basically the same as that of S104. For the related principle description and operation description, please refer to the description in S104, which will not be repeated here.

[0334] It should be understood that Figure 1 The extended or refined embodiment scheme contents corresponding to the illustrated embodiment. Such as lightweighting of the original layer set, adjusting the ratio of training times and setting the activation probability of each layer set, and selecting and retaining the model layer set. It can also be combined with the embodiments of the present application for application without exceeding the protection scope of the present application. The principles and details of the specific application can be parameterized. Figure 1 The corresponding related descriptions in the illustrated embodiment are not repeated here.

[0335] At the same time, based on the embodiment of the present application, the speech recognition model trained by the embodiment of the present application can be set in the terminal device. Since the speech recognition model has been lightweight, it occupies less storage space of the terminal device. At the same time, when the actual running model performs image classification processing, more computing resources can be saved, thereby ensuring the normal use of AI applications in terminal devices.

[0336] It should be noted that, since the extraction of different speech features may have certain differences in the accuracy and operational relationship of weight parameters, the appropriate network structure corresponding to different layer sets in actual applications, as well as the appropriate accuracy of weight parameters corresponding to preset operators, may have certain differences. In an embodiment of the present application, the network structure of the layer set is optimized, and the layer sets with different weight parameter accuracies are mixed for training. The weight parameter accuracy and network structure of the layer set suitable for the model can be selected to the greatest extent. During the training process, by adjusting the proportion of the activation probability of each layer set, it can be ensured that each layer set can be fully trained, thereby ensuring the accurate extraction of speech features, and then ensuring the performance of the model for speech recognition. At the same time, the convergence judgment is made in combination with the model performance index and the lightweight index, and the model structure search is performed with the layer set as the node. While ensuring the performance and lightweight degree of the model, the model layer set can be quickly selected, which greatly improves the efficiency of model training.

[0337] It should also be noted that for other models for time series data processing, such as natural language processing models, machine translation models, and video processing models, you can also refer to Figure 7The method of the embodiment shown in the figure performs lightweight model training without exceeding the protection scope of the present application. Since the principles and operations of the training are basically the same, no further explanation is given here. Figure 7 Description of the illustrated embodiment.

[0338] In the above-mentioned embodiments of the present application, lightweight training of the terminal model is achieved by dividing the layer set, lightweighting the layer set, performing mixed training of weight parameter precision on the layer set, and searching the model network structure with the layer set as the node. Based on the model obtained after lightweight training, the model can be stored locally in an electronic device or in other devices to perform the actual required data processing. Based on the beneficial effects analyzed in the above-mentioned embodiments, compared with the existing model lightweighting method, the embodiments of the present application have at least the following beneficial effects:

[0339] 1. Implement the model training and lightweight process together. A single lightweight training can obtain a usable model that meets the model performance requirements and lightweight requirements. Compared with training the model first and then trying to compress it multiple times, it is time-saving and more efficient.

[0340] 2. The lightweight solution of layer collection can be flexibly selected to adapt to the actual needs of more scenarios and achieve compatibility with multiple scenarios.

[0341] 3. The layer set can be divided by technical personnel according to actual needs, so that the lightweight training of the model in the embodiment of the present application is more flexible and controllable.

[0342] 4. Multiple performance indicators and lightweight indicators can be considered at the same time, so that the final model can better meet the actual needs of both performance and lightweight.

[0343] Corresponding to the above Figure 1 The neural network model training method of the illustrated embodiment, Figure 8 A structural block diagram of a neural network model training device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0344] Reference Figure 8 , the neural network model training device comprises:

[0345] The acquisition module 81 is used to acquire preset sample data and a neural network model to be processed.

[0346] The layer set division module 82 is used to divide the model layers of the neural network model to be processed into at least one first layer set.

[0347] The lightweight submodule 83 is used to perform lightweight processing on each first layer set to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing the weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between model layers in the layer set.

[0348] The iterative training module 84 is used to iteratively train the weight parameters of the second layer set using preset sample data, or iteratively train the weight parameters of the first layer set and the second layer set, until the neural network model to be processed meets the preset convergence conditions and the training is completed.

[0349] The process of each module in the neural network model processing device provided in the embodiment of the present application realizing its own function can be specifically referred to the aforementioned Figure 1 The description of the illustrated embodiment and other related method embodiments will not be repeated here.

[0350] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0351] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0352] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0353] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0354] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0355] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0356] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0357] The neural network model processing method, neural network model training, image classification model training and speech recognition model training methods provided in the embodiments of the present application can be applied to servers, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0358] For example, the electronic device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), customer premise equipment (CPE) and / or other devices for communicating on a wireless system and a next generation communication system, such as electronic devices in a 5G network or an electronic device in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0359] As an example but not limitation, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0360] Fig. 9 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: at least one processor 90 ( Fig. 9 Only one is shown in the figure), a memory 91, and a computer program 92 that can be run on the processor 90 is stored in the memory 91. When the processor 90 executes the computer program 92, the steps in the above-mentioned terminal model processing method, terminal model training, image classification model training and speech recognition model training method embodiments are implemented, such as Figure 1 Steps 101 to 104 shown, Figure 6 Steps 601 to 604 and Figure 7Alternatively, when the processor 90 executes the computer program 92, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 8 The functions of modules 81 to 84 are shown.

[0361] The electronic device 9 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 9 and does not constitute a limitation of the electronic device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include an input sending device, a network access device, a bus, etc.

[0362] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0363] In some embodiments, the memory 91 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. The memory 91 may also be an external storage device of the electronic device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. Further, the memory 91 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 91 may also be used to temporarily store data that has been sent or is to be sent.

[0364] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0365] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0366] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0367] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0368] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0369] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0370] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0371] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A terminal model processing method, characterized in that: include: Obtaining a terminal model to be processed, where the terminal model to be processed is an image classification model, a natural language processing model, a machine translation model, a speech recognition model, or a video processing model; Performing lightweight processing on the terminal model to be processed to obtain a processed terminal model; Acquire the data to be processed, and process the data to be processed by the processed terminal model to obtain corresponding processing results; The lightweight processing of the terminal model to be processed includes: Dividing the model layers of the terminal model to be processed into at least one first layer set; Lightweight processing is performed on each of the first layer sets to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between model layers in the layer set; Using preset sample data, iteratively training the weight parameters of the second layer set, or iteratively training the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence condition and the lightweight processing is completed; Before the iterative training of the weight parameters of the second layer set, the terminal model processing method further includes: obtaining a preset activation probability corresponding to each second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the second layer set; In the process of iteratively training the weight parameters of the second layer set, each training process includes: selecting the second layer set to be activated according to the corresponding preset activation probability, and updating the weight parameters of the selected second layer set; Before the iterative training of the weight parameters of the first layer set and the second layer set, the terminal model processing method further includes: obtaining a preset activation probability corresponding to each of the first layer set and the second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set; In the process of iteratively training the weight parameters of the first layer set and the second layer set, each training process includes: selecting the layer set to be activated from the first layer set and the second layer set according to the corresponding preset activation probability, and updating the weight parameters of the selected layer set.

2. The terminal model processing method according to claim 1, characterized in that: The reducing the number of weight parameters in the layer set includes: Reduce the size of the convolution kernels included in the layer set, or reduce the number of convolution kernels.

3. The terminal model processing method according to claim 1 or 2, characterized in that: The method of iteratively training the weight parameters of the second layer set by using the preset sample data, or iteratively training the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence condition and the lightweight processing is completed, further includes: If the terminal model to be processed does not meet the preset convergence condition after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, obtaining at least one performance indicator parameter of the terminal model to be processed after the training is completed; Using the preset sample data and the at least one performance indicator parameter, the weight parameters of the second layer set are iteratively trained, or the weight parameters of the first layer set and the second layer set are iteratively trained until the terminal model to be processed meets the preset convergence conditions and the lightweight processing is completed.

4. The terminal model processing method according to claim 1 or 2, characterized in that: The method of using the preset sample data to iteratively train the weight parameters of the second layer set until the terminal model to be processed meets the preset convergence condition and the lightweight processing is completed also includes: If the terminal model to be processed meets the preset convergence condition after iterative training, each of the second layer sets is used as a path node, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance indicator requirements, and the model composed of each path node in the layer set path is used as the processed terminal model.

5. The terminal model processing method according to claim 1 or 2, characterized in that: The method of using the preset sample data to iteratively train the weight parameters of the first layer set and the second layer set until the terminal model to be processed meets the preset convergence condition and the lightweight processing is completed also includes: If the terminal model to be processed meets the preset convergence condition after iterative training, the first layer set and the second layer set are used as path nodes, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance indicator requirements, and the model composed of each path node in the layer set path is used as the processed terminal model.

6. A terminal model training method, characterized in that: include: Acquire a terminal model to be processed, and divide a model layer of the terminal model to be processed into at least one first layer set, wherein the terminal model to be processed is an image classification model, a natural language processing model, a machine translation model, a speech recognition model, or a video processing model; Lightweight processing is performed on each of the first layer sets to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing weight parameters in the layer set, reducing the number of weight parameters in the layer set, and adjusting the network structure between model layers in the layer set; Using preset sample data, iteratively training the weight parameters of the second layer set, or iteratively training the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence condition, thereby obtaining a trained terminal model; Before the iterative training of the weight parameters of the second layer set, the terminal model training method further includes: obtaining a preset activation probability corresponding to each second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the second layer set; In the process of iteratively training the weight parameters of the second layer set, each training process includes: selecting the second layer set to be activated according to the corresponding preset activation probability, and updating the weight parameters of the selected second layer set; Before the iterative training of the weight parameters of the first layer set and the second layer set, the terminal model training method further includes: obtaining a preset activation probability corresponding to each of the first layer set and the second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set; In the process of iteratively training the weight parameters of the first layer set and the second layer set, each training process includes: selecting the layer set to be activated from the first layer set and the second layer set according to the corresponding preset activation probability, and updating the weight parameters of the selected layer set.

7. The terminal model training method according to claim 6, characterized in that: The reducing the number of weight parameters in the layer set includes: Reduce the size of the convolution kernels included in the layer set, or reduce the number of convolution kernels.

8. The terminal model training method according to claim 6 or 7, characterized in that: The method of iteratively training the weight parameters of the second layer set by using the preset sample data, or iteratively training the weight parameters of the first layer set and the second layer set, until the terminal model to be processed meets the preset convergence condition, further includes: If the terminal model to be processed does not meet the preset convergence condition after a single training of the second layer set is completed, or after a single training of the first layer set and the second layer set is completed, obtaining at least one performance indicator parameter of the terminal model to be processed after the training is completed; Using the preset sample data and the at least one performance indicator parameter, weight parameters of the second layer set are iteratively trained, or weight parameters of the first layer set and the second layer set are iteratively trained until the terminal model to be processed meets the preset convergence condition.

9. The terminal model training method according to claim 6 or 7, characterized in that: The method of iteratively training the weight parameters of the second layer set by using the preset sample data until the terminal model to be processed meets the preset convergence condition to obtain a trained terminal model further includes: If the terminal model to be processed meets the preset convergence condition after iterative training, each of the second layer sets is used as a path node, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance indicator requirements, and the model composed of each path node in the layer set path is used as the terminal model that has completed the training.

10. The terminal model training method according to claim 6 or 7, characterized in that: Using preset sample data, iteratively training the weight parameters of the first layer set and the second layer set until the terminal model to be processed meets the preset convergence condition, and a trained terminal model is obtained, further comprising: If the terminal model to be processed meets the preset convergence condition after iterative training, the first layer set and the second layer set are used as path nodes, and a layer set path search is performed on the terminal model to be processed to obtain a layer set path that meets the preset performance indicator requirements, and the model composed of each path node in the layer set path is used as the terminal model that has completed training.

11. A terminal model training method, characterized in that: include: Construct an initial model comprising a plurality of layer groups to be trained; wherein each of the layer groups to be trained comprises a plurality of layer sets, each of the layer sets comprises one or more model layers, and the layer sets comprised in each of the layer groups to be trained have at least one of the following differences: the number of bytes occupied by weight parameters in the layer set, the number of weight parameters in the layer set, and the network structure between model layers in the layer set; the initial model is an image classification model, a natural language processing model, a machine translation model, a speech recognition model, or a video processing model; Initializing the weight parameters of the initial model, and iteratively training the weight parameters of each layer set until the initial model meets the preset convergence condition, thereby obtaining a trained terminal model; Before performing iterative training of weight parameters on each of the layer sets, the terminal model training method further includes: Obtaining a preset activation probability corresponding to each of the layer sets, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameter in the layer set; In the process of iteratively training the weight parameters of each layer set, each training process includes: According to the corresponding preset activation probability, the layer set to be activated is selected, and the weight parameters of the selected layer set are updated.

12. A method for training an image classification model, characterized in that: include: Acquire an image classification model to be processed, and divide the hidden layer of the image classification model to be processed into at least one first layer set, wherein each of the first layer sets contains at least one first model layer, and the first model layer is a hidden layer containing a convolution operator, and the convolution operator is used for image feature analysis; Lightweight processing is performed on each of the first layer sets to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing weight parameters included in the convolution operator in the layer set, reducing the number of weight parameters included in the convolution operator in the layer set, and adjusting the network structure between model layers in the layer set; Using preset sample images, iteratively training the weight parameters of the second layer set, or iteratively training the weight parameters of the first layer set and the second layer set, until the image classification model to be processed meets a preset convergence condition, thereby obtaining a trained image classification model; Before the iterative training of the weight parameters of the second layer set, the image classification model training method further includes: obtaining a preset activation probability corresponding to each second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the second layer set; In the process of iteratively training the weight parameters of the second layer set, each training process includes: selecting the second layer set to be activated according to the corresponding preset activation probability, and updating the weight parameters of the selected second layer set; Before the iterative training of the weight parameters of the first layer set and the second layer set, the image classification model training method further includes: obtaining a preset activation probability corresponding to each of the first layer set and the second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set; In the process of iteratively training the weight parameters of the first layer set and the second layer set, each training process includes: selecting the layer set to be activated from the first layer set and the second layer set according to the corresponding preset activation probability, and updating the weight parameters of the selected layer set.

13. A speech recognition model training method, characterized in that: include: Acquire a speech recognition model to be processed, and divide the hidden layer of the speech recognition model to be processed into at least one first layer set; wherein each of the first layer sets contains at least one first model layer, and the first model layer is a hidden layer containing a preset operator, and the preset operator is a time series operator or a convolution operator, the time series operator is used to process time series data, and the convolution operator is used for speech feature analysis; Lightweight processing is performed on each of the first layer sets to obtain a corresponding second layer set, wherein the lightweight processing includes at least two of the following processing: quantizing weight parameters included in preset operators in the layer set, reducing the number of weight parameters included in preset operators in the layer set, and adjusting the network structure between model layers in the layer set; Using preset sample speech, iteratively training the weight parameters of the second layer set, or iteratively training the weight parameters of the first layer set and the second layer set, until the speech recognition model to be processed meets the preset convergence condition, thereby obtaining a trained speech recognition model; Before the iterative training of the weight parameters of the second layer set, the speech recognition model training method further includes: obtaining a preset activation probability corresponding to each second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the second layer set; In the process of iteratively training the weight parameters of the second layer set, each training process includes: selecting the second layer set to be activated according to the corresponding preset activation probability, and updating the weight parameters of the selected second layer set; Before the iterative training of the weight parameters of the first layer set and the second layer set, the speech recognition model training method further includes: obtaining a preset activation probability corresponding to each of the first layer set and the second layer set, wherein the preset activation probability is negatively correlated with the number of bytes occupied by the weight parameters in the first layer set and the second layer set; In the process of iteratively training the weight parameters of the first layer set and the second layer set, each training process includes: selecting the layer set to be activated from the first layer set and the second layer set according to the corresponding preset activation probability, and updating the weight parameters of the selected layer set.

14. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, it implements the steps of the terminal model processing method as described in any one of claims 1 to 5, or the steps of the terminal model training method as described in any one of claims 6 to 11, or the steps of the image classification model training method as described in claim 12, or the steps of the speech recognition model training method as described in claim 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the terminal model processing method as described in any one of claims 1 to 5, or the steps of the terminal model training method as described in any one of claims 6 to 11, or the steps of the image classification model training method as described in claim 12, or the steps of the speech recognition model training method as described in claim 13.

Images