A combination method, device and computer storage medium of image classification model

Abstract

The present application discloses a combination method, device and computer storage medium of an image classification model, which is used to improve the intelligence level of the image classification model. The present application method includes: obtaining at least two offline models for image classification that have been trained; fusing the first offline model and the second offline model into a combination model; inputting the first training data set and the second training data set into the combination model for training; calculating the loss value according to the probability value of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combination model; judging whether the loss value is greater than a preset threshold; if the loss value is less than the preset threshold, determining that the combination model is a target combination model that has been successfully fused; inputting the target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image; and obtaining the classification result of the target image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a combination method, device and computer storage medium of an image classification model. Background Art

[0002] Machine learning is a branch of artificial intelligence, which aims to enable machines to learn based on prior knowledge. Machine learning models continue to develop and are gradually applied to image classification, thereby realizing various intelligent applications based on image classification, such as character recognition, face recognition, object recognition, pedestrian detection, image retrieval, etc.

[0003] In the prior art, the collected sample data is used in the image classification model. Through repeated iterative training and learning, the loss function is converged through the shortest path, thereby determining the target image classification model. The larger the amount of sample data used to train the image classification model, the higher the accuracy of the target image classification model in classifying pictures. The essence of image classification model is to continuously learn the effective features of the image. Deep learning solves the problem of how to automatically learn effective features. It transforms the feature representation of the learning sample in the original space into a new feature space through layer-by-layer feature transformation.

[0004] However, the image classification model in the prior art often has a single function and can only classify one type of image. The existing image classification model has a low degree of intelligence. Summary of the invention

[0005] The present application provides a combination method, device and computer storage medium of an image classification model, which are used to improve the intelligence level of the image classification model.

[0006] The first aspect of the present application provides a combination method of an image classification model, comprising:

[0007] Obtain at least two trained offline models for image classification;

[0008] Merging the first offline model and the second offline model into a combined model;

[0009] Inputting a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model;

[0010] Calculating a loss value according to the probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combined model;

[0011] Determine whether the loss value is greater than a preset threshold;

[0012] If the loss value is less than a preset threshold, the combined model is determined to be a target combined model that has been successfully integrated, and the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0013] Inputting a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image;

[0014] Obtain a classification result of the target image.

[0015] Optionally, fusing the first offline model and the second offline model into a combined model includes:

[0016] Obtain a first model layer of a first offline model, and obtain a first parameter corresponding to the first model layer;

[0017] Obtain a second model layer of a second offline model, and obtain a second parameter corresponding to the second model layer;

[0018] Extracting a target model layer according to the first model layer and the second model layer, wherein the target model layer is a model layer shared by the first model layer and the second model layer;

[0019] Determine the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter;

[0020] The first offline model and the second offline model are fused into a combined model according to the target model layer and the target parameter.

[0021] Optionally, after determining whether the loss value is greater than a preset threshold, the combination method further includes:

[0022] If the loss value is greater than the preset threshold, the target parameters of the combined model are updated according to the loss value, and the first training data set and the second training data set are input into the combined model for training again.

[0023] Optionally, before inputting the first training data set and the second training data set into the combined model for training, the method further includes:

[0024] Obtaining a first training data set and a second training data set;

[0025] Performing a normalization process of mean subtraction on the first training data set and the second training data set;

[0026] The inputting the first training number set and the second training number set into the combined model for retraining comprises:

[0027] The normalized first training data set and the second training data set are input into the combined model for further training.

[0028] Optionally, before inputting the target image to be classified into the target combination model, the combination method further includes:

[0029] Preprocessing the target image to be classified so that the target image carries a target label, and the target label carries information of the first offline model or the second offline model;

[0030] The step of inputting the target image into the target combination model so that the target combination model calls the first classification function or the second classification function to classify the target image comprises:

[0031] The preprocessed target image is input into the target combination model, so that the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label.

[0032] Optionally, after determining that the combined model is a target combined model that has been successfully integrated, the combining method further includes:

[0033] The model size of the target combination model is compressed by a fusion multiplier algorithm.

[0034] Optionally, compressing the model size of the target combination model by using a fusion multiplier algorithm includes:

[0035] Obtaining nodes corresponding to two adjacent target model layers of the target combination model;

[0036] Multiplying the first node and the second node to obtain a third node, and deleting the first node and the second node, wherein the first node is a node of the first target model layer, and the second node is a node of the second target model layer;

[0037] The third node is determined as an intermediate output result, and the third node is used as initial input data of the next layer of neural network.

[0038] A second aspect of the present application provides a combination device of an image classification model, comprising:

[0039] A first acquisition unit, used to acquire at least two trained offline models for image classification;

[0040] A fusion unit, used for fusing the first offline model and the second offline model into a combined model;

[0041] A first input unit, used to input a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model;

[0042] a calculation unit, configured to calculate a loss value according to probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and a loss function of the combined model;

[0043] A judging unit, used to judge whether the loss value is greater than a preset threshold;

[0044] A determination unit, configured to determine that the combined model is a target combined model that has been successfully integrated if the loss value is less than a preset threshold, wherein the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0045] A second input unit, used for inputting a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image;

[0046] The second acquisition unit is used to acquire the classification result of the target image.

[0047] Optionally, the fusion unit is specifically used for:

[0048] Obtain a first model layer of a first offline model, and obtain a first parameter corresponding to the first model layer;

[0049] Obtain a second model layer of a second offline model, and obtain a second parameter corresponding to the second model layer;

[0050] Extracting a target model layer according to the first model layer and the second model layer, wherein the target model layer is a model layer shared by the first model layer and the second model layer;

[0051] Determine the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter;

[0052] The first offline model and the second offline model are fused into a combined model according to the target model layer and the target parameter.

[0053] Optionally, the combined model further includes:

[0054] An updating unit is used to update the target parameters of the combined model according to the loss value if the loss value is greater than the preset threshold, and input the first training data set and the second training data set into the combined model for training again.

[0055] Optionally, the combined model further includes:

[0056] A third acquisition unit, used to acquire the first training data set and the second training data set;

[0057] a normalization processing unit, configured to perform a normalization process of mean reduction on the first training data set and the second training data set;

[0058] The first input unit is specifically used for:

[0059] The normalized first training data set and the second training data set are input into the combined model for further training.

[0060] Optionally, the combined model further includes:

[0061] a preprocessing unit, configured to preprocess a target image to be classified so that the target image carries a target label, and the target label carries information of the first offline model or the second offline model;

[0062] The second input unit is specifically used for:

[0063] The preprocessed target image is input into the target combination model, so that the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label.

[0064] Optionally, the combined model further includes:

[0065] A compression unit is used to compress the model size of the target combination model through a fusion multiplier algorithm.

[0066] Optionally, the compression unit is specifically used for:

[0067] Obtaining nodes corresponding to two adjacent target model layers of the target combination model;

[0068] Multiplying the first node and the second node to obtain a third node, and deleting the first node and the second node, wherein the first node is a node of the first target model layer, and the second node is a node of the second target model layer;

[0069] The third node is determined as an intermediate output result, and the third node is used as initial input data of the next layer of neural network.

[0070] A third aspect of the present application provides a combined device of an image classification model, the device comprising:

[0071] Processor, memory, input-output unit, and bus;

[0072] The processor is connected to the memory, the input and output unit, and the bus;

[0073] The memory stores a program, and the processor calls the program to execute the first aspect and the combination method of any optional image classification model in the first aspect.

[0074] A fourth aspect of the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed on a computer, the program executes the first aspect and the combination method of any optional image classification model in the first aspect.

[0075] It can be seen from the above technical scheme that the present application has the following advantages: the terminal obtains at least two offline models of image classification that have been trained; the terminal fuses the first offline model and the second offline model into a combined model; the terminal inputs the first training data set and the second training data set into the combined model for training, the first training data set is the training data set of the first offline model, and the second training data set is the training data set of the second offline model, the terminal calculates the loss value according to the probability value of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combined model, and the terminal determines whether the loss value is greater than the preset threshold; if the loss value is less than the preset threshold, the terminal determines that the combined model is a target combined model that has been successfully integrated, and the target combined model includes the first image classification function of the first offline model and the second image classification function of the second offline model, and the terminal inputs the target image to be classified into the target combined model so that the target combined model calls the first image classification function or the second image classification function to classify the target image, and the terminal obtains the classification result of the target image. Through the above-mentioned combination method of image classification models, the combined target combined model has more than two types of image classification functions, can classify images of multiple categories, improves the diversity of classification of the target combined model, and thus improves the intelligence of the target classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1A schematic diagram of an embodiment of a combination method of an image classification model provided in this application;

[0078] Figure 2 A schematic flow chart of another embodiment of a combination method of an image classification model provided in the present application;

[0079] Figure 3 A schematic diagram of the structure of an embodiment of a combination device of an image classification model provided in this application;

[0080] Figure 4 A schematic structural diagram of another embodiment of a combination device of an image classification model provided by the present application;

[0081] Figure 5 A schematic structural diagram of an embodiment of a combination device of an image classification model provided in the present application. DETAILED DESCRIPTION

[0082] The present application provides a combination method, device and computer storage medium of an image classification model, which are used to improve the intelligence level of the image classification model.

[0083] It should be noted that the combination method of an image classification model provided in this application can be applied to a terminal or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as the execution subject for example.

[0084] See also Figure 1 , Figure 1 An embodiment of a combination method of an image classification model provided in the present application, the combination method comprising:

[0085] 101. The terminal obtains at least two trained offline models for image classification;

[0086] In this embodiment, the terminal obtains at least two trained image classification models, and the original neural network models of the at least two trained image classification models can be a VGG-16 convolutional neural network model. The network structure of the VGG-16 convolutional neural network model is 16 layers, including: conv3-64, conv3-64, maxpool, conv3-128, conv3-128, maxpool, conv3-256, conv3-256, conv3-256, maxpool, conv3-512, conv3-512, conv3-512, maxpool, conv3-512, conv3-512, conv3-512, maxpool, FC-4096, FC-4096, FC-1000, softmax. Convolutional neural network is a special deep forward neural network model, and its structure is generally composed of input layer, convolution layer, pooling layer, fully connected layer and output layer. The loss functions of the softmax layers of at least two trained offline models for image classification have converged.

[0087] 102. The terminal merges the first offline model and the second offline model into a combined model;

[0088] In this embodiment, fusing the first offline model and the second offline model into a combined model specifically includes: obtaining the first model layer of the first offline model, and obtaining the first parameter corresponding to the first model layer; obtaining the second offline model layer of the second offline model, and obtaining the second parameter corresponding to the second model layer; extracting the target model layer according to the first model layer and the second model layer, the target model layer being a model layer shared by the first model layer and the second model layer; determining the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter; and fusing the first offline model and the second offline model into a combined model according to the target model layer and the target parameters.

[0089] In this embodiment, the basic feature of machine learning image classification is to achieve the description of objects in the image from the bottom to the top by learning hierarchical feature expressions. When the first offline model and the second offline model classify the image to be classified, the features expressed by the feature vectors of the extracted image to be classified are becoming more and more abstract and high-level. For example, the first layer of feature extraction extracts the feature vectors of the line features of the sample image, the second layer of feature extraction extracts the feature vectors of the shape, texture and other detail features of the sample image, and the highest layer of feature extraction extracts the feature vectors of the semantic features of the sample image. For example, in face recognition, the feature vectors of the facial features are the feature vectors of the highest layer. Although the first offline model and the second offline model have classification functions for images of different categories, in the process of low-level feature learning, the loss values ​​of the low-level feature learning of the first offline model and the second offline model for the image to be classified will be similar or close in the parameter space. Similarity or closeness in this embodiment means that the absolute value of the difference between the first parameter and the second parameter is less than a preset threshold, and the preset threshold needs to be determined based on the experience or experiment of the tester. At this time, the average value of the similar or close first parameter and the second parameter is used as the target parameter corresponding to the target model layer, and the convolutional neural network of the low-level feature learning of the first offline model and the second offline model can be unified.

[0090] The convolutional neural network of the combined model can be constructed by arranging the target model layer shared by the first offline model and the second offline model, the target parameters corresponding to the target model layer, and the independent first model layer in the first offline model layer and the independent second model layer in the second offline model layer according to the preset arrangement order. The convolutional neural network of the fused combined model includes a pooling layer, a convolution layer, a fully connected layer, and a softmax layer. At this point, the combined model already has the low-level feature extraction function of the first offline model and the second offline model, and the combined model can quickly summarize the approximate content of the first category image and the second category image based on the low-level feature vector, such as contour, shape, texture, etc.

[0091] 103. The terminal inputs the first training data set and the second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model;

[0092] In this embodiment, the first training data set is input into the combined model for training, so that the combined model learns the extraction of high-level features of images in the first training data set through the first training data set, and the second training data set is input into the combined model for training, so that the combined model learns the extraction of high-level features of images in the second training data set through the second training data set. The combined model performs high-level feature extraction training through the first training data set and the second training data set to obtain high-level feature expressions of the first training data set and the second training data set. Through the first training data set and the second training data set, the combined model loads and fuses high-level feature vectors trained by different models, thereby completing the comprehensive extraction of first category image features and second category image features.

[0093] In this embodiment, the combined model uses a convolution kernel through a convolution layer to extract features of the first training data set and the second training data set and filters and enhances the extracted features. The purpose of convolution is to convolve the feature map output by the previous layer with the convolution kernel and output the feature value through an activation function. The activation function formula is as follows:

[0094]

[0095] Among them, f is the activation function Sigmod, t represents the number of layers of the network, represents the convolution kernel, * represents the convolution operation, Indicates the offset, P j Represents the input feature set.

[0096] In this embodiment, the pooling layer of the combined model uses average pooling. That is, a tensor is averaged into a 1×1×D tensor. The purpose of using the global average pooling layer is to reduce the dimension of N feature maps into a feature map of size 1×N, and then use the classification number class (for example: in this embodiment, it is set to 3, that is, male, female, age) 1×1 convolution kernels in the fully connected layer to convolve the 1×N feature map to obtain a 1×class vector. Taking the Xception network as an example, (nb_samples, 10, 10, 2048) uses the calculation amount after using the fully connected layer Softmax nb_samples×10×10×2048, while the calculation amount after using global average pooling is nb_samples×1×1×2048). Global average pooling makes the convolution structure simpler by strengthening the consistency between the feature map and the category, and uses global average pooling to directly average each activation map output by the convolution layer, reducing the size of the output data file, and to a certain extent reducing the problem of overfitting. Each of the first offline model and the second offline model used for fusion training is a large network structure, and overfitting is a problem that is likely to occur during training. Global average pooling does not require parameter optimization, so the purpose of setting this layer in the combined model is to more effectively avoid overfitting. Moreover, the global average pooling layer sums the spatial information, so the transformation of the input space is more stable, which allows the computing network to extract features from the image while also retaining the characteristics of the image's spatial information.

[0097] 104. The terminal calculates a loss value according to the probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combination model;

[0098] In this embodiment, the terminal calculates the loss value based on the first probability that the first training data set belongs to the first image category, the second probability that the second training data set belongs to the second image category, and the loss function of the combined model. In this embodiment, the loss value is mainly calculated by the cross entropy function and the probability value set. The following example describes the calculation of the loss value: the cross entropy function mainly characterizes the distance between the actual output (probability value) and the expected output (probability value), that is, the smaller the value of the cross entropy, the closer the two probability value distributions are. Assume that probability distribution p is the expected output, probability distribution q is the actual output, and H(p,q) is the cross entropy function, as follows:

[0099] H(p,q)=-∑(p(x)logq(x))

[0100] Assume that N=3, where N is the desired number.

[0101] The expected output is p=(1,0,0).

[0102] The actual output is q1 = (0.5, 0.4, 0.1), q2 = (0.8, 0.1, 0.1), then:

[0103] H(p,q1)=-(1*log0.5+0*log0.4+0*log0.1+0*log0.5+1*log0.9+1*log0.6)=0.3;

[0104] H(p,q2)=-(1*log0.8+0*log0.1+0*log0.1+0*log0.2+1*log0.9+1*log0.9)=0.19;

[0105] It can be seen that q2 is closer to p, and its cross entropy is also smaller, that is, the loss value is smaller.

[0106] 105. The terminal determines whether the loss value is greater than a preset threshold;

[0107] In this embodiment, the terminal determines whether the loss value of step 104 is greater than a preset threshold value, and when the loss value is not greater than the preset threshold value, the terminal executes step 106. For example, the preset threshold value is 0.15, and the loss value calculated in step 104 is 0.1. At this time, the terminal determines that the loss value is less than the preset threshold value, and the terminal executes step 106.

[0108] 106. If the loss value is less than a preset threshold, the terminal determines that the combined model is a target combined model that has been successfully integrated, and the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0109] In this embodiment, the loss value is less than the preset threshold value, which means that the accuracy of the terminal's feature extraction of the first type of image and the feature extraction of the second type of image has reached the preset accuracy. At this time, the classification accuracy of the combined model for the first training data set and the second training data set has reached the preset accuracy. At this time, the terminal determines that the combined model is a target combined model with successful fusion, and the target combined model includes the first image classification function of the first offline model and the second image classification function of the second offline model. For example: the first image classification function of the first offline model is to distinguish the age of the person in the image, and the second image classification function of the second offline model is to distinguish the gender of the person in the image. Then the combined model includes a classification function for distinguishing the age of the person in the image and a classification function for distinguishing the gender of the person in the image.

[0110] 107. The terminal inputs the target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image;

[0111] In this embodiment, for example: the terminal inputs the target image to be classified into the target combination model, the target image to be classified is the age and gender of the person in the classified target image, the first image classification function of the target combination model is the age of the person in the classified image, and the second classification function of the target image is the gender of the person in the classified image. At this time, the terminal calls the first image classification function to classify the age of the target image, and calls the second image classification function to classify the gender of the person in the target image.

[0112] 108. The terminal obtains the classification result of the target image.

[0113] In this embodiment, the terminal obtains the classification result of the target image according to the execution result of step 107. For example, the classification result of the target image is: the age of the person is 50 years old, and the gender of the person is male.

[0114] In this embodiment, the terminal obtains at least two offline models of image classification that have been trained; the terminal fuses the first offline model and the second offline model into a combined model; the terminal inputs the first training data set and the second training data set into the combined model for training, the first training data set is the training data set of the first offline model, and the second training data set is the training data set of the second offline model, the terminal calculates the loss value according to the probability value of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combined model, and the terminal determines whether the loss value is greater than a preset threshold; if the loss value is less than the preset threshold, the terminal determines that the combined model is a successfully fused target combined model, the target combined model includes the first image classification function of the first offline model and the second image classification function of the second offline model, the terminal inputs the target image to be classified into the target combined model, so that the target combined model calls the first image classification function or the second image classification function to classify the target image, and the terminal obtains the classification result of the target image. Through the above-mentioned combination method of image classification models, the combined target combined model has more than two types of image classification functions, can classify images of multiple categories, improves the diversity of classification of the target combined model, and thus improves the intelligence of the target classification model.

[0115] See also Figure 2 , Figure 2 An embodiment of a combination method of an image classification model provided in the present application, the combination method comprising:

[0116] 201. The terminal obtains at least two trained offline models for image classification;

[0117] 202. The terminal merges the first offline model and the second offline model into a combined model;

[0118] Steps 201 to 202 in this embodiment are similar to those in the previous Figure 1Steps 101 to 102 in the embodiment are similar and will not be described in detail here.

[0119] 203. The terminal obtains a first training data set and a second training data set, where the first training data set is a training data set of a first offline model, and the second training data set is a training data set of a second offline model;

[0120] In this embodiment, the first training data set and the second training data set are manually screened, and the first training data set and the second training data set are both selected from images of real life scenes in daily life, so that the first training data set and the second training data set have a more complex image background, and thus the combined model is trained by training the combined model with images with complex backgrounds, so that the combined model after training has stronger practicality.

[0121] 204. The terminal performs a normalization process of mean reduction on the first training data set and the second training data set;

[0122] In this embodiment, the terminal performs a normalization process of mean subtraction on the first training data set and the second training data set, that is, each channel of the training images in the first training data set and the second training data set is decentralized. The purpose of the normalization process is to remove the average brightness value of the training image. In most image classification scenarios, the image brightness is not of interest, but more image content needs to be obtained. Therefore, removing the pixel mean for each data point can weaken the difference between the image brightness, so that the combined model can extract the high-level features of the first training data set and the second training data set more quickly.

[0123] 205. The terminal inputs the normalized first training data set and the second training data set into the combined model for further training;

[0124] In this embodiment, the normalized first training data set and the second training data set are input into the combined model for further training in order to use the first training data set and the second training data set to adjust parameters and optimize the combined model, and to reduce the training time of the combined model.

[0125] 206. The terminal calculates a loss value according to the probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combination model;

[0126] Step 206 in this embodiment is similar to the above Figure 1 Step 104 in the embodiment is similar and will not be described in detail here.

[0127] 207. The terminal determines whether the loss value is greater than a preset threshold;

[0128] In this embodiment, the terminal determines whether the loss value is greater than a preset threshold value. If not, the terminal executes step 208; if so, the terminal executes step 209.

[0129] 208. If the loss value is greater than a preset threshold, the terminal updates the target parameters of the combined model according to the loss value, and inputs the first training data set and the second training data set into the combined model for training again;

[0130] In this embodiment, if the loss value is greater than a preset threshold, it means that the current combined model needs to be trained again. The first training data set and the second training data set are input into the updated combined model for training again. During the model training process, it is usually impossible to directly calculate the global optimal solution of the model loss function, but the gradient of each parameter in the neural convolutional network can be obtained through the loss function, and then the numerical optimization method of gradient descent iteration is used to reduce the loss function.

[0131] In this embodiment, To iteratively update the parameters in the neural convolutional network, where θ represents the parameters in the neural convolutional network, represents the gradient of the parameters.

[0132] In the real-time example, SDGM is used to calculate the combined model parameters, that is, θ = θ + Vt, where Vt represents the update amount of θ after each iteration, so that the model is closer to or reaches the optimal value and minimizes the time function. The specific calculation expression of Vt is as follows: Vt = η × dθ + V t-1 ×momentum. Where η represents the learning rate, that is, the degree of influence of the current gradient on the final update direction, dθ is the gradient of the updated parameters in the network relative to the objective function, and V t-1Represents the amount of the last update. The value of momentum is a random quantity between [0,1], which is used to indicate the degree of retention in the original update direction. The combined model will retain the degree of the original update direction during the training process, and will also use the training results of the current batch to fine-tune the update direction of the next round of iterations, so that the model is more stable during the training process and helps to accelerate the convergence of the model. During the training process of this combined model, the initial training gradient is relatively large, so the initial value of momentum is set to 0.45. After a period of training, when the gradient change is relatively reduced, it is changed to 0.9. In addition, since the learning rate has a great impact on the iterative update parameters of the gradient descent, a learning rate that is too small will lead to a slow convergence speed, and a learning rate that is too large will hinder convergence and cause the loss function to oscillate or even diverge at the minimum value. Therefore, the initial value of η is set to 0.0001. During the training process of the combined model, the loss or accuracy of the first training data set and the second training data set are detected. When the learning stagnates, the learning rate is reduced, otherwise, the learning rate is increased. The training process of a neural network is to forward propagate the input content through the network, and then backpropagate the training error to update the weights. In the above process, some neurons can be temporarily and randomly deleted, that is, when the network is trained forward, the activation value of a certain neuron stops responding with a certain probability P. This makes the training model not overly dependent on certain local features for image classification, thereby improving the generalization ability of the model.

[0133] 209. If the loss value is less than a preset threshold, the terminal determines that the combined model is a target combined model that has been successfully integrated, and the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0134] Step 209 in this embodiment is similar to the above Figure 1 Step 106 in the embodiment is similar and will not be described in detail here.

[0135] 210. The terminal compresses the model size of the target combination model by using a fusion multiplier algorithm;

[0136] In this embodiment, the terminal compresses the model size of the target combination model by the fusion multiplier algorithm as follows: obtain the nodes corresponding to the two adjacent target model layers of the target combination model; multiply the first node and the second node to obtain the third node, the first node is the node of the first target model layer, and the second node is the node of the second target model layer; determine the third node as the intermediate output result, and use the third node as the initial input data of the next layer of neural network. Because the calculation process of the neural network is the value transfer calculation between the nodes layer by layer in the neural network. Each layer in the neural network has several nodes, and several nodes in each layer are connected to each other for calculation, and then the calculation results are passed to the next layer of neural network as the input of the next layer of neural network. For example; the first layer of neural network has node A, the second layer of neural network has node B, and the third layer of neural network has node D, then the fusion multiplier algorithm is used to fuse the A node and the B node into node C, that is, C=A*B, and the node C is used as the input of the third layer of neural network, so that the node C and the node D are used to perform the value transfer calculation of the neural network. Through the above method, the number of nodes is effectively reduced, thereby reducing the size of the target combination model, and then reducing the hardware resources occupied by the target combination model.

[0137] 211. The terminal preprocesses the target image to be classified so that the target image carries a target label, and the target label carries information of the first offline model or the second offline model;

[0138] In this embodiment, before inputting the target image into the target combination model, the target image needs to be preprocessed, that is, the target image is labeled. For example, if the target image is a target image that needs to be classified using the first offline model function, the target image is labeled with the information of the first offline model, and then the target image is input into the target combination model.

[0139] 212. The terminal inputs the preprocessed target image into the compressed target combination model, so that the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label;

[0140] In this embodiment, the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label, thereby accelerating the image classification speed of the target combination model and improving the performance of the target combination model.

[0141] 213. The terminal obtains the classification result of the target image.

[0142] Step 213 in this embodiment is similar to the above Figure 1 Step 108 in the embodiment is similar and will not be described in detail here.

[0143] See also Figure 3, Figure 3 An embodiment of a combination device of an image classification model provided in the present application, the combination device comprises:

[0144] A first acquisition unit 301 is used to acquire at least two trained offline models for image classification;

[0145] A fusion unit 302, configured to fuse the first offline model and the second offline model into a combined model;

[0146] A first input unit 303, used to input a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model;

[0147] A calculation unit 304, configured to calculate a loss value according to probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and a loss function of the combination model;

[0148] A judging unit 305 is used to judge whether the loss value is greater than a preset threshold;

[0149] A determination unit 306, configured to determine that the combined model is a target combined model that has been successfully integrated if the loss value is less than a preset threshold, wherein the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0150] A second input unit 307, configured to input a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image;

[0151] The second acquisition unit 308 is used to acquire the classification result of the target image.

[0152] In this embodiment, the first offline model and the second offline model are fused into a combined model through the fusion unit 302, and the first training data set and the second training data set are input into the combined model for training through the first input unit 303, so that the target combined model after the training has more than two types of image classification functions and can classify images of multiple categories, thereby improving the diversity of classification of the target combined model and further improving the intelligence of the target classification model.

[0153] The following is a detailed description of the combination device of an image classification model provided by this application. Figure 4 , Figure 4Another embodiment of a combined device of an image classification model provided in the present application includes:

[0154] A first acquisition unit 401 is used to acquire at least two trained offline models for image classification;

[0155] A fusion unit 402, configured to fuse the first offline model and the second offline model into a combined model;

[0156] A first input unit 403 is used to input a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model;

[0157] A calculation unit 404, configured to calculate a loss value according to probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and a loss function of the combination model;

[0158] A judging unit 405 is used to judge whether the loss value is greater than a preset threshold;

[0159] A determination unit 406, configured to determine that the combined model is a target combined model that has been successfully integrated if the loss value is less than a preset threshold, wherein the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model;

[0160] A second input unit 407, configured to input a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image;

[0161] The second acquisition unit 408 is used to acquire the classification result of the target image.

[0162] Optionally, the fusion unit 402 is specifically configured to:

[0163] Obtain a first model layer of a first offline model, and obtain a first parameter corresponding to the first model layer;

[0164] Obtain a second model layer of a second offline model, and obtain a second parameter corresponding to the second model layer;

[0165] Extracting a target model layer according to the first model layer and the second model layer, wherein the target model layer is a model layer shared by the first model layer and the second model layer;

[0166] Determine the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter;

[0167] The first offline model and the second offline model are fused into a combined model according to the target model layer and the target parameter.

[0168] Optionally, the combined model further includes:

[0169] The updating unit 409 is used to update the target parameters of the combined model according to the loss value if the loss value is greater than the preset threshold, and input the first training data set and the second training data set into the combined model for training again.

[0170] Optionally, the combined model further includes:

[0171] A third acquisition unit 410 is used to acquire a first training data set and a second training data set;

[0172] A normalization processing unit 411, configured to perform a normalization process of mean reduction on the first training data set and the second training data set;

[0173] The first input unit 403 is specifically used for:

[0174] The normalized first training data set and the second training data set are input into the combined model for further training.

[0175] Optionally, the combined model further includes:

[0176] A preprocessing unit 412, configured to preprocess the target image to be classified so that the target image carries a target label, and the target label carries information of the first offline model or the second offline model;

[0177] The second input unit 407 is specifically used for:

[0178] The preprocessed target image is input into the target combination model, so that the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label.

[0179] Optionally, the combined model further includes:

[0180] The compression unit 413 is used to compress the model size of the target combination model by using a fusion multiplier algorithm.

[0181] Optionally, the compression unit 413 is specifically configured to:

[0182] Obtaining nodes corresponding to two adjacent target model layers of the target combination model;

[0183] Multiplying the first node and the second node to obtain a third node, and deleting the first node and the second node, wherein the first node is a node of the first target model layer, and the second node is a node of the second target model layer;

[0184] The third node is determined as an intermediate output result, and the third node is used as initial input data of the next layer of neural network.

[0185] In this embodiment, the functions of each unit are the same as those described above. Figure 2 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0186] This application also provides a combination device of an image classification model, see Figure 5 , Figure 5 An embodiment of a combination device of an image classification model provided in the present application, the combination device comprises:

[0187] Processor 501, memory 502, input and output unit 503, bus 504;

[0188] The processor 501 is connected to the memory 502, the input and output unit 503 and the bus 504;

[0189] The memory 502 stores a program, and the processor 501 calls the program to execute any one of the above image classification model combination methods.

[0190] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any one of the above-mentioned combination methods of image classification models.

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0193] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A combination method of image classification models, It is characterized in that The combined method comprises: Obtain at least two trained offline models for image classification; Merging the first offline model and the second offline model into a combined model; Inputting a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model; Calculating a loss value according to the probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and the loss function of the combined model; Determine whether the loss value is greater than a preset threshold; If the loss value is less than a preset threshold, the combined model is determined to be a target combined model that has been successfully integrated, and the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model; Inputting a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image; Obtaining a classification result of the target image; The fusing of the first offline model and the second offline model into a combined model comprises: Obtain a first model layer of a first offline model, and obtain a first parameter corresponding to the first model layer; Obtain a second model layer of a second offline model, and obtain a second parameter corresponding to the second model layer; Extracting a target model layer according to the first model layer and the second model layer, wherein the target model layer is a model layer shared by the first model layer and the second model layer; Determine the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter; fusing the first offline model and the second offline model into a combined model according to the target model layer and the target parameter; After determining that the combined model is a target combined model that has been successfully integrated, the combining method further includes: Compressing the model size of the target combination model by fusion multiplier algorithm; The method of compressing the model size of the target combination model by using a fusion multiplier algorithm includes: Obtaining nodes corresponding to two adjacent target model layers of the target combination model; Multiplying the first node and the second node to obtain a third node, and deleting the first node and the second node, wherein the first node is a node of the first target model layer, and the second node is a node of the second target model layer; The third node is determined as an intermediate output result, and the third node is used as initial input data of the next layer of neural network.

2. The combined method according to claim 1, It is characterized in that After determining whether the loss value is greater than a preset threshold, the combination method further includes: If the loss value is greater than the preset threshold, the target parameters of the combined model are updated according to the loss value, and the first training data set and the second training data set are input into the combined model for training again.

3. The combined method according to any one of claims 1 to 2, It is characterized in that Before inputting the first training data set and the second training data set into the combined model for training, the method further includes: Obtaining a first training data set and a second training data set; Performing a normalization process of mean subtraction on the first training data set and the second training data set; The inputting the first training number set and the second training number set into the combined model for retraining comprises: The normalized first training data set and the second training data set are input into the combined model for further training.

4. The combined method according to any one of claims 1 to 2, It is characterized in that Before inputting the target image to be classified into the target combination model, the combination method further includes: Preprocessing the target image to be classified so that the target image carries a target label, and the target label carries information of the first offline model or the second offline model; The step of inputting the target image into the target combination model so that the target combination model calls the first classification function or the second classification function to classify the target image comprises: The preprocessed target image is input into the target combination model, so that the target combination model determines to call the first image classification function or the second image classification function to classify the target image according to the target label.

5. A combined device of an image classification model, It is characterized in that The device comprises: A first acquisition unit, used to acquire at least two trained offline models for image classification; A fusion unit, used for fusing the first offline model and the second offline model into a combined model; A first input unit, used to input a first training data set and a second training data set into the combined model for training, wherein the first training data set is a training data set of the first offline model, and the second training data set is a training data set of the second offline model; a calculation unit, configured to calculate a loss value according to probability values ​​of the first training data set and the second training data set belonging to the first image category and the second image category and a loss function of the combined model; A judging unit, used to judge whether the loss value is greater than a preset threshold; A determination unit, configured to determine that the combined model is a target combined model that has been successfully integrated if the loss value is less than a preset threshold, wherein the target combined model includes a first image classification function of the first offline model and a second image classification function of the second offline model; A second input unit, used for inputting a target image to be classified into the target combination model, so that the target combination model calls the first image classification function or the second image classification function to classify the target image; A second acquisition unit, used to acquire the classification result of the target image; The fusion unit is specifically used for: Obtain a first model layer of a first offline model, and obtain a first parameter corresponding to the first model layer; Obtain a second model layer of a second offline model, and obtain a second parameter corresponding to the second model layer; Extracting a target model layer according to the first model layer and the second model layer, wherein the target model layer is a model layer shared by the first model layer and the second model layer; Determine the target parameters corresponding to each layer of the target model layer according to the first parameter and the second parameter; fusing the first offline model and the second offline model into a combined model according to the target model layer and the target parameter; The combined model also includes: A compression unit, used for compressing the model size of the target combination model by using a fusion multiplier algorithm; The compression unit is specifically used for: Obtaining nodes corresponding to two adjacent target model layers of the target combination model; Multiplying the first node and the second node to obtain a third node, and deleting the first node and the second node, wherein the first node is a node of the first target model layer, and the second node is a node of the second target model layer; The third node is determined as an intermediate output result, and the third node is used as initial input data of the next layer of neural network.

6. A model assembly device, It is characterized in that The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 4.

Images