A model training method and device, equipment and a storage medium thereof
By dividing the feature vectors of the small model into two parts, inheritance and exploration, and using the loss value to train the large model, the problem of slow training speed of large models is solved, achieving efficient training and improved accuracy.
Patent Information
- Application Number
- CN202011240426.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-09
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2040-11-09
AI Technical Summary
How to combine the advantages of large and small models to achieve efficient model training, especially to improve the training speed and accuracy of large models.
By performing knowledge distillation from a small model to a large model, the feature vector of the small model is divided into two parts: inheritance and exploration. The first loss value and the second loss value are used for training respectively, so as to achieve fast and efficient training of the large model.
It improves the training speed and accuracy of large models, enables the effective transfer of knowledge from small models and the feature exploration of large models, and enhances the overall performance of the models.
Smart Images
Figure CN114494758B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present application relate to the technical field of computer, and in particular, to a model training method and device, and a storage medium thereof. BACKGROUND
[0002] With the development of deep learning, various neural network models are widely applied in various fields, such as image detection, text recognition, speech processing, etc. At present, in terms of scale, neural network models can be divided into large models (also referred to as complex models) and small models (also referred to as simple models). Compared with small models, large models have more complex structures and involve more intermediate layers. In the process of model training, although large models have strong expression ability, due to their high model complexity, more parameters are used, and the dimension of features is also high, so the training speed and iteration speed are slow. Small models are fast in training and iteration.
[0003] Therefore, how to combine the advantages of the two types of models to achieve efficient training of the model has become a problem to be solved. SUMMARY
[0004] Therefore, embodiments of the present application provide a model training method, device and storage medium to at least partially solve the above problems.
[0005] In a first aspect, a model training method is provided, which comprises:
[0006] obtaining a first model feature vector output by a first model after processing an input sample through an implicit layer, and a second model feature vector output by a second model after processing the input sample through an implicit layer, wherein the scale of the first model is smaller than that of the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector;
[0007] respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension;
[0008] obtaining a first loss value according to the first intermediate vector and the second intermediate vector, and a second loss value according to the first intermediate vector and the third intermediate vector;
[0009] training the second model according to at least the first loss value and the second loss value.
[0010] In a second aspect, an image classification method is also provided, which comprises:
[0011] determine an image classification model for image classification of a target image to be classified, wherein the image classification model comprises an input layer, a hidden layer and an output layer; receive an image data vector of the target image through the input layer of the image classification model;
[0012] perform image feature extraction according to the image data vector through the hidden layer of the image classification model, and obtain corresponding image features, wherein model parameters of the hidden layer are obtained through joint training of the image classification model and an auxiliary training model, the auxiliary training model has a smaller scale than the image classification model, and the joint training comprises: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through the hidden layer, and a second model feature vector output by the image classification model after processing the input sample through the hidden layer, wherein the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into first, second and third intermediate vectors of a preset dimension respectively; obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector; training the image classification model according to at least the first loss value and the second loss value, and determining model parameters of the hidden layer of the image classification model according to a training result;
[0013] output image category data corresponding to the image features through the output layer of the image classification model, wherein the image category data is used to indicate probabilities of the target image corresponding to different image categories.
[0014] A third aspect of the embodiments of the present application also provides a target object detection method, comprising:
[0015] determine a target object detection model for target object detection of a target image, wherein the target object detection model comprises an input layer, a hidden layer and an output layer;
[0016] receive an image data vector of the target image through the input layer of the target object detection model;
[0017] The image feature is extracted from the image data vector through a hidden layer of the target object detection model, and a corresponding image feature is obtained, wherein model parameters of the hidden layer are obtained through joint training of the target object detection model and an auxiliary training model, the auxiliary training model has a smaller scale than the target object detection model, and the joint training includes: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through a hidden layer, and a second model feature vector output by the target object detection model after processing the input sample through the hidden layer, wherein the second model feature vector includes a first sub-feature vector and a second sub-feature vector; the first model feature vector, the first sub-feature vector, and the second sub-feature vector are respectively converted into a first intermediate vector, a second intermediate vector, and a third intermediate vector of a preset dimension; a first loss value is obtained according to the first intermediate vector and the second intermediate vector; and a second loss value is obtained according to the first intermediate vector and the third intermediate vector; and the target object detection model is trained according to at least the first loss value and the second loss value, and model parameters of the hidden layer of the target object detection model are determined according to a training result.
[0018] The target object data corresponding to the image feature is output through an output layer of the target object detection model.
[0019] In a fourth aspect of the embodiments of the present application, another method for training a model is provided, and the method includes:
[0020] A model training request sent by a client is accepted, and an input sample and a first model and a second model used for training are obtained according to the model training request;
[0021] A first model feature vector output by the first model after processing the input sample through a hidden layer, and a second model feature vector output by the second model after processing the input sample through the hidden layer are obtained, wherein the first model has a smaller scale than the second model, and the second model feature vector includes a first sub-feature vector and a second sub-feature vector;
[0022] The first model feature vector, the first sub-feature vector, and the second sub-feature vector are respectively converted into a first intermediate vector, a second intermediate vector, and a third intermediate vector of a preset dimension;
[0023] A first loss value is obtained according to the first intermediate vector and the second intermediate vector; and a second loss value is obtained according to the first intermediate vector and the third intermediate vector;
[0024] The second model is trained according to at least the first loss value and the second loss value.
[0025] In a fifth aspect, the present application provides a device for training a model, the device comprising:
[0026] a first model feature vector output by the first model after processing the input sample through the hidden layer, and a second model feature vector output by the second model after processing the input sample through the hidden layer, wherein the first model has a smaller scale than the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector;
[0027] a conversion module configured to convert the first model feature vector, the first sub-feature vector and the second sub-feature vector into first, second and third intermediate vectors of a preset dimension, respectively;
[0028] a loss value determination module configured to obtain a first loss value according to the first and second intermediate vectors, and obtain a second loss value according to the first and third intermediate vectors;
[0029] a training module configured to train the second model according to at least the first and second loss values.
[0030] In a sixth aspect, the present application provides an electronic device, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface being in communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the training method as described above.
[0031] In a seventh aspect, the present application provides a computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the training method as described above.
[0032] According to the scheme provided in the embodiments of the present application, the first model feature vector generated by the first model with a smaller scale in the hidden layer and the second model feature vector generated by the second model with a larger scale in the hidden layer are determined, so as to map the first model feature vector and the second model feature vector to the same dimensional vector space, and divide the second model feature vector into a first sub-feature vector and a second sub-feature vector. One branch, such as the first loss value, is used to represent the first sub-feature vector for inheriting the knowledge learned by the first model, and the other branch, such as the second loss value, is used to represent the second sub-feature vector for exploring the knowledge not learned by the first model. Then, the second model is trained based on the first loss value and the second loss value, so as to achieve the goal of increasing the scale of the second model with a smaller scale of the first model, and improve the training speed and accuracy of the second model. BRIEF DESCRIPTION OF DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art based on these drawings.
[0034] Figure 1 The architecture schematic diagram of the system related to the embodiments of the present application;
[0035] Figure 2 The flow schematic diagram of the model training method provided by the embodiments of the present application;
[0036] Figure 3 The flow schematic diagram of the image classification method provided by the embodiments of the present application;
[0037] Figure 4 The flow schematic diagram of the target object detection method provided by the embodiments of the present application;
[0038] Figure 5 The flow schematic diagram of another model training method provided by the embodiments of the present application;
[0039] Figure 6 The architecture schematic diagram of the SaaS platform for model training provided by the embodiments of the present application;
[0040] Figure 7 The schematic diagram of the model training device provided by the embodiments of the present application;
[0041] Figure 8 The schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION
[0042] In order to make the person skilled in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art should belong to the scope of protection of the embodiments of the present application.
[0043] In model training, knowledge distillation has been widely used. Knowledge distillation is to extract the feature representation learned in a model and pass it to another similar function model, thereby assisting the training of another model. However, at present, the features learned by a large-scale model are usually passed to a small-scale model.
[0044] Based on this, the embodiment of the present application provides a scheme of knowledge distillation from a small model to a large model to improve the training efficiency and accuracy of the large model. As shown in the following formula (1), the knowledge distillation is performed on the basis of the first model and the second model. Figure 1 Figure 1 The following is an architecture schematic diagram of the system involved in the embodiment of the present application. In the schematic diagram, the first model is a small model with a small scale, and the second model is a large model with a large scale.
[0045] The first aspect of the embodiment of the present application is shown in the following formula (1), wherein the first model and the second model are neural network models. Figure 2 Figure 2 The following is a flow schematic diagram of a model training method provided by the embodiment of the present application, which includes the following steps.
[0046] S201, obtaining a first model feature vector output by the first model after processing an input sample through an implicit layer, and a second model feature vector output by the second model after processing the input sample through an implicit layer.
[0047] The neural network model can include multiple implicit layers, that is, the first model and the second model can both include multiple implicit layers. The implicit layer used for knowledge extraction can be any one of the multiple implicit layers. Alternatively, because the feature vector of the last implicit layer carries more sufficient feature information, in the embodiment, the feature vectors output by the last implicit layers of the first model and the second model are obtained. However, it is not limited to this, and the feature vectors output by other non-last implicit layers are also applicable to the scheme of the embodiment of the present application.
[0048] In addition, in the embodiment of the present application, the first model and the second model can be models with similar structures but different scales, including but not limited to: a model used for image recognition, a model used for target detection, a model used for text recognition, a model used for speech recognition, a model used for object classification, and any appropriate neural network model, and the embodiment of the present application does not limit this.
[0049] The implicit layers of the first model and the second model obtain the first model feature vector and the second model feature vector processed by the input sample, which are both multi-dimensional vectors. Because the scale of the first model is smaller than that of the second model, the dimension of the first model feature vector is usually smaller than that of the second model feature vector. For example, the first model feature vector can be 100-dimensional, and the second model feature vector can be 200-dimensional.
[0050] The scheme provided in the embodiments of the present application intends to extract the features of the input sample processed by the first model and transfer the features to the second model, so that the second model can achieve a fast and efficient training result. However, the features output by the first model are limited and may not cover all the effective information required by the second model. Therefore, in the embodiments of the present application, the feature vector of the second model is divided into two parts, namely a first sub-feature vector and a second sub-feature vector. The first sub-feature vector is used to inherit the hidden features of the first model, and the second sub-feature vector is used to explore new hidden features. Based on this, the first sub-feature vector needs to be as close as possible (such as high similarity) to the feature vector of the first model, and the second sub-feature vector needs to be as far away as possible (such as low similarity) from the feature vector of the first model. However, the above is only an exemplary description, and in actual application, the first sub-feature vector can also be made to be far away from the feature vector of the first model, and the second sub-feature vector can be made to be close to the feature vector of the first model. The former is taken as an example for description in the embodiments of the present application.
[0051] The specific division method can be determined according to the actual situation. For example, the division can be random. The division can also be of a specified form according to the size difference between the first model and the second model. If the size of the second model is much larger than that of the first model, it is more important to explore new features, and in this case, the second sub-feature vector with a higher dimension can be divided. If the size of the second model is slightly larger than that of the first model, it is more important to inherit the features of the first model, and in this case, the first sub-feature vector with a higher dimension can be divided.
[0052] S203, respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into first intermediate vectors, second intermediate vectors and third intermediate vectors of a preset dimension.
[0053] As described above, since the dimensions of the first model feature vector, the first sub-feature vector and the second sub-feature vector are different, the similarity cannot be compared directly. Therefore, it is necessary to first map them to the same dimension vector space.
[0054] In a feasible manner, converting the first model feature vector into the first intermediate vector of the preset dimension can be implemented as follows: mapping the initial first model feature vector to a vector space of a specified dimension to generate a to-be-determined intermediate vector of the specified dimension; regenerating a reconstructed first model feature vector according to the to-be-determined intermediate vector; and when the difference between the reconstructed first model feature vector and the initial first model feature vector satisfies a preset condition, determining that the intermediate vector is the first intermediate vector. The preset condition can be implemented as follows: the distance between the reconstructed first model feature vector and the initial first model feature vector is less than a distance preset value, or the included angle is less than an included angle preset value.
[0055] Specifically, let the first model feature vector be fs The first intermediate vector is F s The second intermediate vector is F inh The third intermediate vector is F exp Then, f can first be based on a predefined autoencoder. s Mapping to a vector space of a specified dimension yields the intermediate vector F to be determined. s The intermediate vector F to be determined here. s The dimension of the intermediate vector F is usually smaller than the dimension of the first model feature vector and the second model feature vector, for example, the dimension of the intermediate vector F to be determined. s The dimension is 50.
[0056] This allows us to determine the intermediate vector F. s Reconstruction, thus obtaining the first reconstructed model feature vector R(f) s In this process of first compressing the dimensions and then reconstructing them, although R(f) s ) and f s The dimensions are the same, but some information has been lost. Therefore, in order to make R(f) s ) and f s Sufficient proximity can be determined by calculating the differences between the two.
[0057] Because R(f) s ) and f s They exist in the same vector space and have the same dimensions. Therefore, the differences here can include R(f) s ) and f s The spatial distance or the angle between the two. For example, let the reconstruction error of the feature vector of the first model be Lrec=||fs-R(fs)|| 2 By changing the encoding method, different reconstruction errors are obtained. When the reconstruction error is less than a preset value, R(f) can be confirmed. s ) and f s It's close enough. In other words, not much information was lost during the dimensionality transformation; therefore, the intermediate vector F at this point is... s It can represent the feature vector f of the first model s Therefore, F at this time s It is determined to be the first intermediate vector.
[0058] For the second intermediate vector F inh and the third intermediate vector F exp The second intermediate vector F can also be obtained using the method described above. inh That is, the first sub-feature vector, which represents the latent features of the first model, and the third intermediate vector F. expThe season represents a second sub-feature vector for exploring features irrelevant to the first model.
[0059] S205, obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector.
[0060] Specifically, for the second intermediate vector F inh , it should be as close as possible to the first intermediate vector F s during training. s In order to make the first intermediate vector F inh and the second intermediate vector F inh approach each other, the second intermediate vector F inh ,
[0061] That is, first, the second intermediate vector F inh and the first intermediate vector F s are normalized respectively. Wherein ||F inh |2 is the length of the second intermediate vector F inh , and ||F s ||2 is the length of the first intermediate vector, so is the normalized value of the second intermediate vector, is the normalized value of the first intermediate vector, by normalizing two vectors of the same dimension respectively, that is, the loss function L inh can be calculated by the difference between the two (i.e. the Euclidean distance between the two normalized vectors), and the loss function L inh makes the first intermediate vector F s and the second intermediate vector F inh approach each other during training, so as to inherit the features of the first model. That is, the first intermediate vector and the second intermediate vector are normalized to obtain a normalized first intermediate vector and a normalized second intermediate vector; the distance between the normalized first intermediate vector and the normalized second intermediate vector is determined as a first loss value.
[0062] For the third intermediate vector F exp for exploring new implicit features, it should be as far as possible from the first intermediate vector F s during training. In order to make the third intermediate vector F exp and the first intermediate vector F s far away from each other, the third intermediate vector F exp can be constrained by the following loss function L exp , that is:
[0063]
[0064] First, the third intermediate vector F exp and the first intermediate vector F s are normalized respectively. Wherein ||F exp ||2is the length of the third intermediate vector F exp , and ||F s ||2is the length of the first intermediate vector, so is the normalized value of the third intermediate vector, is the normalized value of the first intermediate vector, by normalizing two vectors of the same dimension respectively, that is, the loss function L exp can be calculated by the inverse of the difference between the two (i.e. the Euclidean distance of the two normalized vectors) exp In the process of training, the first intermediate vector F s and the third intermediate vector F exp are constantly away from each other, so as to explore new features. That is, the first intermediate vector and the third intermediate vector are normalized respectively to obtain the normalized first intermediate vector and the normalized third intermediate vector; the inverse of the distance between the normalized first intermediate vector and the normalized third intermediate vector is determined as the second loss value.
[0065] S207, at least according to the first loss value and the second loss value, the second model is trained.
[0066] Specifically, in the process of training the second model, the second model itself has a basic loss value L goal for the input sample (i.e. the difference between the predicted value of the second model for the input sample and the true value of the input sample), therefore, the sum of the first loss value, the second loss value and the basic loss value can be used as the overall loss value of the second model, and the training of the second model is based on the overall loss. The calculation of the basic loss value can be obtained by the loss function of the second model itself. When the second model is used for image classification task, L goal may be softmax loss; when the second model is used for target detection task, L goal may be a candidate box loss function or a class loss function.
[0067] For example, the weights of the first loss value and the second loss value can be determined in advance respectively, that is, the inheritance weight λ inh and the exploration weight λ exp corresponding to the first loss value, so that the sum of the weights of the three loss values L=L goal +λinh L inh +λ exp L exp as the overall loss value.
[0068] According to the scheme provided in the embodiments of the present application, the first model feature vector generated by the first model with a smaller scale in the hidden layer and the second model feature vector generated by the second model with a larger scale in the hidden layer are determined, so as to map the first model feature vector and the second model feature vector to the same dimension vector space, and divide the second model feature vector into a first sub-feature vector and a second sub-feature vector, and use the first loss value to represent the first sub-feature vector for inheriting the knowledge learned by the first model, and use the second loss value to represent the second sub-feature vector for exploring the knowledge not learned by the first model, and then train the second model based on the first loss value and the second loss value, so as to complete the goal of the first model with a smaller scale improving the second model with a larger scale, and improve the training speed and accuracy of the second model.
[0069] In an embodiment, the first model can be a model that has been trained in advance, at this time, the first model feature vector and the first intermediate vector are fixed and unchangeable, and can be used as a static reference. The second model adjusts the second model feature vector according to the overall loss value during the training process, so as to realize the inheritance of part of the features by the first model and the exploration of new features by the other part.
[0070] In an embodiment, the first model can also be trained synchronously with the second model. The synchronous training refers to the same number of iterations of the two models. In each iteration process, since the first model has a small scale, the calculation speed is high, and therefore the first model feature vector can always be obtained first. Then, the second model can use the first model feature vector in the current iteration as a reference for iteration to respectively inherit and explore the features, so as to obtain the corresponding first loss value and second loss value, and train the second model according to the first loss value and the second loss value.
[0071] In an embodiment, the selected hidden layer can be a fully connected layer, especially a fully connected layer used for result recognition. Selecting this hidden layer can more effectively inherit and explore the features of the first model.
[0072] The second aspect of the embodiments of the present application also provides an image classification method, as shown in Figure 3 The flowchart of the image classification method provided by the embodiments of the present application comprises: Figure 3 S301, determining an image classification model used for image classification of a target image to be classified, wherein the image classification model comprises an input layer, a hidden layer and an output layer.
[0073] S301, determining an image classification model used for image classification of a target image to be classified, wherein the image classification model comprises an input layer, a hidden layer and an output layer.
[0074] S303, receiving an image data vector of the target image through an input layer of the image classification model;
[0075] S305, performing image feature extraction through a hidden layer of the image classification model according to the image data vector, to obtain corresponding image features, wherein model parameters of the hidden layer are obtained through joint training of the image classification model and an auxiliary training model, the auxiliary training model has a smaller scale than the image classification model, and the joint training includes: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through a hidden layer, and a second model feature vector output by the image classification model after processing the input sample through the hidden layer, wherein the second model feature vector includes a first sub-feature vector and a second sub-feature vector; converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into first, second and third intermediate vectors of a preset dimension respectively; obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector; training the image classification model according to at least the first loss value and the second loss value, and determining model parameters of the hidden layer of the image classification model according to a training result.
[0076] The hidden layer includes, for example, a convolution layer, a connection layer and a BN layer, and the model parameters include, for example, weight values of convolution kernels in the convolution layer, bias values of channels in the convolution layer, weight values of the full connection layer and channel bias values.
[0077] S307, outputting image category data corresponding to the image features through an output layer of the image classification model, the image category data being used to indicate probabilities of the target image corresponding to different image categories.
[0078] The specific manner of the joint training has been described above, and will not be repeated here. Through the joint training, the image classification model obtained will have a part of the image features extracted through the hidden layer close to the image features extracted by the auxiliary training model, and another part of the image features extracted through the hidden layer far from the image features extracted by the auxiliary training model.
[0079] Since the auxiliary training model is a small-scale model, the auxiliary training model itself can be quickly iterated to obtain a large number of effective classification features, and after joint training, the image classification model can inherit part of the features beneficial to image classification from the auxiliary training model, and also explore part of the features beneficial to image classification. Therefore, the effective features obtained by the auxiliary model training can be quickly transferred to the image classification model after joint training, so as to train the image classification model with more accurate classification effect. Specifically, the image features extracted in the hidden layer of the image classification model have better class features, and the class probability determined by the image features in the output layer is closer to the actual class of the image.
[0080] A third aspect of the embodiments of the present application also provides a target object detection method, as shown in Figure 4 Figure 4 A flowchart of a target object detection method provided by the embodiments of the present application is shown in
[0081] S401, determining a target object detection model for target object detection of a target image, wherein the target object detection model includes an input layer, a hidden layer and an output layer;
[0082] S403, receiving an image data vector of the target image through the input layer of the target object detection model;
[0083] S405, performing image feature extraction through the hidden layer of the target object detection model according to the image data vector to obtain corresponding image features, wherein the model parameters of the hidden layer are obtained through joint training of the target object detection model and an auxiliary training model, the scale of the auxiliary training model is smaller than that of the target object detection model, and the joint training includes: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through the hidden layer, and a second model feature vector output by the target object detection model after processing the input sample through the hidden layer, wherein the second model feature vector includes a first sub-feature vector and a second sub-feature vector; respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension; obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector; training the target object detection model according to at least the first loss value and the second loss value, and determining the model parameters of the hidden layer of the target object detection model according to the training result;
[0084] S407, output target object data corresponding to the image features through an output layer of the target object detection model.
[0085] The target object detection model is also obtained through the joint training manner described above, and the auxiliary training model used in the joint training period is also a target object detection model with a smaller scale.
[0086] The target object detection model obtained based on the joint training manner described above also has similar effects of the image classification model described in the third aspect when extracting features, that is, part of the extracted features are similar to the auxiliary training model, and part of the extracted features are far away from the auxiliary training model, so that the target object detection model can quickly learn effective image features from the auxiliary training model and improve the accuracy of the target object detection model.
[0087] The fourth aspect of the embodiments of the present application also provides another model training method, as shown in Figure 5 Figure 5 The flowchart of another model training method provided by the embodiments of the present application, the method comprises:
[0088] S501, accepting a model training request sent by a client, and obtaining an input sample, a first model and a second model for training according to the model training request.
[0089] S503, obtaining a first model feature vector output by the first model after processing the input sample through the hidden layer, and a second model feature vector output by the second model after processing the input sample through the hidden layer, wherein the scale of the first model is smaller than that of the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector.
[0090] S505, respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector with a preset dimension.
[0091] S507, obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector.
[0092] S509, training the second model at least according to the first loss value and the second loss value.
[0093] In an implementation, the input sample, the first model and the second model can be pre-deployed on a SaaS (Software-as-a-Service) platform, a model training request of a client can be received by the SaaS platform, and the input sample, the first model and the second model used for training can be obtained from the SaaS platform locally according to the model training request. In this case, the input sample, the first model and the second model are stored locally on the SaaS platform, and thus can be directly obtained, thereby improving the speed and efficiency of model training.
[0094] In another implementation, the SaaS platform can receive a model training request of a client, and obtain an input sample used for training from a third party. The input sample can be obtained from a third-party website or a data interface of a third-party application through a network. Further, the obtained input sample is used to train the second model stored locally on the SaaS platform, or the first model and the second model stored locally. In this case, the SaaS platform obtains relevant resources from the third party, and does not need to store locally, thereby saving storage resources of the SaaS platform.
[0095] In another implementation, the SaaS platform can obtain an input sample used for training from the client according to the model training request. Further, the obtained input sample is used to train the second model stored locally on the SaaS platform, or the first model and the second model stored locally. In this case, the input sample used for training is stored in the client, and the SaaS platform obtains relevant resources from the client, thereby training a model that is more suitable for the client.
[0096] An example is taken to illustrate the case of deploying the input sample and the first model and the second model on the SaaS platform. As shown in Figure 6 , an architecture schematic diagram of a SaaS platform for model training is provided in the embodiments of the present application. Figure 6
[0097] In the schematic diagram, the client inputs a model training request, and the SaaS platform obtains the input sample, the first model and the second model used for training from the local, further performs relevant training of the second model on the features in the first model, and sends notification information to the client after the training is completed. Subsequently, the client can use the second model according to actual requirements.
[0098] A fifth aspect of the embodiments of the present application further provides a model training device, as shown in Figure 7 , a schematic diagram of a model training device is provided in the embodiments of the present application. The device comprises: Figure 7
[0099] The acquisition module 701 acquires a first model feature vector output by the first model after processing an input sample through a hidden layer, and a second model feature vector output by the second model after processing the input sample through a hidden layer, wherein the scale of the first model is smaller than that of the second model, and the second model feature vector includes a first sub-feature vector and a second sub-feature vector;
[0100] The conversion module 703 converts the first model feature vector, the first sub-feature vector and the second sub-feature vector into first, second and third intermediate vectors of a preset dimension, respectively.
[0101] The loss value determination module 705 obtains a first loss value according to the first intermediate vector and the second intermediate vector, and obtains a second loss value according to the first intermediate vector and the third intermediate vector.
[0102] The training module 707 trains the second model according to at least the first loss value and the second loss value.
[0103] Optionally, the conversion module 703 maps an initial first model feature vector to a vector space of a specified dimension to generate a specified dimension intermediate vector to be determined, regenerates a reconstructed first model feature vector according to the intermediate vector to be determined, and determines the intermediate vector as a first intermediate vector when the difference between the reconstructed first model feature vector and the initial first model feature vector meets a preset condition.
[0104] Optionally, the conversion module 703, the difference between the reconstructed first model feature vector and the initial first model feature vector meeting the preset condition includes that the distance between the reconstructed first model feature vector and the initial first model feature vector is less than a distance preset value, or the included angle is less than an included angle preset value.
[0105] Optionally, the loss value determination module 705 normalizes the first intermediate vector and the second intermediate vector to obtain a normalized first intermediate vector and a normalized second intermediate vector, and determines the distance between the normalized first intermediate vector and the normalized second intermediate vector as the first loss value.
[0106] Optionally, the loss value determination module 705 normalizes the first intermediate vector and the third intermediate vector to obtain a normalized first intermediate vector and a normalized third intermediate vector, and determines the reciprocal of the distance between the normalized first intermediate vector and the normalized third intermediate vector as the second loss value.
[0107] Optionally, the training module 707 determines a basic loss value of the second model for the input sample; and trains the second model according to a sum of the basic loss value, the first loss value and the second loss value.
[0108] Optionally, the training module 707 synchronously trains the first model when training the second model.
[0109] The training apparatus of the model of the embodiment is used to implement the corresponding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the function implementation of each module in the training apparatus of the embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here either.
[0110] A sixth aspect of the embodiment of the present application further provides an electronic device. As shown in Figure 8 Figure 8 is a schematic diagram of an electronic device provided by the embodiment of the present application. The electronic device can include a processor 602, a communications interface 604, a memory 606, and a communications bus 608.
[0111] Among them:
[0112] The processor 602, the communications interface 604, and the memory 606 complete mutual communication through the communications bus 608.
[0113] The communications interface 604 is configured to communicate with other electronic devices or servers.
[0114] The processor 602 is configured to execute the program 610, and specifically can execute the related steps in the foregoing method embodiments.
[0115] Specifically, the program 610 can include program code including computer operation instructions, and can be used to make the processor 602 execute the following operations:
[0116] obtaining a first model feature vector output by the first model after processing an input sample through an implicit layer, and a second model feature vector output by the second model after processing the input sample through the implicit layer, wherein a scale of the first model is smaller than that of the second model, and the second model feature vector includes a first sub-feature vector and a second sub-feature vector;
[0117] respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension;
[0118] obtaining a first loss value according to the first intermediate vector and the second intermediate vector; and obtaining a second loss value according to the first intermediate vector and the third intermediate vector;
[0119] training the second model according to at least the first loss value and the second loss value.
[0120] The processor 602 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.
[0121] The memory 606 is configured to store a program 610. The memory 606 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.
[0122] A seventh aspect of an embodiment of the present application further provides a computer storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method of training the model as Figures 2 to 5 described above.
[0123] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.
[0124] The above-described methods according to embodiments of the present application can be implemented in hardware, firmware, or software, or any combination thereof, and can be stored in a recording medium such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded by network from a remote recording medium or non-transitory machine-readable medium originally stored in a local recording medium and then stored in a local recording medium, so that the methods described herein can be processed by such software using a general purpose computer, a special purpose processor, or programmable or dedicated hardware (such as ASIC or FPGA). It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor, or hardware, the method of model training described herein is implemented. Furthermore, when a general purpose computer accesses code for implementing the method of model training shown herein, the execution of the code will convert the general purpose computer into a special purpose computer for executing the method of model training shown herein.
[0125] Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.
[0126] The above embodiments are only used to illustrate the present application, and not to limit the present application, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore all equivalent technical solutions also belong to the scope of the present application, the patent protection scope of the present application should be defined by the claims.
Claims
1. A method for training a model, the method comprising: obtaining a first model feature vector output by a first model after processing an input sample through a hidden layer, and a second model feature vector output by a second model after processing the input sample through a hidden layer, wherein a size of the first model is smaller than that of the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into first, second and third intermediate vectors of a preset dimension, respectively; obtaining a first loss value according to the first and second intermediate vectors, and a second loss value according to the first and third intermediate vectors, wherein the first loss value is determined by a distance between the normalized first and second intermediate vectors, and the second loss value is determined by a reciprocal of a distance between the normalized first and third intermediate vectors; training the second model according to at least the first and second loss values, wherein the second model is used to recognize at least images, texts or voices.
2. The method of claim 1, wherein, converting the first model feature vector into a first intermediate vector of a preset dimension comprises: mapping an initial first model feature vector to a vector space of a specified dimension to generate a to-be-determined intermediate vector of the specified dimension; regenerating a reconstructed first model feature vector according to the to-be-determined intermediate vector; when a difference between the reconstructed first model feature vector and the initial first model feature vector meets a preset condition, determining the intermediate vector as the first intermediate vector.
3. The method of claim 2, wherein, the difference between the reconstructed first model feature vector and the initial first model feature vector meets a preset condition comprises: a distance between the reconstructed first model feature vector and the initial first model feature vector is less than a distance preset value, or an included angle is less than an included angle preset value.
4. The method of claim 2, wherein, obtaining a first loss value according to the first and second intermediate vectors comprises: normalizing the first and second intermediate vectors to obtain normalized first and second intermediate vectors, respectively; determining a distance between the normalized first and second intermediate vectors as the first loss value.
5. The method of claim 2, wherein, obtaining a second loss value according to the first and third intermediate vectors comprises: normalizing the first and third intermediate vectors to obtain normalized first and third intermediate vectors, respectively; determining a reciprocal of a distance between the normalized first and third intermediate vectors as the second loss value.
6. The method of claim 1, wherein, training the second model according to at least the first and second loss values comprises: determining a basic loss value of the second model for the input sample; training the second model according to a sum of the basic loss value, the first loss value and the second loss value.
7. The method of claim 6, wherein, training the second model, comprising: synchronously training the first model while training the second model.
8. An image classification method, comprising: determining an image classification model for image classification of a target image to be classified, wherein the image classification model comprises an input layer, a hidden layer, and an output layer; receiving an image data vector of the target image through the input layer of the image classification model; performing image feature extraction according to the image data vector through the hidden layer of the image classification model to obtain corresponding image features, wherein model parameters of the hidden layer are obtained through joint training of the image classification model and an auxiliary training model, the auxiliary training model has a smaller scale than the image classification model, and the joint training comprises: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through the hidden layer, and a second model feature vector output by the image classification model after processing the input sample through the hidden layer, wherein the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; converting the first model feature vector, the first sub-feature vector, and the second sub-feature vector into first, second, and third intermediate vectors of a preset dimension, respectively; obtaining a first loss value according to the first and second intermediate vectors; and obtaining a second loss value according to the first and third intermediate vectors; training the image classification model according to at least the first and second loss values, and determining model parameters of the hidden layer of the image classification model according to a training result, wherein the first loss value is determined by a distance between the normalized first intermediate vector and the normalized second intermediate vector, and the second loss value is determined by a reciprocal of a distance between the normalized first intermediate vector and the normalized third intermediate vector; outputting image class data corresponding to the image features through the output layer of the image classification model, the image class data being used to indicate probabilities of the target image corresponding to different image classes.
9. A target object detection method, comprising: determining a target object detection model for target object detection of a target image, wherein the target object detection model comprises an input layer, a hidden layer, and an output layer; receiving an image data vector of the target image through the input layer of the target object detection model; The image feature is extracted from the image data vector through a hidden layer of the target object detection model, and a corresponding image feature is obtained, wherein model parameters of the hidden layer are obtained through joint training of the target object detection model and an auxiliary training model, the auxiliary training model has a smaller scale than the target object detection model, the joint training comprises: obtaining a first model feature vector output by the auxiliary training model after processing an input sample through a hidden layer, and a second model feature vector output by the target object detection model after processing the input sample through the hidden layer, wherein the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; the first model feature vector, the first sub-feature vector and the second sub-feature vector are respectively converted into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension; a first loss value is obtained according to the first intermediate vector and the second intermediate vector; and a second loss value is obtained according to the first intermediate vector and the third intermediate vector; the target object detection model is trained according to at least the first loss value and the second loss value, and model parameters of the hidden layer of the target object detection model are determined according to a training result, wherein the first loss value is determined by a distance between the normalized first intermediate vector and the normalized second intermediate vector, and the second loss value is determined by a reciprocal of a distance between the normalized first intermediate vector and the normalized third intermediate vector; Target object data corresponding to the image feature is output through an output layer of the target object detection model.
10. A model training method, the method comprising: receiving a model training request sent by a client, and obtaining an input sample and a first model and a second model for training according to the model training request; obtaining a first model feature vector output by the first model after processing the input sample through a hidden layer, and a second model feature vector output by the second model after processing the input sample through the hidden layer, wherein the first model has a smaller scale than the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; respectively converting the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension; obtaining a first loss value according to the first intermediate vector and the second intermediate vector, and a second loss value according to the first intermediate vector and the third intermediate vector, wherein the first loss value is determined by a distance between the normalized first intermediate vector and the normalized second intermediate vector, and the second loss value is determined by a reciprocal of a distance between the normalized first intermediate vector and the normalized third intermediate vector; training the second model according to at least the first loss value and the second loss value, wherein the second model is used for identifying at least images, texts and voices.
11. The method of claim 10, wherein, According to the model training request, an input sample for training and a first model are obtained, comprising: According to the model training request, the input sample for training is obtained from the SaaS platform locally; or, According to the model training request, the input sample for training is obtained from a third party by the SaaS platform; or, According to the model training request, the input sample for training is obtained from the client by the SaaS platform.
12. A model training device, the device comprising: an acquisition module, which acquires a first model feature vector output by a first model after processing an input sample through a hidden layer, and a second model feature vector output by a second model after processing the input sample through a hidden layer, wherein the scale of the first model is smaller than that of the second model, and the second model feature vector comprises a first sub-feature vector and a second sub-feature vector; a conversion module, which respectively converts the first model feature vector, the first sub-feature vector and the second sub-feature vector into a first intermediate vector, a second intermediate vector and a third intermediate vector of a preset dimension; a loss value determination module, which obtains a first loss value according to the first intermediate vector and the second intermediate vector, and obtains a second loss value according to the first intermediate vector and the third intermediate vector, wherein the first loss value is determined by the distance between the normalized first intermediate vector and the normalized second intermediate vector, and the second loss value is determined by the inverse of the distance between the normalized first intermediate vector and the normalized third intermediate vector; a training module, which trains the second model according to at least the first loss value and the second loss value, wherein the second model is used for recognizing at least images, texts and voices.
13. An electronic device comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to perform the corresponding operation of the method in any one of claims 1-9.
14. A computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the method in any one of claims 1-9.