Model generation method, device, electronic device and medium

Abstract

The present application discloses a method, device, electronic device and medium for generating a model. By applying the technical solution of the present application, the Laplace matrices of the teacher model and the student model can be adjusted to determine that the features output by the two have a similar manifold structure, thereby assisting the student model to learn discriminative features from the teacher model and obtain higher performance. This avoids the problem existing in the related art that the compressed model usually has a reduced accuracy of the algorithm due to the small number of model parameters.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a method, device, electronic device and medium for generating a model. Background Art

[0002] In recent years, deep neural networks have enabled the performance of many computer vision tasks to reach unprecedented levels.

[0003] The more complex the model structure of the neural network and the more parameters it has, the richer the knowledge the network can learn and the better the learning effect. However, the high storage space and large computing resource consumption make it difficult for large network models to be widely used on various hardware platforms. Model compression, as an effective solution, has attracted the attention of more and more researchers. Model compression methods obtain small models by trimming parameters and decomposing weights of large models, or use large models (teacher models) to guide the training of small models (student models) through methods such as model distillation, so as to reduce the model's consumption of computing space and time.

[0004] However, the compressed model usually has the problem of reduced algorithm accuracy due to the small number of model parameters, thereby affecting the processing efficiency of upper-level services. Summary of the invention

[0005] The embodiments of the present application provide a model generation method, device, electronic device and medium to solve the problem in the related art that a compressed model usually has a small number of model parameters, which leads to reduced accuracy of the algorithm.

[0006] According to one aspect of an embodiment of the present application, a method for generating a model is provided, comprising:

[0007] Get the trained teacher model and the initial student model;

[0008] Obtaining a first feature output by the teacher model, and obtaining a second feature output by the initial student model;

[0009] Based on the first feature and the second feature, construct a manifold structure loss function;

[0010] The initial student model is trained by the overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, the target student model is obtained, wherein the overall classification loss function is composed of the distillation classification loss function and the manifold structure loss function.

[0011] Optionally, in another embodiment of the method of the present application, the step of obtaining a trained teacher model includes:

[0012] An image classification training data set is constructed, wherein the training data set includes at least one sample image data and a classification label corresponding to the sample image data.

[0013] The teacher model is trained using the training data set.

[0014] Optionally, in another embodiment of the method of the present application, the obtaining of the first feature output by the teacher model and the obtaining of the second feature output by the initial student model include:

[0015] The same batch of sample image data is input into the teacher model and the initial student model respectively to obtain the first feature and the second feature.

[0016] Optionally, in another embodiment of the method of the present application, after obtaining the first feature and the second feature, the method further includes:

[0017] Using the first feature, constructing a first Laplace matrix corresponding to the teacher model; and,

[0018] Using the second feature, constructing a second Laplace matrix corresponding to the student model;

[0019] The manifold structure loss function is constructed based on a matrix difference between the first Laplacian matrix and the second Laplacian matrix.

[0020] Optionally, in another embodiment based on the above method of the present application, the manifold structure loss function is constructed by the following formula:

[0021]

[0022] Among them, L teacher is the first Laplace matrix corresponding to the teacher model, L student is the second Laplace matrix corresponding to the initial student model.

[0023] Optionally, in another embodiment of the method of the present application, after constructing the manifold structure loss function based on the first feature and the second feature, the method further includes:

[0024] Obtaining the overall loss function, wherein the overall loss function is composed of the manifold structure loss function, the weight of the manifold structure loss function, and the distillation classification loss function;

[0025] Based on the overall loss function, the loss of the initial student model is calculated, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm.

[0026] Optionally, in another embodiment of the method of the present application, the loss of the initial student model is calculated based on the overall loss function, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm, further comprising:

[0027] Input different batches of sample image data, iteratively optimize the initial student model until the preset training conditions are met, stop optimizing the model, and obtain the trained target student model, wherein the training conditions include one of the number of iterations exceeding the preset upper limit value and the model loss value being less than the preset threshold value.

[0028] According to another aspect of the embodiment of the present application, a model generation device is provided, characterized in that it includes:

[0029] A first acquisition module is configured to acquire a trained teacher model and an initial student model;

[0030] A second acquisition module is configured to acquire a first feature output by the teacher model and to acquire a second feature output by the initial student model;

[0031] A construction module is configured to construct a manifold structure loss function based on the first feature and the second feature;

[0032] A generation module is configured to train the initial student model through an overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function.

[0033] According to another aspect of the embodiments of the present application, an electronic device is provided, including:

[0034] A memory for storing executable instructions; and

[0035] A display is used to execute the executable instructions together with the memory to complete the operation of any of the above-mentioned model generation methods.

[0036] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided for storing computer-readable instructions, which, when executed, perform the operations of any of the above-mentioned model generation methods.

[0037] In the present application, a trained teacher model and an initial student model can be obtained; a first feature outputted by the teacher model is obtained, and a second feature outputted by the initial student model is obtained; a manifold structure loss function is constructed based on the first feature and the second feature; the initial student model is trained by the overall loss function until it is determined that the features outputted by the initial student model have a similar manifold structure to the features outputted by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and a manifold structure loss function. By applying the technical solution of the present application, the Laplace matrices of the teacher model and the student model can be adjusted to determine that the features outputted by the two have a similar manifold structure, thereby assisting the student model to learn discriminative features from the teacher model and obtain higher performance. This avoids the problem that the compressed model usually has a small number of model parameters, which reduces the accuracy of the algorithm.

[0038] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0040] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0041] Figure 1 A schematic diagram of a method for generating a model proposed in this application;

[0042] Figure 2 A flowchart of the working principle of the teacher model and the initial student model proposed for this application;

[0043] Figure 3 A schematic diagram of the structure of an electronic device for generating a model proposed in this application;

[0044] Figure 4 A schematic diagram of the structure of an electronic device for generating a model proposed in this application. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0046] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present application, its application, or uses.

[0048] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0051] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0052] Combine the following Figure 1-Figure 2 To describe the method for generating a model according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0053] The present application also proposes a model generation method, device, electronic device and medium.

[0054] Figure 1 The following schematically shows a flow chart of a method for generating a model according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0055] S101, obtain the trained teacher model and the initial student model.

[0056] S102, obtaining a first feature output by using the teacher model, and obtaining a second feature output by using the initial student model.

[0057] S103: construct a manifold structure loss function based on the first feature and the second feature.

[0058] S104, training the initial student model through the overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, the target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and a manifold structure loss function.

[0059] In related technologies, deep neural networks have enabled the performance of many computer vision tasks to reach unprecedented heights. The more complex the model structure of the neural network and the more parameters it has, the richer the knowledge the network can learn and the better the learning effect.

[0060] However, the high storage space and large amount of computing resources make it difficult for large network models to be widely used on various hardware platforms. Model compression, as an effective solution, has attracted the attention of more and more researchers. Model compression methods obtain small models by pruning parameters, decomposing weights, or distilling models, thereby reducing the model's consumption of computing space and time.

[0061] Among them, in the model distillation method, a large model with high accuracy can be pre-trained, called the teacher model. Then the model parameters learned from the teacher model are used to guide the training of a student model with fewer parameters. Among them, the student model improves its performance by learning information from the teacher model that is beneficial to classification. And because the student model has fewer parameters and runs fast, it can be easily deployed on various hardware platforms.

[0062] However, the compressed model usually has the problem of reduced algorithm accuracy due to the small number of model parameters, which also affects the business processing efficiency.

[0063] Further, such as Figure 2 As shown in the figure, the learning model is taken as an example of an image classification model. Since an image classification learning model usually consists of two parts: a feature extraction subnetwork and a classification subnetwork. Among them, the feature extraction network passes the input image into a series of feature extraction layers and finally outputs the extracted features. The classification network is usually a fully connected layer. Its input is the features extracted by the feature extraction network, and its output is the confidence that the image belongs to each category.

[0064] Therefore, it is assumed that the features extracted by the network are N-dimensional vectors. For the classification task, the N-dimensional feature vectors extracted from all images constitute a low-dimensional manifold embedded in the N-dimensional space. The structure of the manifold describes the local structural features of the manifold, especially the similarity relationship between local feature points.

[0065] In one way, the embodiment of the present application requires that the features output by the initial student model have a similar manifold structure to the features output by the teacher model. In order to better model the data distribution. It can be understood that the embodiment of the present application adopts the method of constraining the manifold structure of the student model feature points to be consistent with or highly similar to the manifold structure of the teacher model feature points, so that the features extracted by the student model have the advantage of being consistent with the teacher model.

[0066] In one embodiment, the present application can be implemented by adjusting the Plaus matrices of the two models to be the same or highly similar matrices in the process of constraining the manifold structure of the student model feature points to be consistent with the manifold structure of the teacher model feature points.

[0067] Specifically, for example, the number of sample images used in each batch of training of the classification network is K, and the extracted features are K points distributed in the N-dimensional space. The manifold structure of the feature points can be represented by their Laplacian matrix.

[0068] Furthermore, in the process of training the learning model, the present application can simultaneously input the same batch of sample images into the teacher model and the initial student model each time. So that the features extracted from the teacher model and the initial student model (i.e., the second feature and the first feature) can be used to construct Laplace matrices (i.e., the first Laplace matrix and the second Laplace matrix) respectively. And it is assumed that the Laplace matrix of the teacher model is:

[0069] L teacher ∈R K×K .

[0070] Among them, L teacher is a symmetric matrix. It is constructed as follows:

[0071] First, all the first feature points form a graph and construct the adjacency matrix W of the graph teacher ∈R K×K . The adjacency matrix is ​​constructed as follows:

[0072] For each feature point, use the KNN algorithm to find the M feature points closest to it, and build connecting edges between the feature point and the M feature points.

[0073] Determine the connection weight for each edge. Use the heat kernel function to calculate the weight value:

[0074]

[0075] Construct the degree matrix D of the planar graph teacher ∈R K×K The degree matrix is ​​a diagonal matrix, and the elements on the diagonal are constructed as follows:

[0076]

[0077] Finally, the Laplacian matrix is ​​obtained by the matrix difference between the adjacency matrix and the degree matrix, which is obtained by the following formula:

[0078] L teacher =D teacher -W teacher .

[0079] It should be noted that in the process of training the student model in the embodiment of the present application, the feature vector extracted from the student model is also constructed in the same way as the Laplace matrix L student ∈R K×K (i.e., the second Laplace matrix), and constrain the similarity matrix of the initial student model to remain the same or highly similar to the similarity matrix of the teacher model, that is, minimize

[0080] It can be understood that the embodiments of the present application can assist the student model in learning features with similar distribution to the teacher model by constraining the manifold structure of the features extracted by the student model to be the same as or highly similar to that of the teacher model, thereby achieving higher performance.

[0081] In the present application, a trained teacher model and an initial student model can be obtained; a first feature outputted by the teacher model is obtained, and a second feature outputted by the initial student model is obtained; a manifold structure loss function is constructed based on the first feature and the second feature; the initial student model is trained by the overall loss function until it is determined that the features outputted by the initial student model have a similar manifold structure to the features outputted by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and a manifold structure loss function. By applying the technical solution of the present application, the Laplace matrices of the teacher model and the student model can be adjusted to determine that the features outputted by the two have a similar manifold structure, thereby assisting the student model to learn better similarity information between categories from the teacher model and obtain higher performance. This avoids the problem that the compressed model usually has a small number of model parameters, which reduces the accuracy of the algorithm.

[0082] Optionally, in another embodiment of the method of the present application, the step of obtaining a trained teacher model includes:

[0083] An image classification training data set is constructed, wherein the training data set includes at least one sample image data and a classification label corresponding to the sample image data.

[0084] The teacher model is trained using the training data set.

[0085] Optionally, in another embodiment of the method of the present application, the obtaining of the first feature output by the teacher model and the obtaining of the second feature output by the initial student model include:

[0086] The same batch of sample image data is input into the teacher model and the initial student model respectively to obtain the first feature and the second feature.

[0087] Optionally, in another embodiment of the method of the present application, after obtaining the first feature and the second feature, the method further includes:

[0088] Using the first feature, constructing a first Laplace matrix corresponding to the teacher model; and,

[0089] Using the second feature, constructing a second Laplace matrix corresponding to the student model;

[0090] The manifold structure loss function is constructed based on a matrix difference between the first Laplacian matrix and the second Laplacian matrix.

[0091] Optionally, in another embodiment based on the above method of the present application, the manifold structure loss function is constructed by the following formula:

[0092]

[0093] Among them, L teacher is the first Laplace matrix corresponding to the teacher model, L student is the second Laplace matrix corresponding to the initial student model.

[0094] Optionally, in another embodiment of the method of the present application, after constructing the manifold structure loss function based on the first feature and the second feature, the method further includes:

[0095] Obtaining the overall loss function, wherein the overall loss function is composed of the manifold structure loss function, the weight of the manifold structure loss function, and the distillation classification loss function;

[0096] Based on the overall loss function, the loss of the initial student model is calculated, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm.

[0097] Optionally, in another embodiment of the method of the present application, the loss of the initial student model is calculated based on the overall loss function, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm, further comprising:

[0098] Input different batches of sample image data, iteratively optimize the initial student model until the preset training conditions are met, stop optimizing the model, and obtain the trained target student model, wherein the training conditions include one of the number of iterations exceeding the preset upper limit value and the model loss value being less than the preset threshold value.

[0099] In one way, the embodiment of the present application requires that the first feature output by the initial student model and the second feature output by the teacher model have a similar manifold structure. In order to better model the data distribution. It can be understood that the embodiment of the present application adopts the method of constraining the manifold structure of the student model feature points to be the same as or highly similar to the manifold structure of the teacher model feature points, so that the features extracted by the student model have the advantage of being consistent with the teacher model.

[0100] In one embodiment, the present application can be implemented by adjusting the Plaus matrices of the two learning models to be the same or highly similar matrices in the process of constraining the manifold structure of the student model feature points to be consistent with the manifold structure of the teacher model feature points.

[0101] Specifically, adjusting the Plath matrices of the two learning models to be the same or highly similar matrices can be obtained by the following formula:

[0102]

[0103] Through the above calculation based on the difference of the two similarity matrices, the loss function of the manifold structure can be obtained, and then the overall loss function can be obtained based on the manifold structure loss function, the weight of the manifold structure loss function and the distillation classification loss function. Finally, the model parameters of the initial student model can be updated using the stochastic gradient descent algorithm according to the overall loss function.

[0104] Specifically, for obtaining the overall loss function of the student model, it can be specifically:

[0105]

[0106] Among them, λ is the weight of the manifold structure loss function, L distill is the distillation classification loss function of the model and is defined as,

[0107]

[0108] Among them, p and q are the softmax layer output vectors of the teacher model and the student model respectively. The softmax layer is defined as:

[0109]

[0110] Among them, vector z is the output vector of the fully connected layer.i represents the probability of the i-th category, z i represents the i-th dimension of the fully connected layer output vector, z j Represents the jth dimension of the fully connected layer output vector. T is a parameter that controls the smoothness of the output. The larger T is, the smoother the output probability distribution is.

[0111] In one approach, during the training of the teacher model and the student model, the softmax layer of the teacher model and the student model both use a parameter T greater than 1 (e.g., T=10). Since the probability distribution is smoother, the output of the teacher model has a higher information entropy, thus containing more information for the student model to learn the model parameters.

[0112] Furthermore, in the process of obtaining the trained teacher model and the initial student model compressed based on the teacher model, the present application can obtain the following methods:

[0113] Construct an image classification training dataset. The training dataset includes image data and the classification label of each image.

[0114] Using the training dataset, train a large model as the teacher model (e.g., ResNet101). Also, build a small classification model to be trained as the student model, such as ResNet18.

[0115] In the embodiment of the present application, the batch size of the number of images used for each training model can be set, and the training data set is input to the student model and the teacher model respectively according to the batches.

[0116] Furthermore, the loss of the student model can be calculated based on the proposed overall loss function, and the student model parameters can be updated using the stochastic gradient descent algorithm. The student model can be iteratively optimized by continuously inputting different batches of sample learning data. When the number of iterations is equal to the preset upper limit, or the loss value is less than the preset threshold, the optimization model is stopped to obtain the final trained target student model.

[0117] In the present application, a trained teacher model and an initial student model can be obtained; a first feature output by the teacher model is obtained, and a second feature output by the initial student model is obtained; a manifold structure loss function is constructed based on the first feature and the second feature; the initial student model is trained by the overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, the target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and a manifold structure loss function.

[0118] By applying the technical solution of the present application, the Laplace matrices of the teacher model and the student model can be adjusted to determine that the features output by the two have similar manifold structures, thereby assisting the student model to learn discriminative features from the teacher model and obtain higher performance. This avoids the problem in the related art that the compressed model usually has a small number of model parameters, resulting in reduced algorithm accuracy.

[0119] Optionally, in another embodiment of the present application, Figure 3 As shown, the present application also provides a model generation device. It includes:

[0120] A first acquisition module is configured to acquire a trained teacher model and an initial student model;

[0121] A second acquisition module is configured to acquire a first feature output by the teacher model and to acquire a second feature output by the initial student model;

[0122] A construction module is configured to construct a manifold structure loss function based on the first feature and the second feature;

[0123] A generation module is configured to train the initial student model through an overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function.

[0124] In the present application, a trained teacher model and an initial student model can be obtained; a first feature outputted by the teacher model is obtained, and a second feature outputted by the initial student model is obtained; a manifold structure loss function is constructed based on the first feature and the second feature; the initial student model is trained by the overall loss function until it is determined that the features outputted by the initial student model have a similar manifold structure to the features outputted by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and a manifold structure loss function. By applying the technical solution of the present application, the Laplace matrices of the teacher model and the student model can be adjusted to determine that the features outputted by the two have a similar manifold structure, thereby assisting the student model to learn better similarity information between categories from the teacher model and obtain higher performance. This avoids the problem that the compressed model usually has a small number of model parameters, which reduces the accuracy of the algorithm.

[0125] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0126] An image classification training data set is constructed, wherein the training data set includes at least one sample image data and a classification label corresponding to the sample image data.

[0127] The teacher model is trained using the training data set.

[0128] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0129] The same batch of sample image data is input into the teacher model and the initial student model respectively to obtain the first feature and the second feature.

[0130] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0131] Using the first feature, constructing a first Laplace matrix corresponding to the teacher model; and,

[0132] Using the second feature, constructing a second Laplace matrix corresponding to the student model;

[0133] The manifold structure loss function is constructed based on a matrix difference between the first Laplacian matrix and the second Laplacian matrix.

[0134] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0135] The manifold structure loss function is constructed by the following formula:

[0136]

[0137] Among them, L teacher is the first Laplace matrix corresponding to the teacher model, L student is the second Laplace matrix corresponding to the initial student model.

[0138] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0139] Obtaining the overall loss function, wherein the overall loss function is composed of the manifold structure loss function, the weight of the manifold structure loss function, and the distillation classification loss function;

[0140] Based on the overall loss function, the loss of the initial student model is calculated, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm.

[0141] In another embodiment of the present application, the first acquisition module 201 is configured to execute the following steps:

[0142] Input different batches of sample image data, iteratively optimize the initial student model until the preset training conditions are met, stop optimizing the model, and obtain the trained target student model, wherein the training conditions include one of the number of iterations exceeding the preset upper limit value and the model loss value being less than the preset threshold value.

[0143] Figure 4 3 is a logical structure block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0144] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory including instructions, the instructions can be executed by an electronic device processor to complete the above-mentioned model generation method, the method comprising: obtaining a trained teacher model and an initial student model; obtaining a first feature output by the teacher model, and obtaining a second feature output by the initial student model; constructing a manifold structure loss function based on the first feature and the second feature; training the initial student model through the overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, the target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function. Optionally, the above-mentioned instructions can also be executed by a processor of an electronic device to complete other steps involved in the above-mentioned exemplary embodiment. For example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, an optical data storage device, etc.

[0145] In an exemplary embodiment, an application / computer program product is also provided, including one or more instructions, which can be executed by a processor of an electronic device to complete the above-mentioned model generation method, the method comprising: obtaining a trained teacher model and an initial student model; obtaining a first feature output by the teacher model, and obtaining a second feature output by the initial student model; constructing a manifold structure loss function based on the first feature and the second feature; training the initial student model through an overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function. Optionally, the above-mentioned instructions can also be executed by a processor of an electronic device to complete other steps involved in the above-mentioned exemplary embodiment.

[0146] Those skilled in the art will understand that Figure 4 It is only an example of the electronic device 300 and does not constitute a limitation of the electronic device 300. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 300 may also include input and output devices, network access devices, buses, etc.

[0147] The processor 302 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor 302 may also be any conventional processor, etc. The processor 302 is the control center of the electronic device 300, and uses various interfaces and lines to connect various parts of the entire electronic device 300.

[0148] The memory 301 can be used to store computer-readable instructions. The processor 302 implements various functions of the electronic device 300 by running or executing the computer-readable instructions or modules stored in the memory 301 and calling the data stored in the memory 301. The memory 301 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device 300, etc. In addition, the memory 301 can include a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM) or other non-volatile / volatile storage devices.

[0149] If the module integrated in the electronic device 300 is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete it through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0150] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0151] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a model, characterized in that: include: Obtaining a trained teacher model and an initial student model, wherein both the teacher model and the initial student model are used for image classification; Obtaining a first feature output by the teacher model, and obtaining a second feature output by the initial student model; Based on the first feature and the second feature, construct a manifold structure loss function; The initial student model is trained by an overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function; The constructing the manifold structure loss function based on the first feature and the second feature includes: Using the first feature, constructing a first Laplace matrix corresponding to the teacher model; and, Using the second feature, constructing a second Laplace matrix corresponding to the student model; The manifold structure loss function is constructed based on a matrix difference between the first Laplacian matrix and the second Laplacian matrix.

2. The method according to claim 1, characterized in that The step of obtaining the trained teacher model includes: Constructing an image classification training data set, wherein the training data set includes at least one sample image data and a classification label corresponding to the sample image data; The teacher model is trained using the training data set.

3. The method according to claim 1, characterized in that The obtaining of the first feature output by the teacher model and the obtaining of the second feature output by the initial student model include: The same batch of sample image data is input into the teacher model and the initial student model respectively to obtain the first feature and the second feature.

4. The method according to claim 1, characterized in that The manifold structure loss function is constructed by the following formula: ； in, is the first Laplace matrix corresponding to the teacher model, is the second Laplace matrix corresponding to the initial student model.

5. The method according to claim 1, characterized in that After constructing the manifold structure loss function based on the first feature and the second feature, the method further includes: Obtaining the overall loss function, wherein the overall loss function is composed of the manifold structure loss function, the weight of the manifold structure loss function, and the distillation classification loss function; Based on the overall loss function, the loss of the initial student model is calculated, and the model parameters of the initial student model are updated using a stochastic gradient descent algorithm.

6. The method according to claim 5, characterized in that The method of calculating the loss of the initial student model based on the overall loss function and updating the model parameters of the initial student model using a stochastic gradient descent algorithm further includes: Input different batches of sample image data, iteratively optimize the initial student model until the preset training conditions are met, stop optimizing the model, and obtain the trained target student model, wherein the training conditions include one of the number of iterations exceeding the preset upper limit value and the model loss value being less than the preset threshold value.

7. A model generation device, characterized in that: include: A first acquisition module is configured to acquire a trained teacher model and an initial student model, wherein both the teacher model and the initial student model are used for image classification; A second acquisition module is configured to acquire a first feature output by the teacher model and to acquire a second feature output by the initial student model; A construction module is configured to construct a manifold structure loss function based on the first feature and the second feature; A generation module is configured to train the initial student model through an overall loss function until it is determined that the features output by the initial student model have a similar manifold structure to the features output by the teacher model, and after the overall loss function converges, a target student model is obtained, wherein the overall loss function is composed of a distillation classification loss function and the manifold structure loss function; The construction module is configured to construct a first Laplace matrix corresponding to the teacher model using the first feature; and to construct a second Laplace matrix corresponding to the student model using the second feature; The manifold structure loss function is constructed based on a matrix difference between the first Laplacian matrix and the second Laplacian matrix.

8. An electronic device, characterized in that: include: A memory for storing executable instructions; as well as, A processor is used to execute the executable instructions with the memory to complete the operation of the model generation method described in any one of claims 1-6.

9. A computer-readable storage medium for storing computer-readable instructions, characterized in that: When the instructions are executed, the operations of the model generation method described in any one of claims 1-6 are performed.

Images