Knowledge distillation model training method and device

Abstract

The present disclosure relates to the field of joint learning technology, and provides a knowledge distillation model training method and device. The method includes: in the case of receiving a request instruction from a model demander, determining the model provider corresponding to the model demander according to the request instruction; performing privacy calculation on the training data of the model demander and the teacher model of the model provider respectively; sharing the training data and the teacher model after privacy calculation between the model demander and the model provider; based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain interactive results; based on the interactive results and the training data, the knowledge distillation model of the model demander is trained to obtain the student model of the model demander.

Description

Technical Field

[0001] The present disclosure relates to the field of joint learning technology, and in particular to a knowledge distillation model training method and device. Background Art

[0002] In the training of knowledge distillation models, because some large enterprises have more industry data and expert knowledge, the models trained using the data of large enterprises have high accuracy; while for some small enterprises, because they lack sufficient data accumulation and expert knowledge (expert knowledge can be used to label data, and the more expert knowledge accumulated by one party, the more accurate the data labeling by that party), the models trained using the data of small enterprises have low accuracy. If small and large enterprises exchange their respective data, both small and large enterprises will worry about data security risks. There is currently no effective solution to this problem.

[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following technical problems in the related art: the model trained by a party lacking training data and expert knowledge has the problem of low accuracy. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a knowledge distillation model training method, device, electronic device and computer-readable storage medium to solve the problem in the prior art that the model trained by a party lacking training data and expert knowledge has low accuracy.

[0005] According to a first aspect of an embodiment of the present disclosure, a knowledge distillation model training method is provided, comprising: upon receiving a request instruction from a model demander, determining a model provider corresponding to the model demander according to the request instruction; performing privacy calculation on the training data of the model demander and the teacher model of the model provider respectively; sharing the training data and the teacher model after privacy calculation between the model demander and the model provider; based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain an interactive result; and performing knowledge distillation model training on the model demander based on the interactive result and the training data to obtain a student model of the model demander.

[0006] According to a second aspect of an embodiment of the present disclosure, a knowledge distillation model training device is provided, comprising: a determination module, configured to, upon receiving a request instruction from a model demander, determine a model provider corresponding to the model demander according to the request instruction; a privacy calculation module, configured to perform privacy calculation on the training data of the model demander and the teacher model of the model provider, respectively; a sharing module, configured to share the training data and the teacher model after privacy calculation between the model demander and the model provider; an interactive calculation module, configured to perform interactive calculation between the model demander and the model provider based on the training data and the teacher model after privacy calculation to obtain an interactive result; and a training module, configured to perform knowledge distillation model training of the model demander based on the interactive result and the training data to obtain a student model of the model demander.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the disclosed embodiments have the following beneficial effects: upon receiving a request instruction from a model demander, the model provider corresponding to the model demander is determined according to the request instruction; privacy calculation is performed on the training data of the model demander and the teacher model of the model provider respectively; the training data and the teacher model after privacy calculation are shared between the model demander and the model provider; based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain interactive results; based on the interactive results and the training data, the knowledge distillation model of the model demander is trained to obtain the student model of the model demander. The above technical means can solve the problem of low precision of the model trained by the party lacking training data and expert knowledge in the prior art, thereby improving the precision of the model trained by the party lacking training data and expert knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1It is a hardware structure block diagram of a computer terminal of a knowledge distillation model training method provided by an embodiment of the present disclosure;

[0012] Figure 2 It is a flowchart of a knowledge distillation model training method provided by an embodiment of the present disclosure;

[0013] Figure 3 is a structural schematic diagram of a knowledge distillation model training device provided by an embodiment of the present disclosure;

[0014] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0016] The method embodiments provided in the embodiments of the present disclosure can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 The hardware structure block diagram of a computer terminal of a knowledge distillation model training method according to an embodiment of the present disclosure is schematically shown. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD) and other processing devices and a memory 104 for storing data. Optionally, the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0017] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the knowledge distillation model training method in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0018] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0019] Figure 2 It is a flowchart of a knowledge distillation model training method provided in an embodiment of the present disclosure. Figure 2 The knowledge distillation model training method can be Figure 1 The server executes. Figure 2 As shown, the knowledge distillation model training method includes:

[0020] S201, upon receiving a request instruction from a model demander, determining a model provider corresponding to the model demander according to the request instruction;

[0021] S202, performing privacy calculation on the training data of the model demander and the teacher model of the model provider respectively;

[0022] S203, sharing the training data and teacher model after privacy calculation between the model demander and the model provider;

[0023] S204, based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain an interactive result;

[0024] S205, performing knowledge distillation model training of the model demander based on the interaction results and the training data to obtain a student model of the model demander.

[0025] The disclosed embodiments can be understood as utilizing the teacher model of the model provider to obtain the student model of the model demander. This process can be regarded as a process of knowledge distillation. The knowledge distillation model training of the model demander is performed based on the interaction results and the training data. This is both the knowledge distillation of the student model from the teacher model and the model training of the model demander.

[0026] The interactive result can be understood as the label corresponding to the training data. Based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation, which is the process of using the teacher model to determine the label corresponding to the training data. The embodiment of the disclosure adds privacy calculation to better protect the data security of the model demander and the model provider.

[0027] The execution subject server of the embodiment of the present disclosure can be regarded as a third party other than the model demander and the model provider. The model demander can be a small enterprise that lacks training data and expert knowledge, and the model provider can be a large enterprise that has a large amount of training data and expert knowledge. Expert knowledge can be used to label data. The richer the expert knowledge accumulated by one party, the more accurate the data labeling by the party. It should be noted that training data and expert knowledge are related. The more data a party accumulates, that is, the more training data, the more expert knowledge it has. Small enterprises often accumulate less data and have less expert knowledge, while large enterprises accumulate more data and have more expert knowledge. Expert knowledge can be understood as knowledge that distinguishes the category of each sample in the training data. Because small enterprises have less expert knowledge, they cannot label the training data with their own ability when training models, which directly leads to the problem of low accuracy of the models trained by small enterprises. In response to this problem, the expert knowledge owned by large enterprises can be used to label the training data of small enterprises. Specifically, the training data of small enterprises can be identified by using the models trained by large enterprises, and the labels of the training data of small enterprises can be obtained. However, if this method is simply adopted, there are data security risks. The disclosed embodiment proposes a method that can use the expert knowledge of large enterprises to label the training data of small enterprises while avoiding the problem of data security risks.

[0028] According to the technical solution provided by the embodiment of the present disclosure, when a request instruction is received from a model demander, the model provider corresponding to the model demander is determined according to the request instruction; privacy calculation is performed on the training data of the model demander and the teacher model of the model provider respectively; the training data and the teacher model after privacy calculation are shared between the model demander and the model provider; based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain interactive results; based on the interactive results and the training data, the knowledge distillation model of the model demander is trained to obtain the student model of the model demander. The above technical means can solve the problem of low accuracy of the model trained by the party lacking training data and expert knowledge in the prior art, thereby improving the accuracy of the model trained by the party lacking training data and expert knowledge.

[0029] In step S202, privacy calculations are performed on the training data of the model demander and the teacher model of the model provider, respectively, including: randomly generating a first random number by the model demander, and randomly generating a second random number by the model provider; based on the first random number and the second random number, privacy calculations are performed on the training data of the model demander and the teacher model of the model provider.

[0030] Here is a simple example: the model demander randomly generates a first random number, the model provider randomly generates a second random number, and the model demander and the model provider obtain each other's random numbers through interaction; on the model demander, the first random number is subtracted from the training data, and the second random number is added to obtain the first data, which is the training data after privacy calculation; on the model provider, the second random number is subtracted from the teacher model, and the first random number is added to obtain the second data, which is the teacher model after privacy calculation. Among them, the teacher model minus the second random number can be understood as the model data corresponding to the teacher model minus the second random number.

[0031] In step S201, when a request instruction from a model demander is received, a model provider corresponding to the model demander is determined according to the request instruction, including: parsing the request instruction to obtain at least one of the following: the training task of the model demander, the type of training data, and the purpose of the student model; determining the model provider corresponding to the model demander according to at least one of the following: the training task of the model demander, the type of training data, and the purpose of the student model.

[0032] The model demander is considered as a small enterprise that lacks training data and expert knowledge, and the model provider is considered as a large enterprise that has a large amount of training data and expert knowledge. There are many small and large enterprises, and it is necessary to select the large enterprise that is suitable for the small enterprise. The model provider can be selected based on one or more of the training tasks of the model demander, the type of training data, and the purpose of the student model. For example, a large enterprise with the same type of training data as the model demander is selected as the model provider.

[0033] The disclosed embodiments can be applied to any scenario where a party lacks training data and expert knowledge in knowledge distillation model training, such as indoor security inspection scenarios, face recognition scenarios, etc. For example, in indoor security inspection scenarios, model training of a small vendor providing security inspection services can be done with the help of data from a large vendor providing security inspection services.

[0034] In step S203, a multi-party computing protocol is used to perform privacy calculations on the training data and the teacher model respectively to obtain the training data and the teacher model after privacy calculations.

[0035] Among them, the multi-party computing protocol includes: PrivC protocol and ABY protocol. The multi-party computing protocol is an MPC protocol, including: PrivC protocol and ABY protocol. MPC is an important branch of cryptography, which aims to solve the problem of privacy-protecting collaborative computing between a group of mutually distrusting participants, and provide data demanders with multi-party collaborative computing capabilities without leaking original data. The first data and the second data are also calculated using the multi-party computing protocol.

[0036] For example, the model demander randomly generates a first random number, and the model provider randomly generates a second random number. The model demander and the model provider obtain each other's random numbers through interaction; on the model demander, the first random number is subtracted from the training data, and the second random number is added to obtain the first data, that is, the training data after privacy calculation; on the model provider, the second random number is subtracted from the teacher model, and the first random number is added to obtain the second data, that is, the teacher model after privacy calculation. Privacy calculation can be, the first data plus the second data, to obtain the interaction result, the interaction result includes the training data and the teacher model, and then the target label corresponding to the training data is determined based on the interaction result (the interaction result can be understood as the label corresponding to the training data, and can also be understood as the result of the present disclosure embodiment including the training data and the teacher model).

[0037] After executing step S201, that is, when receiving the request instruction from the model demander, after determining the model provider corresponding to the model demander according to the request instruction, the method also includes: obtaining data features corresponding to the training data of the model demander; inputting the data features into the teacher model of the model provider to obtain feature labels corresponding to the data features; determining the target labels corresponding to the training data based on the data features and the feature labels; and using the training data and the target labels to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

[0038] The training data is data with many pictures as samples, and the data features can be commonly used picture features, such as directional gradient histogram features. The disclosed embodiment may not be used for third parties, and directly utilizes the interaction between the model demander and the model provider. Because the data features are representative (the data features of the training data of many parties have something in common, the difference is the training data), for the model demander, the privacy level of the data features is lower than that of the training data, so the data features corresponding to the training data can be extracted, and the data features can be sent to the model provider. The model provider inputs the data features into the teacher model to obtain the feature labels corresponding to the data features, and then compares the data features corresponding to the feature labels with the data features corresponding to the training data, so as to achieve the pairing of each label in the feature label and each sample in the training data, and thus determine each label in the target label corresponding to each sample in the training data. Both the target label and the feature label can be understood as a label set.

[0039] After executing step S201, that is, when receiving the request instruction from the model demander, after determining the model provider corresponding to the model demander according to the request instruction, the method also includes: obtaining encrypted data corresponding to the model demander, wherein the encrypted data is the result of the training data being processed by homomorphic encryption technology; obtaining the encrypted model corresponding to the model provider, wherein the encrypted model is the result of the teacher model being processed by homomorphic encryption technology; inputting the encrypted data into the encrypted model to obtain an encrypted label; decrypting the encrypted label using homomorphic encryption technology to obtain a target label corresponding to the training data; and training the knowledge distillation model of the model demander using the training data and the target label to obtain the student model of the model demander.

[0040] The disclosed embodiment may not be used for a third party, but directly utilizes the interaction between the model demander and the model provider. Homomorphic encryption technology is an existing technology, and the disclosed embodiment is the first to apply homomorphic encryption technology to model training based on two-party interaction.

[0041] In step S205, the knowledge distillation model of the model demander is trained based on the interaction results and the training data to obtain the student model of the model demander, including: using the cross entropy loss function to calculate the loss value corresponding to the training data and the interaction results; based on the loss value, the knowledge distillation model of the model demander is trained to obtain the student model of the model demander.

[0042] In this embodiment, the interaction result can be understood as the label corresponding to the training data. Based on the loss value, the model can be updated using the gradient back propagation method.

[0043] In step S205, the knowledge distillation model of the model demander is trained based on the interaction results and the training data to obtain the student model of the model demander, including: according to the request instruction, the training data and the interaction results are screened to obtain the screening results; and the knowledge distillation model of the model demander is trained using the screening results to obtain the student model of the model demander.

[0044] Because the training data of the model demander may contain data that does not meet the current training task, these data that do not meet the current training task will have a bad impact on the model training, so the disclosed embodiment performs data screening on the training data and target labels according to the request instruction, and uses the data after data screening to train the knowledge distillation model of the model demander. The screening result is the data after data screening. The request instruction includes this training task.

[0045] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0046] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0047] Figure 3 Schematic diagram of a knowledge distillation model training device provided by an embodiment of the present disclosure. Figure 3 As shown, the knowledge distillation model training device includes:

[0048] The determination module 301 is configured to determine the model provider corresponding to the model demander according to the request instruction when receiving the request instruction from the model demander;

[0049] The privacy calculation module 302 is configured to perform privacy calculation on the training data of the model demander and the teacher model of the model provider respectively;

[0050] A sharing module 303 is configured to share the training data and the teacher model after privacy calculation between the model demander and the model provider;

[0051] The interactive computing module 304 is configured to perform interactive computing between the model demander and the model provider based on the training data and the teacher model after the privacy computing, and obtain the interactive result;

[0052] The training module 305 is configured to perform knowledge distillation model training of the model demander based on the interaction results and training data to obtain the student model of the model demander. The interaction results can be understood as the labels corresponding to the training data. Based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculations, which is the process of using the teacher model to determine the labels corresponding to the training data. The addition of privacy calculation in the disclosed embodiment can better protect the data security of the model demander and the model provider.

[0053] The execution subject server of the embodiment of the present disclosure can be regarded as a third party other than the model demander and the model provider. The model demander can be a small enterprise that lacks training data and expert knowledge, and the model provider can be a large enterprise that has a large amount of training data and expert knowledge. Expert knowledge can be used to label data. The richer the expert knowledge accumulated by one party, the more accurate the data labeling by the party. It should be noted that training data and expert knowledge are related. The more data a party accumulates, that is, the more training data, the more expert knowledge it has. Small enterprises often accumulate less data and have less expert knowledge, while large enterprises accumulate more data and have more expert knowledge. Expert knowledge can be understood as knowledge that distinguishes the category of each sample in the training data. Because small enterprises have less expert knowledge, they cannot label the training data with their own ability when training models, which directly leads to the problem of low accuracy of the models trained by small enterprises. In response to this problem, the expert knowledge owned by large enterprises can be used to label the training data of small enterprises. Specifically, the training data of small enterprises can be identified by using the models trained by large enterprises, and the labels of the training data of small enterprises can be obtained. However, if this method is simply adopted, there are data security risks. The disclosed embodiment proposes a method that can use the expert knowledge of large enterprises to label the training data of small enterprises while avoiding the problem of data security risks.

[0054] According to the technical solution provided by the embodiment of the present disclosure, when a request instruction is received from a model demander, the model provider corresponding to the model demander is determined according to the request instruction; privacy calculation is performed on the training data of the model demander and the teacher model of the model provider respectively; the training data and the teacher model after privacy calculation are shared between the model demander and the model provider; based on the training data and the teacher model after privacy calculation, the model demander and the model provider perform interactive calculation to obtain interactive results; based on the interactive results and the training data, the knowledge distillation model of the model demander is trained to obtain the student model of the model demander. The above technical means can solve the problem of low accuracy of the model trained by the party lacking training data and expert knowledge in the prior art, thereby improving the accuracy of the model trained by the party lacking training data and expert knowledge.

[0055] Optionally, the privacy calculation module 302 is also configured to randomly generate a first random number through the model demander and randomly generate a second random number through the model provider; based on the first random number and the second random number, perform privacy calculation on the training data of the model demander and the teacher model of the model provider.

[0056] Here is a simple example: the model demander randomly generates a first random number, the model provider randomly generates a second random number, and the model demander and the model provider obtain each other's random numbers through interaction; on the model demander, the first random number is subtracted from the training data, and the second random number is added to obtain the first data, which is the training data after privacy calculation; on the model provider, the second random number is subtracted from the teacher model, and the first random number is added to obtain the second data, which is the teacher model after privacy calculation. Among them, the teacher model minus the second random number can be understood as the model data corresponding to the teacher model minus the second random number.

[0057] Optionally, the determination module 301 is also configured to parse the request instruction to obtain at least one of the following: the training task of the model demander, the type of training data and the purpose of the student model; and determine the model provider corresponding to the model demander based on at least one of the following: the training task of the model demander, the type of training data and the purpose of the student model.

[0058] The model demander is considered as a small enterprise that lacks training data and expert knowledge, and the model provider is considered as a large enterprise that has a large amount of training data and expert knowledge. There are many small and large enterprises, and it is necessary to select the large enterprise that is suitable for the small enterprise. The model provider can be selected based on one or more of the training tasks of the model demander, the type of training data, and the purpose of the student model. For example, a large enterprise with the same type of training data as the model demander is selected as the model provider.

[0059] The disclosed embodiments can be applied to any scenario where a party lacks training data and expert knowledge in knowledge distillation model training, such as indoor security inspection scenarios, face recognition scenarios, etc. For example, in indoor security inspection scenarios, model training of a small vendor providing security inspection services can be done with the help of data from a large vendor providing security inspection services.

[0060] Optionally, the sharing module 303 is also configured to use a multi-party computing protocol to perform privacy calculations on the trained data and the teacher model respectively, to obtain the training data and the teacher model after privacy calculations.

[0061] Among them, the multi-party computing protocol may include: PrivC (custom) protocol and ABY (collaborative extension) protocol.

[0062] The multi-party computing protocol is an MPC protocol, including: PrivC protocol and ABY protocol. MPC is an important branch of cryptography, which aims to solve the problem of privacy-protecting collaborative computing between a group of mutually distrusting participants, and provide multi-party collaborative computing capabilities for data demanders without leaking original data. The first data and the second data are also calculated using the multi-party computing protocol.

[0063] For example, the model demander randomly generates a first random number, and the model provider randomly generates a second random number. The model demander and the model provider obtain each other's random numbers through interaction; on the model demander, the first random number is subtracted from the training data, and the second random number is added to obtain the first data, that is, the training data after privacy calculation; on the model provider, the second random number is subtracted from the teacher model, and the first random number is added to obtain the second data, that is, the teacher model after privacy calculation. Privacy calculation can be, the first data plus the second data, to obtain the interaction result, the interaction result includes the training data and the teacher model, and then the target label corresponding to the training data is determined based on the interaction result (the interaction result can be understood as the label corresponding to the training data, and can also be understood as the result of the present disclosure embodiment including the training data and the teacher model).

[0064] Optionally, the determination module 301 is also configured to obtain data features corresponding to the training data of the model demander; input the data features into the teacher model of the model provider to obtain feature labels corresponding to the data features; determine the target labels corresponding to the training data based on the data features and the feature labels; and use the training data and the target labels to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

[0065] The training data is data with many pictures as samples, and the data features can be commonly used picture features, such as directional gradient histogram features. The disclosed embodiment may not be used for third parties, and directly utilizes the interaction between the model demander and the model provider. Because the data features are representative (the data features of the training data of many parties have something in common, the difference is the training data), for the model demander, the privacy level of the data features is lower than that of the training data, so the data features corresponding to the training data can be extracted, and the data features can be sent to the model provider. The model provider inputs the data features into the teacher model to obtain the feature labels corresponding to the data features, and then compares the data features corresponding to the feature labels with the data features corresponding to the training data, so as to achieve the pairing of each label in the feature label and each sample in the training data, and thus determine each label in the target label corresponding to each sample in the training data. Both the target label and the feature label can be understood as a label set.

[0066] Optionally, the determination module 301 is also configured to obtain encrypted data corresponding to the model demander, wherein the encrypted data is the result of the training data being processed by homomorphic encryption technology; obtain the encrypted model corresponding to the model provider, wherein the encrypted model is the result of the teacher model being processed by homomorphic encryption technology; input the encrypted data into the encrypted model to obtain an encrypted label; use the homomorphic encryption technology to decrypt the encrypted label to obtain a target label corresponding to the training data; use the training data and the target label to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

[0067] The disclosed embodiment may not be used for a third party, but directly utilizes the interaction between the model demander and the model provider. Homomorphic encryption technology is an existing technology, and the disclosed embodiment is the first to apply homomorphic encryption technology to the knowledge distillation model training based on two-party interaction.

[0068] Optionally, the training module 305 is also configured to calculate the loss value corresponding to the training data and the interaction result using the cross entropy loss function; and perform knowledge distillation model training of the model demander based on the loss value to obtain the student model of the model demander.

[0069] In this embodiment, the interaction result can be understood as the label corresponding to the training data. Based on the loss value, the model can be updated using the gradient back propagation method.

[0070] Optionally, the training module 305 is also configured to perform data screening on the training data and interaction results according to the request instruction to obtain screening results; and use the screening results to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

[0071] Because the training data of the model demander may contain data that does not meet the current training task, these data that do not meet the current training task will have a bad impact on the model training, so the disclosed embodiment performs data screening on the training data and target labels according to the request instruction, and uses the data after data screening to train the knowledge distillation model of the model demander. The screening result is the data after data screening. The request instruction includes this training task.

[0072] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0073] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0074] Exemplarily, the computer program 403 may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to complete the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 403 in the electronic device 4.

[0075] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0076] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0077] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device. The memory 402 may also be used to temporarily store data that has been output or is to be output.

[0078] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0081] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0085] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A knowledge distillation model training method, characterized in that: include: In case of receiving a request instruction from a model demander, determining a model provider corresponding to the model demander according to the request instruction; Performing privacy calculations on the training data of the model demander and the teacher model of the model provider respectively; Sharing the training data and teacher model after the privacy calculation between the model demander and the model provider; Based on the training data and the teacher model after the privacy calculation, the model demander and the model provider perform interactive calculation to obtain an interactive result; Performing knowledge distillation model training of the model demander based on the interaction result and the training data to obtain a student model of the model demander; Among them, privacy calculation is performed on the training data of the model demander and the teacher model of the model provider respectively, including: randomly generating a first random number by the model demander, randomly generating a second random number by the model provider, and the model demander and the model provider obtain each other's random numbers through interaction; on the model demander, subtracting the first random number from the training data, and adding the second random number to obtain the training data after privacy calculation; on the model provider, subtracting the second random number from the teacher model, and adding the first random number to obtain the teacher model after privacy calculation.

2. The method according to claim 1, characterized in that In the case of receiving a request instruction from a model demander, determining a model provider corresponding to the model demander according to the request instruction includes: Parsing the request instruction to obtain at least one of the following: the training task of the model demander, the type of the training data, and the purpose of the student model; The model provider corresponding to the model demander is determined based on at least one of the following: the training task of the model demander, the type of the training data, and the purpose of the student model.

3. The method according to claim 1, characterized in that The privacy calculation is performed on the training data of the model demander and the teacher model of the model provider respectively, including: The privacy calculation is performed on the training data and the teacher model respectively using a multi-party computing protocol to obtain the training data and the teacher model after the privacy calculation.

4. The method according to claim 1, characterized in that: In the case of receiving a request instruction from a model demander, after determining a model provider corresponding to the model demander according to the request instruction, the method further includes: Obtaining encrypted data corresponding to the model demander, wherein the encrypted data is the result of the training data being processed by homomorphic encryption technology; Obtaining an encrypted model corresponding to the model provider, wherein the encrypted model is a result of the teacher model being processed by the homomorphic encryption technology; Inputting the encrypted data into the encryption model to obtain an encrypted label; Decrypting the encrypted label using the homomorphic encryption technology to obtain a target label corresponding to the training data; The training data and the target label are used to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

5. The method according to claim 1, characterized in that The performing of knowledge distillation model training of the model demander based on the interaction result and the training data to obtain a student model of the model demander includes: Calculating the loss value corresponding to the training data and the interaction result using a cross entropy loss function; The knowledge distillation model of the model demander is trained based on the loss value to obtain the student model of the model demander.

6. The method according to claim 1, characterized in that The performing of knowledge distillation model training of the model demander based on the interaction result and the training data to obtain a student model of the model demander includes: According to the request instruction, the training data and the interaction result are screened to obtain a screening result; The screening results are used to train the knowledge distillation model of the model demander to obtain the student model of the model demander.

7. A knowledge distillation model training device, characterized in that: include: A determination module is configured to, upon receiving a request instruction from a model demander, determine a model provider corresponding to the model demander according to the request instruction; A privacy computing module is configured to perform privacy computing on the training data of the model demander and the teacher model of the model provider respectively; A sharing module, configured to share the training data and the teacher model after the privacy calculation between the model demander and the model provider; An interactive computing module is configured to perform interactive computing between the model demander and the model provider based on the training data and the teacher model after the privacy computing to obtain an interactive result; A training module is configured to perform knowledge distillation model training of the model demander based on the interaction result and the training data to obtain a student model of the model demander; The privacy computing module is further configured to randomly generate a first random number through the model demander, and randomly generate a second random number through the model provider, and the model demander and the model provider obtain each other's random number through interaction; At the model demander, the first random number is subtracted from the training data, and the second random number is added to obtain the training data after privacy calculation; On the model provider side, the teacher model is subtracted from the second random number and then added with the first random number to obtain a teacher model after privacy calculation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images