Large model training sample construction and student model training method for private data

Abstract

The invention relates to a large model training sample construction and student model training method and system for private data. The construction method comprises the steps that a client obtains an original sample; the client divides an original sample into a first sub-original sample and a second sub-original sample based on a division rule, and sends the first sub-original sample and the second sub-original sample to the first server and the second server; the first server and the second server respectively predict the first sub-original sample and the second sub-original sample based on a stored first sub-teacher model and a stored second sub-teacher model, and correspondingly generate and send a first prediction result and a second prediction result to a third server; the third server combines the first prediction result and the second prediction result based on a secret sharing protocol, and generates and sends a complete prediction result to the client; and the client obtains a pseudo label of the original sample according to the complete prediction result, and forms a training sample containing a label by the pseudo label and the original sample. According to the invention, the risk of data leakage is reduced, and the privacy of data is improved.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a method and system for constructing large model training samples and student model training for privacy data. Background Art

[0002] In the era of big data, the massive growth of data has led to a rapid increase in the demand for data transmission, but the transmission of large amounts of data will also lead to the risk of privacy leakage. There is a risk of privacy leakage at all stages of data collection, transmission and sharing. Especially when the amount of data surges, using clients and servers to interact with data to share and process data more efficiently has become the choice of more and more people. However, during the data interaction between the client and the server, the transmitted data will face attacks such as interception, tampering or abuse, resulting in the leakage of sensitive information. Therefore, it is necessary to provide a method and system for constructing large model training samples and student model training for privacy data. Summary of the invention

[0003] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a method for constructing a large model training sample for privacy data, a method and system for training a student model, so as to improve the problems that the prior art is prone to insufficient data validity and large resource occupation when performing data privacy protection.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for constructing a large model training sample for privacy data, wherein the training sample is used to train a student model, and the method comprises: a client obtains an original sample; the client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends the samples to a first server and a second server respectively; wherein the first server and the second server respectively store a first sub-teacher model and a second sub-teacher model of a trained teacher model, the first sub-teacher model and the second sub-teacher model have the same model architecture and different parameters, and an identity authentication mechanism is respectively set between the client and the first server and the second server; the first server and the second server respectively predict the first sub-original sample and the second sub-original sample based on the stored first sub-teacher model and the second sub-teacher model, and correspondingly generate and send a first prediction result and a second prediction result to a third server; the third server merges the first prediction result and the second prediction result based on a preset secret sharing protocol, generates and sends a complete prediction result to the client; the client obtains a pseudo-label of the original sample according to the complete prediction result, and forms a labeled training sample with the pseudo-label and the original sample.

[0005] In one embodiment of the present invention, the client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends them to a first server and a second server respectively, including: the client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule; the client authenticates the first server and sends the first sub-original sample to the first server after the authentication is passed; the client authenticates the second server and sends the second sub-original sample to the second server after the authentication is passed.

[0006] In one embodiment of the present invention, the first server and the second server predict the first sub-original sample and the second sub-original sample respectively based on the saved first sub-teacher model and the second sub-teacher model, generate and send the first prediction result and the second prediction result to the third server, including: the first server inputs the first sub-original sample into the first sub-teacher model to generate an initial first prediction result, and blinds the initial first prediction result according to the first blinding matrix, generates and sends the blinded first prediction result to the third server; wherein the first blinding matrix is ​​randomly generated by the third server and sent to the first server; the second server inputs the second sub-original sample into the second sub-teacher model to generate an initial second prediction result, and blinds the initial second prediction result according to the second blinding matrix, generates and sends the blinded second prediction result to the third server; the second blinding matrix is ​​randomly generated by the third server and sent to the second server.

[0007] In one embodiment of the present invention, when there is one teacher model, the third server merges the first prediction result and the second prediction result based on a secret sharing protocol, generates and sends a complete prediction result to the client, including: the third server merges the first prediction result and the second prediction result to obtain a comprehensive prediction result; the third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result to obtain a complete prediction result.

[0008] In one embodiment of the present invention, when there are multiple teacher models, each teacher model is trained based on a different training set, and each teacher model corresponds to a first blinding matrix and a second blinding matrix. The third server merges the first prediction result and the second prediction result based on a secret sharing protocol, generates and sends a complete prediction result to the client, including: for the first prediction result and the second prediction result generated by each teacher model: the third server merges the first prediction result and the second prediction result to obtain a comprehensive prediction result of the teacher model; the third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result to obtain the complete prediction result of the teacher model; the third server performs weighted voting on the complete prediction results of each teacher model, generates and sends the final complete prediction result to the client.

[0009] In one embodiment of the present invention, the third server performs weighted voting on the complete prediction results of each teacher model to generate and send the final complete prediction result to the client, including: the third server performs weighted voting on the complete prediction results of each teacher model to obtain an aggregated prediction result; the third server performs noise encryption processing on the aggregated prediction result to obtain the final complete prediction result.

[0010] In one embodiment of the present invention, the third server performs noise encryption processing on the aggregated prediction result to obtain a final complete prediction result, including: the third server determines the information entropy based on the confidence of each prediction category in the aggregated prediction result; the third server determines the noise coefficient based on the information entropy and the total amount of prediction categories; the third server adds the noise coefficient to the aggregated prediction result to obtain the final complete prediction result.

[0011] In one embodiment of the present invention, the client obtains a pseudo label of the original sample based on the complete prediction result, and forms a labeled training sample with the pseudo label and the original sample, including: the client selects the prediction category with the highest confidence from the complete prediction result as the pseudo label of the original sample; the client adds the pseudo label to the original sample to form a training sample.

[0012] In one embodiment of the present invention, a method for training a student model is also provided, the training method comprising: obtaining a training sample set; wherein the training samples in the training sample set are generated using any of the training sample construction methods described above; inputting the training sample set into a student model to be trained, performing multiple rounds of iterative training on the student model to be trained, and obtaining a trained student model.

[0013] In one embodiment of the present invention, a training sample construction system is also provided, the system comprising a client, a first server, a second server and a third server which are communicatively connected to each other: the client comprises: a data acquisition module for acquiring original samples; a sample segmentation module for dividing the original samples into a first sub-original sample and a second sub-original sample based on a preset division rule; a client communication module for sending the first sub-original sample and the second sub-original sample to the first server and the second server respectively, and receiving a complete prediction result sent by the third server; wherein the first server and the second server respectively store a first sub-teacher model and a second sub-teacher model of a trained teacher model, the model architectures of the first sub-teacher model and the second sub-teacher model are the same but the parameters are different, and an identity authentication mechanism is respectively set between the client and the first server and the second server; a sample construction module for obtaining a pseudo-label of the original sample according to the complete prediction result, and comparing the pseudo-label and the third server; The original samples constitute labeled training samples; the first server includes: a first prediction result generation module, which is used to predict the first sub-original sample based on the saved first sub-teacher model to generate a first prediction result; a first communication module, which is used to send the first prediction result to the third server, and receive the first original sub-sample sent by the client; the second server includes: a second prediction result generation module, which is used to predict the second sub-original sample based on the saved second sub-teacher model to generate a second prediction result; a second communication module, which is used to send the second prediction result to the third server, and receive the second original sub-sample sent by the client; the third server includes: a result merging module, which is used to merge the first prediction result and the second prediction result based on a preset secret sharing protocol to generate a complete prediction result; a third communication module, which is used to send the complete prediction result to the client, and receive the first prediction result sent by the first server and the second prediction result sent by the second server.

[0014] As described above, a method and system for constructing a large model training sample and training a student model for privacy data of the present invention has the following beneficial effects: the original sample is divided into two sub-samples and sent to the first server and the second server respectively. During the data transmission, the protection of data privacy is achieved through the identity authentication mechanism, and the security during the data transmission is improved. The first server and the second server use part of the model parameters to predict the two sub-samples to generate the first prediction result and the second prediction result. Since only part of the model parameters are used to predict the sub-samples, the probability of data leakage can be effectively reduced. The first server and the second server send the corresponding prediction results to the third server, and the third server merges the two prediction results according to the secret sharing protocol to obtain a complete prediction result and send it to the client, so that the client obtains the pseudo label of the original sample according to the prediction result for training the student model. Through this merging process, not only can the privacy of data and model parameters be effectively ensured, but also the prediction results of different sub-teacher models can be integrated, so that the prediction results finally generated are more accurate. This method of the present invention makes it impossible for a single server to access the complete data and model parameters, thereby effectively reducing the risk of data leakage and improving the privacy of data. In addition, since different servers process different data, the amount of data processed by each server is relatively small, which improves the problem of high load caused by large data volume during data transmission in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an application scenario of the method for constructing training samples provided in an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of a flow chart of a method for constructing a training sample provided by an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of the process of training a teacher model for federated learning provided in an embodiment of the present invention;

[0018] Figure 4 Shown is a structural block diagram of a system for constructing training samples provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0021] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0022] In the era of big data, the massive growth of data has accelerated the development of machine learning. In recent years, deep learning has been widely used as a frontier field in machine learning research. However, while deep learning has brought a lot of gains to scientific development, economic development and national development, its huge demand for data has also brought the risk of privacy leakage to society. In the application process of deep learning, there is a risk of privacy leakage in each stage of data collection, data sharing, model training and model use. Therefore, it is of great research significance and application value to protect privacy at each stage and effectively prevent privacy threats in the application process of deep learning.

[0023] The risks of privacy leakage mainly exist in the following four paths: first, unreliable data collectors may actively collect local privacy; second, when the privacy model is completely open to attackers, attackers can attack through white-box member reasoning; third, when the privacy model only predicts the results to the attacker, the attacker can perform a reconstruction attack through the black box; fourth, the attacker can restore the original data from the privacy data that is not strictly protected. The harm of privacy leakage ranges from the disclosure of personal privacy to the threat to the security of companies and even countries. Affected by this risk, data in some important fields, such as finance, medical care, and statistics, rarely have public data sets, which directly affects the development of machine learning in these fields. The data of each company and institution with private data is like an island and cannot be learned together.

[0024] However, most of the existing privacy protection focuses on data privacy protection. Due to the low interpretability of deep learning algorithms, there are relatively few studies on the privacy protection of existing models. The typical privacy protection methods for deep learning today are mainly differential privacy, homomorphic encryption, and secure multi-party computation. However, differential privacy is strict and comprehensive in protecting the query results of privacy, which also brings about the problem of excessive protection reducing data validity. The development of the above combination principle makes the application of differential privacy closer and closer to true differential privacy, which in turn improves the validity of data. However, deep learning has the characteristics of poor interpretability, especially when the attacker cannot access the model, the amount of noise added to the query results, that is, the data output, is a problem worth exploring. But on the other hand, we cannot only consider the current status of deep learning attack models without considering the future development of science. Otherwise, the current de-privatized data may be restored to privacy by more advanced technologies, such as the k-anonymity algorithm.

[0025] The present invention provides a method for constructing a large model training sample for private data, wherein the original sample is divided into two sub-samples and sent to a first server and a second server respectively. During data transmission, data privacy protection is achieved through an identity authentication mechanism, thereby improving the security during data transmission. The first server and the second server predict the two sub-samples using part of the model parameters to generate a first prediction result and a second prediction result. Since only part of the model parameters are used to predict the sub-samples, the probability of data leakage can be effectively reduced. The first server and the second server send the corresponding prediction results to a third server, which merges the two prediction results according to a secret sharing protocol to obtain a complete prediction result and sends it to a client, so that the client obtains a pseudo label of the original sample according to the prediction result for training a student model. Through this merging process, not only can the privacy of data and model parameters be effectively ensured, but also the prediction results of different sub-teacher models can be integrated, so that the prediction results finally generated are more accurate. This method of the present invention makes it impossible for a single server to access the complete data and model parameters, thereby effectively reducing the risk of data leakage and improving the privacy of data. In addition, since different servers process different data, the amount of data processed by each server is small, thereby improving the problem of high load caused by large amount of data during data transmission in the prior art.

[0026] See also Figure 1, the client divides the original sample into a first sub-original sample and a second sub-original sample, and sends them to the first server and the second server respectively. The first server inputs the first sub-original sample into the first sub-teacher model for prediction to obtain an initial first prediction result, and combines the initial first prediction result with the first blinding matrix sent by the third server to obtain a blinded first prediction result. Similarly, the second server inputs the second sub-original sample into the second sub-teacher model for prediction to obtain an initial second prediction result, and combines the initial second prediction result with the second blinding matrix sent by the third server to obtain a blinded second prediction result. The first and second servers respectively send the corresponding blinded prediction results to the third server, and the third server merges the two blinded prediction results and performs unblinding processing to obtain the final complete prediction result, and sends it to the client for training the student model.

[0027] See also Figure 2 , the method for constructing training samples includes the following steps:

[0028] S1. The client obtains the original sample.

[0029] The types of original samples include but are not limited to unlabeled data such as images, text, audio or video, and the original samples are consistent with the type of training data used by the teacher model. The original samples are used as training data for the student model, and the student model is supervised through the pseudo labels generated by the teacher model so that the student model can learn the knowledge of the teacher model.

[0030] It should be noted that when obtaining the original sample, an access control mechanism is also set to ensure the security and privacy of the data. Specifically, when obtaining the original sample, the access to the data can be restricted by querying the preset access control list through the identity authentication and authorization mechanism to ensure that only verified and authorized users or system entities can access the original sample data. Among them, in the access control list, corresponding access rules are set for preset users or user groups to allow or deny access requests to the original sample, thereby effectively controlling and managing data access rights. When the client needs to obtain the original sample, it sends a data request to the data provider through the gateway, and the data request is presented in the form of a data packet. After the gateway receives the requested data packet, it will check the target address, port number and other information of the data packet according to the preset access control list to ensure that the request made by the client complies with the preset access control rules. Only when the data request complies with the rules in the access control list will the gateway establish a connection between the client and the data provider. Among them, the specific content of the access control list can be adaptively set based on different security policies and business needs, and is not limited here. By setting the rules of the access control list, it can be ensured that the public network IP cannot access the data provider to ensure data security. Specifically, the access control list is preset with relevant rules according to the source address, destination address, port number and other information of the message. After the rules are set, the access control request list needs to be applied to the interface of the device to work properly. After the gateway receives the data packet, it checks each rule in the access control list one by one to determine whether it matches the current request packet: if it matches, the gateway grants or denies the client's access request according to the predefined permission level in the rule. On the contrary, if all the rules do not match the request packet, the request packet will be discarded and the client's access request will be denied. Further, after the data request passes the access control list check, the client will also use the identity authentication mechanism to verify the data provider to ensure that the identity of the data provider is authentic and reliable. After completing the identity authentication, the communication between the client and the data provider will be carried out through an encrypted channel to protect the privacy and integrity of the original sample during the transmission process. It can be understood that it is not limited to the acquisition of the original sample. In the process of the client sending data to the first server and the second server, and the first and second servers sending data to the third server, and the third server sending the final complete prediction results to the client, data privacy protection can be performed through the above access rules.

[0031] S2. The client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends them to the first server and the second server respectively; wherein the first server and the second server respectively store a first sub-teacher model and a second sub-teacher model of the trained teacher model, the first sub-teacher model and the second sub-teacher model have the same model architecture and different parameters, and an identity authentication mechanism is set between the client and the first server and the second server respectively.

[0032] After the teacher model is trained using data consistent with the original sample type, the parameters of the trained teacher model are divided into two parts to form a first sub-teacher model and a second sub-teacher model, and the first sub-teacher model is deployed on the first server, and the second sub-teacher model is deployed on the second server. It can be understood that the first sub-teacher model and the second sub-teacher model have the same model architecture, but each contains only a part of the parameters of the teacher model. Since the parameters of the two sub-teacher models are different, the two are independent of each other and cannot restore the original teacher model during the reasoning process of the model, thereby enhancing the privacy protection of the data. The client divides the original sample into a first sub-original sample and a second sub-original sample based on the data type and model requirements. It can be understood that when the client divides the first sub-original sample and the second sub-original sample, each sub-original sample will be adapted to the corresponding sub-teacher model. For example, the image is divided into two parts, the left half and the right half, and the first sub-teacher model and the second sub-teacher model are input respectively. After the client divides the original sample, the first sub-original sample is sent to the first server, the second sub-original sample is sent to the second server, and before sending the data, the legitimacy of the server is ensured through identity authentication to enhance transmission security. Since the first server and the second server use the corresponding sub-teacher model to calculate the received sub-original samples, each server can only access part of the data and part of the parameters of the teacher model, and cannot restore the complete original sample and teacher model. Therefore, in this way, data privacy is effectively guaranteed.

[0033] In an embodiment of the present invention, the client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends the samples to the first server and the second server respectively, including:

[0034] The client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule;

[0035] The client authenticates the first server, and sends the first sub-original sample to the first server after the authentication is passed;

[0036] The client authenticates the second server and sends the second sub-original sample to the second server after the authentication is successful.

[0037] Based on the preset division rules, the client divides the original sample into a first sub-original sample and a second sub-original sample for distributed processing and model training. It is understandable that the division rules can be adaptively set based on actual needs. Before sending the first sub-original sample to the first server, the client authenticates the first server by confirming the identity of the server, checking the encryption certificate, etc., to ensure the security of communication between the client and the first server. After the verification is passed, the client sends the first sub-original sample to the first server through a secure channel so that the first server performs relevant processing. Similarly, the client will authenticate the second server, and after the verification is passed, the second sub-original sample will be sent to the second server, waiting for the second server to perform corresponding processing. Through identity authentication, data security during communication between the client and each server is ensured to avoid data leakage during transmission.

[0038] S3. The first server and the second server predict the first sub-original sample and the second sub-original sample respectively based on the saved first sub-teacher model and the second sub-teacher model, and correspondingly generate and send the first prediction result and the second prediction result to the third server.

[0039] After receiving the first sub-original sample, the first server inputs the first sub-original sample into the first sub-teacher model to obtain the first prediction result. Similarly, the second server inputs the received second sub-original sample into the second sub-teacher model to obtain the second prediction result. Since each sub-teacher model only contains part of the parameters of the complete teacher model, the respective prediction results only reflect part of the information. The first server and the second server send their respective prediction results to the third server respectively, and the third server merges the two prediction results to generate a complete prediction result. Exemplarily, the client divides the image into the left half (i.e., the first sub-original sample) and the right half (i.e., the second sub-original sample). After the first sub-teacher model processes the first sub-original sample, it generates a first prediction result [0.6, 0.3, 0.1], which respectively represents the probability that the image belongs to the three categories of cat, dog, and bird. Similarly, after the second sub-teacher model processes the second sub-original sample, the second prediction result obtained is [0.5, 0.4, 0.1]. The third server combines the two prediction results by weighted averaging, simple averaging, etc., and obtains a complete prediction result of [0.55, 0.35, 0.1].

[0040] In one embodiment of the present invention, the first server and the second server predict the first sub-original sample and the second sub-original sample respectively based on the saved first sub-teacher model and the second sub-teacher model, and generate and send the first prediction result and the second prediction result to the third server, including:

[0041] The first server inputs the first sub-original sample into the first sub-teacher model to generate an initial first prediction result, and blinds the initial first prediction result according to a first blinding matrix, and generates and sends the blinded first prediction result to the third server; wherein the first blinding matrix is ​​randomly generated by the third server and sent to the first server;

[0042] The second server inputs the second sub-original sample into the second sub-teacher model to generate an initial second prediction result, and blinds the initial second prediction result according to a second blinding matrix, generates and sends the blinded second prediction result to the third server; the second blinding matrix is ​​randomly generated by the third server and sent to the second server.

[0043] After receiving the first sub-original sample sent by the client, the first server inputs it into the first sub-teacher model, predicts the first sub-original sample and generates an initial first prediction result. In order to protect privacy, the first server also uses the first blinding matrix provided by the third server to blind the initial first prediction result to obtain the first prediction result, as shown in formula (1):

[0044]

[0045] in, is the first prediction result corresponding to the i-th teacher model, <y i >0 is the initial first prediction result corresponding to the i-th teacher model, <m>0 is the first blinding matrix. Blinding refers to adding randomly generated noise to the prediction result. By adding noise, the prediction result is disturbed, so that even if the prediction result is intercepted, the third party cannot obtain the specific prediction value. The first server sends the first prediction result after blinding to the third server to ensure that the data remains private during transmission. Similarly, the second server also uses the second blinding matrix randomly generated by the third server to blind the initial second prediction result, thereby obtaining the blinded second prediction result and sending it to the third server. The second prediction result is shown in formula (2):

[0046]

[0047] in, is the second prediction result corresponding to the i-th teacher model, <y i >1 is the initial second prediction result corresponding to the i-th teacher model, <m>1 is the second blinding matrix. After receiving the blinded prediction results from the first and second servers, the third server recovers the complete prediction results through unblinding and sends them to the client. The blinding technology is used to protect the privacy of the prediction results, ensuring that even if the data is intercepted or accessed by a third party during the entire prediction process, the actual prediction value cannot be obtained or the content of the original data cannot be inferred.

[0048] S4. The third server combines the first prediction result and the second prediction result based on a preset secret sharing protocol, generates and sends a complete prediction result to the client.

[0049] To ensure privacy protection, the third server receives the first prediction result and the second prediction result. When merging, it does not directly access the specific content of the data, but uses a pre-set secret sharing protocol to calculate the data to ensure that no single server can access the complete prediction result during the calculation process. The third server generates the complete prediction result by weighted average or simple average and sends it to the client. In this way, the third server effectively guarantees data security and privacy, while realizing the merging of the prediction outputs of the teacher model.

[0050] In one embodiment of the present invention, when there is only one teacher model, the third server combines the first prediction result and the second prediction result based on the secret sharing protocol, generates and sends a complete prediction result to the client, including:

[0051] The third server combines the first prediction result and the second prediction result to obtain a comprehensive prediction result;

[0052] The third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result to obtain a complete prediction result.

[0053] When a teacher model is used, after the third server receives the first prediction result from the first server and the second prediction result from the second server, the first prediction result and the second prediction result are combined to obtain a comprehensive prediction result. The comprehensive prediction result contains a blinded noise value, so it is also necessary to subtract the first blinding matrix and the second blinding matrix from each prediction value of the comprehensive prediction result, as shown in formula (3):

[0054]

[0055] Among them, y is the complete prediction result. In this way, the influence of blinding noise is removed, and the unblinded complete prediction result is obtained and sent to the client for subsequent student model training.

[0056] In another embodiment of the present invention, when there are multiple teacher models, each teacher model is trained based on a different training set, each teacher model corresponds to a first blinding matrix and a second blinding matrix, and the third server merges the first prediction result and the second prediction result based on a secret sharing protocol, generates and sends a complete prediction result to the client, including:

[0057] For each teacher model, the first and second predictions are:

[0058] The third server combines the first prediction result and the second prediction result to obtain a comprehensive prediction result of the teacher model;

[0059] The third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result,

[0060] Get the complete prediction results of the teacher model;

[0061] The third server performs weighted voting on the complete prediction results of each teacher model, generates and sends the final complete prediction result to the client.

[0062] In order to enhance the diversity of data and improve the robustness of the student model, the present invention also combines the prediction results of multiple teacher models. It should be noted that when training the teacher model, the training samples of each teacher model are their own private data and do not intersect with each other. For each teacher model, the corresponding blinding matrix is ​​different. The third server receives the first prediction result of the first server and the second prediction result of the second server, and merges the two prediction results to obtain the comprehensive prediction result of the teacher model. And remove the first blinding matrix and the second blinding matrix to obtain the complete prediction result of the teacher model. After obtaining the complete prediction results of each teacher model, the third server uses weighted voting according to the preset weights of each teacher model to generate the final complete prediction results, as shown in formula (4):

[0063]

[0064] Among them, y is the final complete prediction result, N is the total number of teacher models, and y i is the complete prediction result of the i-th teacher model. After the third server obtains the final complete prediction result, it sends it to the client.

[0065] In one embodiment of the present invention, the third server performs weighted voting on the complete prediction results of each teacher model, generates and sends the final complete prediction result to the client, including:

[0066] The third server performs weighted voting on the complete prediction results of each teacher model to obtain an aggregated prediction result;

[0067] The third server performs noise encryption processing on the aggregated prediction result to obtain a final complete prediction result.

[0068] When there are multiple teacher models, there are also multiple corresponding complete prediction results. Each teacher model has a preset weight. The third server performs weighted summation on each complete prediction result according to the weight to combine the prediction results of all teacher models to obtain an aggregated prediction result. In order to protect the privacy of the data during transmission to the client, the third server also performs noise encryption on the aggregated prediction result. By adding noise to the aggregated prediction result based on the differential privacy method, even if a third party intercepts the result, it is impossible to recover the prediction content from it. This final complete prediction result not only contains the comprehensive judgment of multiple teacher models, but also improves the privacy of the data through noise encryption.

[0069] In one embodiment of the present invention, the third server performs noise encryption processing on the aggregated prediction result to obtain a final complete prediction result, including:

[0070] The third server determines information entropy according to the confidence of each prediction category in the aggregated prediction result;

[0071] The third server determines a noise coefficient according to the information entropy and the total amount of the predicted categories;

[0072] The third server adds the noise coefficient to the aggregated prediction result to obtain a final complete prediction result.

[0073] After the third server generates the aggregated prediction result, it calculates the information entropy according to the confidence of each prediction category in the result, as shown in formula (5):

[0074]

[0075] Among them, y is the aggregate prediction result, p j is the confidence of the jth prediction category, n is the total number of prediction categories, and S(y) is the information entropy. By calculating the information entropy, the information uncertainty in the prediction result can be effectively measured. After determining the information entropy, the third server calculates the noise coefficient for noise encryption based on the information entropy and the total number of prediction categories. The noise coefficient is used to adjust the intensity of the noise in order to strike a balance between privacy protection and prediction accuracy. The calculation of the noise coefficient is shown in formula (6):

[0076]

[0077] Wherein, N is the noise coefficient. The third server adds the noise coefficient N to the aggregate prediction result, thereby performing noise encryption. Specifically, as shown in formula (7):

[0078] y'=y+N*c*(ny) (7)

[0079] Where c is a parameter for controlling the amount of noise, and y' is the final complete prediction result. In this way, the third server adds noise to the aggregated prediction result, so that the third party cannot recover the original information from a single prediction result.

[0080] It should be noted that the teacher model in the present invention is trained by a federated learning algorithm, such as Figure 3 As shown in the figure, when the teacher model needs to be trained, the central server sends the initial parameters of the model to each client. Each client trains the model according to its own data and then uploads the trained model to the central server. The central server aggregates the model parameters according to the aggregation algorithm and sends them to each client. The above steps are repeated until each client trains the optimal teacher model. This algorithm is used to solve the data privacy and data island problems in machine learning. Instead of collecting the user's original data, it collects model parameters from each client, thereby protecting user privacy and solving data island problems.

[0081] S5. The client obtains a pseudo label of the original sample according to the complete prediction result, and forms a labeled training sample with the pseudo label and the original sample.

[0082] The third server sends the complete prediction result to the client, and the client selects the category with the highest confidence from the result as the pseudo label corresponding to the original sample. For example, if the complete prediction result is [0.55, 0.35, 0.1], the client will use category 0 (i.e. the category corresponding to 0.55) as the pseudo label to form a labeled training sample with the original sample set. After obtaining multiple labeled training samples in this way, the client uses this training sample to train the student model so that the student model can be effectively trained without labeled data.

[0083] In one embodiment of the present invention, the client obtains a pseudo label of the original sample according to the complete prediction result, and forms a labeled training sample with the pseudo label and the original sample, including:

[0084] The client selects the prediction category with the highest confidence from the complete prediction results as the pseudo label of the original sample;

[0085] The client adds the pseudo label to the original sample to form a training sample.

[0086] The client selects the prediction category with the highest confidence from the complete prediction results as the pseudo label of the original sample, and combines the pseudo label with the original sample to form a labeled training sample for training the student model, so that it can gradually learn and imitate the knowledge of the teacher model in the process of supervised learning.

[0087] In one embodiment of the present invention, a training method for a student model is also provided, the training method comprising:

[0088] Acquire a training sample set; wherein the training samples in the training sample set are generated using any of the training sample construction methods described above;

[0089] The training sample set is input into the student model to be trained, and the student model to be trained is subjected to multiple rounds of iterative training to obtain a trained student model.

[0090] When training the student model, the client inputs the constructed training sample set into the student model. The student model undergoes multiple rounds of iterative training using these samples and their pseudo labels. Each round of training adjusts parameters based on the sample data to gradually optimize the model's performance. In each round of iteration, the student model's prediction results are compared with the pseudo labels, the model's error is measured by calculating the loss function, and the model parameters are adjusted based on the error feedback. As the model is continuously iterated, the student model gradually learns the knowledge of the teacher model, and eventually a well-trained student model is obtained.

[0091] See also Figure 4 , the training sample construction system includes a client, a first server, a second server and a third server that are connected to each other in communication. Among them: the client includes: a data acquisition module 11 for acquiring the original sample. The sample segmentation module 12 is used to divide the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule. The client communication module 13 is used to send the first sub-original sample and the second sub-original sample to the first server and the second server respectively, and receive the complete prediction result sent by the third server; wherein the first server and the second server respectively store the first sub-teacher model and the second sub-teacher model of the trained teacher model, the model architecture of the first sub-teacher model and the second sub-teacher model are the same and the parameters are different, and the client is respectively provided with an identity authentication mechanism with the first server and the second server. The sample construction module 14 is used to obtain the pseudo-label of the original sample according to the complete prediction result, and the pseudo-label and the original sample constitute a labeled training sample. The first server includes: a first prediction result generation module 15 is used to predict the first sub-original sample based on the saved first sub-teacher model to generate a first prediction result. The first communication module 16 is used to send the first prediction result to the third server, and to receive the first original sub-sample sent by the client. The second server includes: a second prediction result generation module 17 is used to predict the second sub-original sample based on the saved second sub-teacher model to generate a second prediction result. The second communication module 18 is used to send the second prediction result to the third server, and to receive the second original sub-sample sent by the client. The third server includes: a result merging module 19 is used to merge the first prediction result and the second prediction result based on a preset secret sharing protocol to generate a complete prediction result. The third communication module 20 is used to send the complete prediction result to the client, and to receive the first prediction result sent by the first server and the second prediction result sent by the second server.

[0092] For the specific definition of the training sample construction system, please refer to the definition of the training sample construction method above, which will not be repeated here. Each module in the above training sample construction system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0093] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0094] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the method for constructing training samples described in various embodiments of the present application.

[0095] In summary, the present invention discloses a method and system for constructing a large model training sample and training a student model for privacy data, which divides the original sample into two sub-samples and sends them to the first server and the second server respectively. During the data transmission, the protection of data privacy is achieved through the identity authentication mechanism, which improves the security during the data transmission. The first server and the second server use part of the model parameters to predict the two sub-samples and generate the first prediction result and the second prediction result. Since only part of the model parameters are used to predict the sub-samples, the probability of data leakage can be effectively reduced. The first server and the second server send the corresponding prediction results to the third server, and the third server merges the two prediction results according to the secret sharing protocol to obtain a complete prediction result and send it to the client, so that the client obtains the pseudo label of the original sample according to the prediction result for training the student model. Through this merging process, not only can the privacy of data and model parameters be effectively ensured, but also the prediction results of different sub-teacher models can be integrated, so that the prediction results finally generated are more accurate. This method of the present invention makes it impossible for a single server to access the complete data and model parameters, thereby effectively reducing the risk of data leakage and improving the privacy of data. In addition, since different servers process different data, the amount of data processed by each server is relatively small, which improves the problem of large data volume leading to high load during data transmission in the prior art. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0096] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.< / m> < / m>

Claims

1. A method for constructing a large model training sample for private data, characterized in that: The training samples are used to train the student model, and the method includes: The client obtains the original sample; The client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends the samples to a first server and a second server respectively; wherein the first server and the second server respectively store a first sub-teacher model and a second sub-teacher model of a trained teacher model, the first sub-teacher model and the second sub-teacher model have the same model architecture and different parameters, and an identity authentication mechanism is set between the client and the first server and the second server respectively; The first server and the second server predict the first sub-original sample and the second sub-original sample respectively based on the saved first sub-teacher model and the second sub-teacher model, and correspondingly generate and send the first prediction result and the second prediction result to the third server; The third server combines the first prediction result and the second prediction result based on a preset secret sharing protocol, generates and sends a complete prediction result to the client; The client obtains a pseudo label of the original sample according to the complete prediction result, and forms a labeled training sample with the pseudo label and the original sample.

2. The method for constructing a large model training sample for private data according to claim 1, characterized in that: The client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule, and sends the samples to the first server and the second server respectively, including: The client divides the original sample into a first sub-original sample and a second sub-original sample based on a preset division rule; The client authenticates the first server, and sends the first sub-original sample to the first server after the authentication is passed; The client authenticates the second server and sends the second sub-original sample to the second server after the authentication is successful.

3. The method for constructing a large model training sample for private data according to claim 1, characterized in that: The first server and the second server predict the first sub-original sample and the second sub-original sample respectively based on the saved first sub-teacher model and the second sub-teacher model, and generate and send the first prediction result and the second prediction result to the third server, including: The first server inputs the first sub-original sample into the first sub-teacher model to generate an initial first prediction result, and blinds the initial first prediction result according to a first blinding matrix, and generates and sends the blinded first prediction result to the third server; wherein the first blinding matrix is ​​randomly generated by the third server and sent to the first server; The second server inputs the second sub-original sample into the second sub-teacher model to generate an initial second prediction result, and blinds the initial second prediction result according to a second blinding matrix, generates and sends the blinded second prediction result to the third server; the second blinding matrix is ​​randomly generated by the third server and sent to the second server.

4. The method for constructing a large model training sample for private data according to claim 3, characterized in that: When there is only one teacher model, the third server combines the first prediction result and the second prediction result based on the secret sharing protocol, generates and sends a complete prediction result to the client, including: The third server combines the first prediction result and the second prediction result to obtain a comprehensive prediction result; The third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result to obtain a complete prediction result.

5. The method for constructing a large model training sample for private data according to claim 3, characterized in that: When there are multiple teacher models, each teacher model is trained based on a different training set, and each teacher model corresponds to a first blinding matrix and a second blinding matrix. The third server merges the first prediction result and the second prediction result based on a secret sharing protocol, generates and sends a complete prediction result to the client, including: For each teacher model, the first and second predictions are: The third server combines the first prediction result and the second prediction result to obtain a comprehensive prediction result of the teacher model; The third server removes the first blinding matrix and the second blinding matrix from the comprehensive prediction result to obtain a complete prediction result of the teacher model; The third server performs weighted voting on the complete prediction results of each teacher model, generates and sends the final complete prediction result to the client.

6. The method for constructing a large model training sample for private data according to claim 5, characterized in that: The third server performs weighted voting on the complete prediction results of each teacher model, generates and sends the final complete prediction result to the client, including: The third server performs weighted voting on the complete prediction results of each teacher model to obtain an aggregated prediction result; The third server performs noise encryption processing on the aggregated prediction result to obtain a final complete prediction result.

7. The method for constructing a large model training sample for private data according to claim 6, characterized in that: The third server performs noise encryption processing on the aggregated prediction result to obtain a final complete prediction result, including: The third server determines information entropy according to the confidence of each prediction category in the aggregated prediction result; The third server determines a noise coefficient according to the information entropy and the total amount of the predicted categories; The third server adds the noise coefficient to the aggregated prediction result to obtain a final complete prediction result.

8. The method for constructing a large model training sample for private data according to claim 1, characterized in that: The client obtains a pseudo label of the original sample according to the complete prediction result, and forms a labeled training sample with the pseudo label and the original sample, including: The client selects the prediction category with the highest confidence from the complete prediction results as the pseudo label of the original sample; The client adds the pseudo label to the original sample to form a training sample.

9. A student model training method, characterized in that: The training method comprises: Acquire a training sample set; wherein the training samples in the training sample set are generated using the training sample construction method described in any one of claims 1 to 8; The training sample set is input into the student model to be trained, and the student model to be trained is subjected to multiple rounds of iterative training to obtain a trained student model.

10. A system for constructing training samples, characterized in that: The system includes a client, a first server, a second server and a third server which are communicatively connected to each other: The client comprises: A data acquisition module, used to obtain original samples; A sample segmentation module, configured to divide the original sample into a first sub-original sample and a second sub-original sample based on a preset segmentation rule; A client communication module, used for sending the first sub-original sample and the second sub-original sample to the first server and the second server respectively, and receiving the complete prediction result sent by the third server; wherein the first server and the second server respectively store the first sub-teacher model and the second sub-teacher model of the trained teacher model, the model architecture of the first sub-teacher model and the second sub-teacher model are the same but the parameters are different, and the client is respectively provided with an identity authentication mechanism with the first server and the second server; A sample construction module, used to obtain a pseudo label of the original sample according to the complete prediction result, and to form a labeled training sample with the pseudo label and the original sample; The first server comprises: A first prediction result generating module, used for predicting the first sub-original sample based on the saved first sub-teacher model to generate a first prediction result; A first communication module, configured to send the first prediction result to the third server, and receive the first original subsample sent by the client; The second server comprises: A second prediction result generating module, used for predicting the second sub-original sample based on the saved second sub-teacher model to generate a second prediction result; A second communication module, configured to send the second prediction result to the third server, and receive the second original subsample sent by the client; The third server comprises: A result merging module, used to merge the first prediction result and the second prediction result based on a preset secret sharing protocol to generate a complete prediction result; The third communication module is used to send the complete prediction result to the client, and receive the first prediction result sent by the first server and the second prediction result sent by the second server.

Images