Model joint training method and device, electronic equipment and storage medium

Abstract

The invention provides a model joint training method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a pseudo-twin network architecture comprising two sub-networks, and deploying the pseudo-twin network architecture at any two computing nodes of a distributed network; each computing node comprises corresponding private data; respectively pre-training the two sub-networks through the private data of the calculation nodes, comparing the performances of the pre-trained two sub-networks, and determining a teacher model and a student model in the two sub-networks; taking a prediction result of the teacher model as a soft label, and training a student model in combination with a real label; and exchanging the teacher model and the student model, and iteratively training the process of the student model until the performance of the two sub-networks reaches a preset performance standard. According to the method, the data security in the model joint training process can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of distributed machine learning, and specifically relates to a model joint training method, device, electronic device and storage medium. Background Art

[0002] Traditional large-scale data analysis relies on a distributed server architecture. Although the distributed server architecture provides powerful computing capabilities, it also has some limitations. For example, data transmission between servers may cause delays, and the load between servers may be unbalanced. These factors limit the effective use of computing resources in a decentralized environment.

[0003] To address these challenges, collaborative learning methods (such as federated learning) have emerged, aiming to enhance model performance by jointly training multiple data sources. During the joint training process, although the existing twin networks can work without directly sharing the original data, they still need to access paired input data during the training process, which may contain sensitive information. With data privacy and regulations becoming increasingly stringent, this direct access to data may lead to the risk of privacy leakage and fail to ensure the confidentiality of data during the training process. Summary of the invention

[0004] In order to solve the problem of poor data security in the model joint training process in the prior art, the present invention provides a model joint training method, device, electronic device and storage medium.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A model joint training method, comprising:

[0007] Build a pseudo-twin network architecture consisting of two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes in the distributed network; each computing node contains corresponding private data;

[0008] The two sub-networks are pre-trained respectively by using the private data of the computing nodes, the performance of the two pre-trained sub-networks is compared, and the teacher model and the student model in the two sub-networks are determined;

[0009] The prediction results of the teacher model are used as soft labels and combined with the real labels to train the student model;

[0010] The teacher model and the student model are exchanged, and the process of training the student model is iteratively repeated until the performance of the two sub-networks reaches the preset performance standard.

[0011] Optionally, the prediction results of the teacher model are used as soft labels and combined with the true labels to train the student model, including:

[0012] Input the target data set into the teacher model and the student model respectively, and obtain the predicted probability distribution of the teacher model and the student model respectively;

[0013] Compare the predicted probability distribution of the student model with the predicted probability distribution of the teacher model and the true label, and calculate the loss value; update the parameters of the student model based on the loss value.

[0014] Optionally, the predicted probability distribution of the student model is compared with the predicted probability distribution of the teacher model and the true label, respectively, and the loss value is calculated, including:

[0015] Calculate the KL divergence based on the predicted probability distribution of the teacher model and the predicted probability distribution of the student model; use the KL divergence as a distance metric and calculate the contrast loss through the contrast loss function;

[0016] Compare the predicted probability distribution of the student model with the true label and calculate the cross entropy loss through the cross entropy loss function;

[0017] The total loss value is obtained by weighted summing the cross entropy loss and the contrast loss.

[0018] Optionally, two sub-networks are pre-trained separately using private data of the computing node, including:

[0019] The private data of the two computing nodes are input into the two sub-networks respectively, and the predicted values ​​are obtained through forward propagation;

[0020] The cross entropy loss function is used to calculate the loss between the two sub-network predictions and the true labels of the private data.

[0021] Update the parameters of the two sub-networks through the loss value between the predicted values ​​of the two sub-networks and the true labels of the private data;

[0022] The process of iterative forward propagation, loss calculation and parameter update is repeated until the preset stop condition is reached to obtain two pre-trained sub-networks.

[0023] Optionally, in the process of pre-training the two sub-networks respectively by using the private data of the computing node, the process includes:

[0024] Perform principal component analysis on private data to extract principal component data and calculate orthogonal projection data of the principal component data;

[0025] The principal component data and orthogonal projection data are used alternately for parameter update.

[0026] Optionally, orthogonal projection data of the principal component data is calculated by singular value decomposition, including:

[0027] Centralized preprocessing of private data to obtain a data matrix;

[0028] Decompose the data matrix into a matrix consisting of left singular vectors, a diagonal matrix consisting of singular values, and the transpose of the matrix consisting of right singular vectors by singular value decomposition;

[0029] Select the first k principal components in the singular values, project the data into the principal component space by constructing a projection matrix, and obtain the orthogonal projection data.

[0030] The present invention also provides a model joint training device, characterized in that it comprises:

[0031] A construction module is used to construct a pseudo-twin network architecture containing two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes in the distributed network; each computing node contains corresponding private data;

[0032] The first training module is used to pre-train the two sub-networks respectively by using the private data of the computing nodes, compare the performances of the two pre-trained sub-networks, and determine the teacher model and the student model in the two sub-networks;

[0033] The second training module is used to train the student model using the prediction results of the teacher model as soft labels combined with the real labels;

[0034] The iteration module is used to exchange the teacher model and the student model and iterate the process of training the student model until the performance of the two sub-networks reaches the preset performance standard.

[0035] The model joint training method provided by the present invention has the following beneficial effects:

[0036] The present invention constructs a pseudo-twin network architecture including two sub-networks, and deploys the pseudo-twin network architecture on any two computing nodes of a distributed network; the two sub-networks are trained separately by private data in the computing nodes; the teacher model and the student model are determined in the two sub-networks, and the training of the student model is guided by the teacher model to achieve mutual learning of the two sub-networks; since the pseudo-twin network allows the two sub-networks to have the same or similar architecture but do not share weights, the pseudo-twin network can independently process the two inputs without directly sharing data or parameters, and work together in some way (such as comparing outputs); secondly, the teacher model and the student model learn from each other, and each model plays the dual role of the teacher model and the student model. On the one hand, it distills its own knowledge to other models; on the other hand, it also learns from the distillation of other models. This mutual learning enables the two models to learn useful information from each other's predictions without directly sharing data or parameters. Therefore, the present invention adopts a pseudo-twin network architecture, pre-trains the two sub-networks through private data, and enables the two models to learn from each other after pre-training through mutual knowledge distillation. During the joint training of the models, the training data and network parameters do not need to be shared. Only the real labels and predicted labels need to be shared during the output process of the teacher model guiding the student model, thereby reducing the risk of data leakage during the joint training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic flow chart of a model joint training method provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a pseudo twin network architecture provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of collaborative training of multiple models provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of the structure of a model joint training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0043] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0044] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0045] Before introducing the model joint training method provided by the present invention, the terms involved in the present invention are first explained.

[0046] Federated learning: It is a distributed machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. Federated learning uses encryption technology to achieve data transmission and interaction between multiple parties, thereby jointly modeling without sharing the original data, focusing on the security calculation between data exchanges. This technology can effectively solve the problem of data silos, allowing participants to jointly model without sharing data, thereby achieving AI collaboration.

[0047] Siamese network architecture: The twin network consists of two neural networks with the same structure and shared weights. The two networks process different input data and generate their own embedded representations. Then, the similarity between the two embedded representations (such as Euclidean distance or cosine similarity) is calculated to determine whether the two input samples are similar.

[0048] Although there are many secure and privacy-preserving federated learning methods, participants may still infer other participants' training data through uploaded parameters, facing different types of risks including member inference attacks, unintentional data leakage, inference reconstruction, and inference attacks based on generative adversarial networks (GANs). In addition, due to the inherent cryptographic properties of blockchain, if transactions within a block are illegally modified, the hash values ​​of all subsequent blocks will change, making it easier to detect and track the source of the modification. Therefore, federated learning still faces the threat of information leakage.

[0049] Based on this, the present invention provides a model joint training method, which aims to ensure that participants cannot access each other's training data or model parameters, eliminate the privacy risks associated with data or gradient sharing, and thus ensure the privacy and security of sensitive information during the training process.

[0050] For example, Figure 1 As shown, the method includes:

[0051] S1, build a pseudo-twin network architecture consisting of two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes in the distributed network.

[0052] The pseudo-twin network architecture consists of two independent networks, and the type of network can be selected according to the actual task, such as a convolutional neural network. It should be understood that at least two independent networks can be networks with different structures or networks with the same structure.

[0053] Figure 2 A schematic diagram of a pseudo-twin network architecture provided for an embodiment of the present invention. Exemplarily, the pseudo-twin network architecture consists of two sub-network models A and model B, and the two models are previously trained through a total loss function.

[0054] Furthermore, the pseudo-twin network architecture is deployed on any two computing nodes in a distributed network. It should be understood that there are generally multiple computing nodes in a distributed network, and each node is usually an independent computing unit. Although the nodes are independent, each node needs to work together to complete the tasks of the entire network.

[0055] Each computing node contains corresponding private data, which are only stored in the node terminal and are not shared with each other. The present invention does not limit the specific type of private data.

[0056] S2, pre-trains the two sub-networks respectively by computing the private data of the nodes, compares the performance of the two pre-trained sub-networks, and determines the teacher model and the student model in the two sub-networks.

[0057] Exemplarily, taking model A and model B as an example, the private data of the computing nodes where model A and model B are located are used as training data and input into model A and model B respectively. Model A and model B undergo forward propagation to obtain predicted values. Forward propagation includes a series of processing processes. Since the present invention does not limit the type of model, the forward propagation process can be determined according to the specific model selected in the actual task.

[0058] After model A and model B output the predicted values, the loss values ​​between the predicted values ​​of the two sub-networks and the true labels of the private data are calculated through their respective cross entropy loss functions, where the cross entropy loss function of model A is:

[0059]

[0060] in, Representation Class The i-th element of The logarithmic probability of the i-th category in the prediction result, represents the true label of the i-th element on the k-th sample, N represents the number of paired samples, and τ represents the temperature.

[0061] The cross entropy loss function of model B is:

[0062]

[0063] It should be understood that the meaning of the parameters involved in the cross entropy loss function of model B can refer to the relevant explanations of the parameters in the cross entropy loss function of model A.

[0064] Furthermore, the parameters of the two sub-networks are updated by back-propagating the loss value between the private data and the true label. Specifically, the gradient of the loss relative to the output is calculated, and then, through the chain rule, the gradient of the loss relative to the output is propagated back to each layer of the network to calculate the gradient of the loss relative to each parameter; after obtaining the gradient, the optimizer is used to update the parameters of the sub-network.

[0065] Then, the process of iterative forward propagation, loss value calculation and parameter update is repeated until a preset stop condition is reached (for example, a preset maximum number of iterations is reached), and two pre-trained sub-networks are obtained.

[0066] Furthermore, the performances of the two pre-trained sub-networks are compared, such as accuracy, recall, and F1 score, and the sub-network with better performance is used as the teacher model, and the sub-network with worse performance is used as the student model.

[0067] It should be understood that since the gradient of the loss function lies in the span of the input data points, the gradient with respect to the weight W can be expressed as:

[0068]

[0069] In the formula, σ i represents the coefficient of the linear combination, x i represents a data point, and N is the total number of paired data points. According to the principle of singular value decomposition (SVD), there is a reversible projection matrix P that projects the data points into the principal component space, evolving the above equation into the following formula:

[0070]

[0071] Where P -1 is the inverse matrix of P, To project the data points into the principal component space, reasonable components are retained. The approximate equation means that the gradient mainly depends on the principal components of the data points, while the non-principal components are ignored in the gradient calculation. To overcome the principal component bias and improve the generalization ability of the model, the present invention optimizes the model performance by alternately using the principal components of the data and the orthogonal projection technology in the gradient calculation.

[0072] Specifically, principal component analysis is performed on the private data to extract principal component data, orthogonal projection data of the principal component data is calculated, and the principal component data and orthogonal projection data are used alternately to update parameters.

[0073] For example, the principal component analysis is performed on the private data by the PCA method to extract the principal component of the data; then, the data is projected into a space orthogonal to the principal component to obtain non-principal component data; in the optimization process of stochastic gradient descent (SGD), the principal component data and the non-principal component data after orthogonal projection are used alternately to update the parameters. For example, the principal component data may be used to update the parameters in one iteration, and the non-principal component data may be used to update the parameters in the next iteration.

[0074] Optionally, the orthogonal projection data of the principal component data can be calculated by singular value decomposition, the process is as follows:

[0075] First, the private data is centrally preprocessed to obtain a data matrix to ensure that the mean of each dimension of the data is 0; then, the data matrix is ​​decomposed into a matrix composed of left singular vectors, a diagonal matrix composed of singular values, and the transpose of the matrix composed of right singular vectors through singular value decomposition. Finally, the first k principal components of the singular values ​​are selected, and the data are projected into the principal component space by constructing a projection matrix to obtain orthogonal projection data.

[0076] For example, assume that the dataset D A =[x 1 ,...,x N ]∈R N×d ,D B =[x N+1 ,...,x2N ]∈R N×d are the private data of model A and model B in the pseudo twin network, where d is the dimension of the feature data point. A =[x 1 ,...,x n ]∈R n ×d and X B =[x n+1 ,...,x 2n ]∈R n×d From D A , D B The private data of model A and model B selected in , n represents the number of data points. By performing SVD, we can get:

[0077]

[0078] Among them U A ,U B ∈R n×n ,V A ,V B ∈R d×d , U A and U B is the left singular vector matrix, ∑ A and∑ B is a diagonal matrix, V A and V B is the right singular vector matrix.

[0079] Take V A , V B The first k rows of P are used as the transformation matrix, A =V A [1:k,:]∈R d×N ,P B =V B [1:k,:]∈R k×d , and further get: Y A =P A D A ∈R k×N ,Y B =P B D B ∈R k×N .

[0080] By concatenating zero elements, we get the data points without the non-principal components:

[0081] By projecting back to conventional coordinates, and Can be used for training as principal component data.

[0082] The model parameters are updated according to the gradient information obtained by alternating projections until the model converges. In this way, the model can effectively learn different aspects of the data and improve generalization ability.

[0083] S3, uses the prediction results of the teacher model as soft labels and combines them with the true labels to train the student model.

[0084] It should be understood that S3 is a knowledge distillation process, that is, the teacher model guides the student model and encourages the student model to imitate the predicted probability distribution of the teacher model.

[0085] Exemplarily, the target data set is input into the teacher model and the student model respectively, and the predicted probability distribution of the teacher model and the student model are obtained respectively, and the predicted probability distribution of the teacher model is used as a soft label, and the predicted probability distribution of the student model is compared with the predicted probability distribution of the teacher model and the true label respectively, and the loss value is calculated; the parameters of the student model are updated based on the loss value. Among them, the target data set input into the teacher model and the student model is the same data set.

[0086] Among them, the predicted probability distribution of the student model is compared with the predicted probability distribution of the teacher model and the true label, and the process of calculating the loss value is as follows:

[0087] Step 1: Calculate the KL divergence (Kullback-Leibler Divergence) based on the predicted probability distribution of the teacher model and the predicted probability distribution of the student model; use the KL divergence as a distance metric and calculate the contrastive loss through the contrastive loss function.

[0088] Among them, the calculation method of KL divergence can refer to the existing technology, and the formula of the contrast loss function is as follows:

[0089]

[0090] Wherein, N represents the number of paired samples, y is a label, indicating whether two samples belong to the same category (usually y=1 represents a positive sample pair, i.e., the two samples belong to the same category; y=0 represents a negative sample pair, i.e., the two samples belong to different categories), d is the distance between the two samples in the feature space (in the present invention, KL divergence is used as the distance); m represents the minimum distance that the negative sample pair should maintain in the feature space.

[0091] It should be understood that the goal of the contrastive loss is to minimize the output difference between similar input pairs, while maximizing the output difference between different input pairs.

[0092] Step 2: Compare the predicted probability distribution of the student model with the true label, and calculate the cross entropy loss through the cross entropy loss function. The form of the cross entropy loss function is introduced in the above embodiment and will not be repeated here.

[0093] Step 3: Perform a weighted sum of the cross entropy loss and the contrast loss to get the total loss value. The formula is as follows:

[0094] L total =(1-α)L c +αL e ;

[0095] Among them, α is the weight coefficient, its value can be 0.1, L c is the contrast loss, L e is the cross entropy loss.

[0096] Furthermore, the parameters of the student model are updated based on the total loss value.

[0097] S4, the process of exchanging the teacher model and the student model and iteratively training the student model until the performance of the two sub-networks reaches the preset performance standard.

[0098] The above S2 and S3 are the process of distillation learning. After completing a round of distillation learning, the roles of the teacher model and the student model can be exchanged, that is, the original teacher model is used as the teacher model, and the original student model is used as the teacher model. The above distillation learning process is repeated until both models reach the preset performance standards.

[0099] In the above embodiment, mutual distillation is used to achieve mutual learning between the two models, thereby improving their respective performances.

[0100] In an embodiment of the present invention, when joint training of multiple (more than two) models is involved, a model can directly learn from its left and right neighbors through a pseudo twin network architecture, such as Figure 3 As shown in the figure, assuming that multiple models are distributed in a cycle, adjacent sub-networks are adjacent sub-networks in the cycle, and each sub-network can learn directly from its left and right neighbors (i.e., the left and right adjacent networks) through the pseudo-twin network architecture.

[0101] For example, if model A and model B are trained together, model A is first trained on its private data to generate a predicted probability distribution. Then, model B uses the output of model A as the target and adjusts its own parameters by minimizing the Kullback-Leibler divergence, thereby achieving mutual knowledge transfer. The resulting model B is expressed as The superscript "1" indicates the first round of training, and the subscript "ab" indicates the joint training between model A and model B. When a round of training is completed, model B receives new parameters by averaging the weights generated by two adjacent training pairs. and : In each iteration, model B updates itself based on its neighboring models (e.g. and ) After collaborative training, it averages the updated parameters from neighboring models with its own parameters to update its own model parameters. When this round ends, a new training iteration begins. This process is repeated until the model performance stabilizes.

[0102] In the above embodiment, by introducing the pseudo-twin network and the mutual knowledge distillation method, collaborative training is achieved without sharing data or parameters, thereby effectively protecting data privacy. By introducing contrast loss and knowledge distillation, the method enables the two network branches to learn from each other, thereby improving their respective performance. The traditional gradient descent method is often affected by the principal component of the input data when calculating the gradient, resulting in the non-principal component being ignored. By using the alternating iterative method and training with the principal component and orthogonal projection data, the principal component deviation problem in the gradient calculation is overcome, and the overall performance and generalization ability of the model are improved. In addition, the method can be easily extended to the collaborative training of multiple distributed models, allowing models of different architectures to transfer knowledge and enhancing the flexibility of model design. This scalability enables the method to be applicable to larger-scale distributed learning environments, further improving the training efficiency and performance of the model.

[0103] Next, the performance improvement effect of the present invention on the model is verified through experiments.

[0104] Experiments were conducted on multiple datasets (such as Fashion-MNIST, CIFAR-100, and PET) to verify that the joint training method of the model can effectively protect data privacy while improving model performance. Three representative convolutional neural networks were selected: ResNet, VGGNet, and GoogleNet, and the pseudo-twin network was tested on three commonly used datasets: Fashion-MNIST, CIFAR-100, and PET.

[0105] Table 1 Classification comparison results on Fashion-MNIST, CIFAR100 and PET

[0106]

[0107] In Table 1, accuracy (%) is used as the evaluation indicator. K≈950 means that 950 principal components of the projected data points (accounting for 30% of the total dimension) are used for training. When K≈950, the network obtains similar or better results than the network trained on normal data points.

[0108] Then, the training results of each model on the private data are compared with the results of training in the pseudo-twin framework of the present invention. The comparison results are shown in Table 2.

[0109] Table 2 Accuracy (%) and performance comparison results of two widely used deep learning convolutional neural network (CNN) architectures ResNet and VGGNet trained under the pseudo-twin architecture

[0110]

[0111] These data are based on experimental results conducted on different datasets (such as Fashion-MNIST, CIFAR100 and PET), and the unit is percentage (%). The left two columns of Table 2 are the results of the model trained on its own private data, and the right two columns are the results of the model after training under the pseudo-Siamese framework.

[0112] It can be seen from Table 2 that the performance of these models is improved when trained using the pseudo-twin framework of the present invention.

[0113] Based on the same inventive concept, the embodiment of the present invention also provides a model joint training device, such as Figure 4 As shown, the device comprises:

[0114] Construction module 41 is used to construct a pseudo-twin network architecture including two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes of the distributed network; each computing node contains corresponding private data.

[0115] The first training module 42 is used to pre-train the two sub-networks respectively by using the private data of the computing nodes, compare the performances of the two pre-trained sub-networks, and determine the teacher model and the student model in the two sub-networks.

[0116] The second training module 43 is used to train the student model using the prediction results of the teacher model as soft labels in combination with real labels.

[0117] The iteration module 44 is used to exchange the teacher model and the student model, and iteratively train the student model until the performance of the two sub-networks reaches a preset performance standard.

[0118] Each module in the above-mentioned model joint training device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0119] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the embodiment of the model joint training method. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0120] Furthermore, the present invention also provides a non-temporary computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, a non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the embodiment of the model joint training method can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0121] It will be appreciated by those skilled in the art that embodiments of the present invention may provide methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0125] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the invention has been described in detail in this specification and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the patent for the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belongs to the protection scope of the present invention.

Claims

1. A model joint training method, characterized in that: include: Build a pseudo-twin network architecture consisting of two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes in the distributed network; each computing node contains corresponding private data; The two sub-networks are pre-trained respectively by using the private data of the computing nodes, the performance of the two pre-trained sub-networks is compared, and the teacher model and the student model in the two sub-networks are determined; The prediction results of the teacher model are used as soft labels and combined with the real labels to train the student model; The teacher model and the student model are exchanged, and the process of training the student model is iteratively repeated until the performance of the two sub-networks reaches the preset performance standard.

2. The model joint training method according to claim 1, characterized in that: The prediction results of the teacher model are used as soft labels and combined with the real labels to train the student model, including: Input the target data set into the teacher model and the student model respectively, and obtain the predicted probability distribution of the teacher model and the student model respectively; Compare the predicted probability distribution of the student model with the predicted probability distribution of the teacher model and the true label, and calculate the loss value; update the parameters of the student model based on the loss value.

3. The model joint training method according to claim 2, characterized in that: Compare the predicted probability distribution of the student model with the predicted probability distribution of the teacher model and the true label, and calculate the loss value, including: Calculate the KL divergence based on the predicted probability distribution of the teacher model and the predicted probability distribution of the student model; use the KL divergence as a distance metric and calculate the contrast loss through the contrast loss function; Compare the predicted probability distribution of the student model with the true label and calculate the cross entropy loss through the cross entropy loss function; The total loss value is obtained by weighted summing the cross entropy loss and the contrast loss.

4. The model joint training method according to any one of claims 1 to 3, characterized in that: The two sub-networks are pre-trained using the private data of the computing nodes, including: The private data of the two computing nodes are input into the two sub-networks respectively, and the predicted values ​​are obtained through forward propagation; The cross entropy loss function is used to calculate the loss between the two sub-network predictions and the true labels of the private data. Update the parameters of the two sub-networks through the loss value between the predicted values ​​of the two sub-networks and the true labels of the private data; The process of iterative forward propagation, loss calculation and parameter update is repeated until the preset stop condition is reached to obtain two pre-trained sub-networks.

5. The model joint training method according to any one of claims 1 to 3, characterized in that: The process of pre-training the two sub-networks respectively through the private data of the computing nodes includes: Perform principal component analysis on private data to extract principal component data and calculate orthogonal projection data of the principal component data; The principal component data and orthogonal projection data are used alternately for parameter update.

6. The model joint training method according to claim 5, characterized in that: Calculate the orthogonal projection data of the principal component data through singular value decomposition, including: Centralized preprocessing of private data to obtain a data matrix; Decompose the data matrix into a matrix consisting of left singular vectors, a diagonal matrix consisting of singular values, and the transpose of the matrix consisting of right singular vectors by singular value decomposition; Select the first k principal components in the singular values, project the data into the principal component space by constructing a projection matrix, and obtain the orthogonal projection data.

7. A model joint training device, characterized in that: include: A construction module is used to construct a pseudo-twin network architecture containing two sub-networks, and deploy the pseudo-twin network architecture on any two computing nodes in the distributed network; each computing node contains corresponding private data; The first training module is used to pre-train the two sub-networks respectively by using the private data of the computing nodes, compare the performances of the two pre-trained sub-networks, and determine the teacher model and the student model in the two sub-networks; The second training module is used to train the student model using the prediction results of the teacher model as soft labels combined with the real labels; The iteration module is used to exchange the teacher model and the student model and iterate the process of training the student model until the performance of the two sub-networks reaches the preset performance standard.

8. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model joint training method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the model joint training method described in any one of claims 1-6 is implemented.

Images