Model training data transmission method and device, electronic equipment and medium
By using reversible perturbation matrix to process word vector data in large model training, the problem of privacy leakage during model training is solved, and the security and privacy protection of data transmission are improved. It is suitable for distributed and multi-party collaborative training scenarios.
Patent Information
- Application Number
- CN202411007283.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-06
AI Technical Summary
During the training process of big model, although data filtering and desensitization can protect user privacy to a certain extent, there are still potential privacy risks, especially the update of model parameters may leak information about the training data, resulting in privacy leakage.
By obtaining the current time data during the model training process, a reversible perturbation matrix is generated, and the word vector data is processed to generate transmission data, ensuring that the original data is not directly leaked during the transmission process, and only the receiver solves the inverse through the same time factor The original data can only be restored after the matrix.
It improves security during data transmission, ensures privacy protection of word vector data during transmission, reduces the risk of privacy leakage, and is suitable for distributed training and multi-party collaborative training scenarios.
Smart Images

Figure CN119089486A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data transmission, and in particular to a method and device, electronic equipment, and medium for transmitting model training data. Background Art
[0002] With the rapid development of artificial intelligence (AI) and deep learning technologies, large-scale models are increasingly used in various fields. The training of these large models usually requires a large amount of data, which often contains users' personal privacy information. Therefore, it is particularly important to ensure the secure transmission and privacy protection of data during the training of large models.
[0003] In practical applications, the training data for large models can come from a variety of different sources, including but not limited to the following:
[0004] Public datasets: Many research projects use public datasets for model training. These datasets are usually anonymized or desensitized to protect personal privacy.
[0005] Internal enterprise data: Many enterprises have massive amounts of internal data, which may contain sensitive information such as users’ personal information and transaction records.
[0006] Collaborative data: Companies may collaborate on data sharing and use their own data for model training. In this case, secure data transmission and privacy protection are crucial.
[0007] When processing this data, data filtering and desensitization operations are usually required to reduce the risk of leakage of sensitive information.
[0008] In practical applications, data filtering refers to filtering out information useful for model training from raw data and removing irrelevant or sensitive data. This helps reduce noise and unnecessary interference in model training while protecting user privacy.
[0009] Data desensitization is a common data protection method that anonymizes or encrypts data, making it difficult to restore the original information even if the data is leaked. Common desensitization technologies include data encryption, data replacement, data perturbation, etc.
[0010] Although data filtering and desensitization can protect user privacy to a certain extent, there are still some potential privacy risks during model training. For example, the update of model parameters may leak some information about the training data, resulting in privacy leakage. Summary of the invention
[0011] In view of the above problems, a method and device for transmitting model training data, an electronic device, and a medium are proposed to overcome the above problems or at least partially solve the above problems, including:
[0012] A method for transmitting model training data, the method comprising:
[0013] When it is detected that the word vector data in the model training process will be sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained;
[0014] generating a reversible disturbance matrix based on the current time data;
[0015] Processing the word vector data according to the reversible perturbation matrix to obtain transmission data;
[0016] The transmission data is transmitted to the second data processing node.
[0017] Optionally, generating a reversible disturbance matrix based on the current time data includes:
[0018] Determining a first vector dimension of the word vector data;
[0019] Based on the first vector dimension, determining a second vector dimension of an invertible perturbation matrix;
[0020] A reversible perturbation matrix is generated according to the second vector dimension and the current time data.
[0021] Optionally, generating a reversible perturbation matrix according to the second vector dimension and the current time data includes:
[0022] determining a preset formula for generating elements of a reversible matrix;
[0023] Inputting the row and column values corresponding to the second vector dimension and the current time data into the preset formula to determine the matrix elements corresponding to the row and column values in the reversible disturbance matrix;
[0024] The matrix elements are combined according to corresponding row and column values to obtain a reversible disturbance matrix.
[0025] Optionally, determining a second vector dimension of a reversible perturbation matrix based on the first vector dimension includes:
[0026] Taking the number of columns in the first vector dimension as the number of rows of the reversible perturbation matrix;
[0027] Determining the number of columns of the reversible perturbation matrix based on the minimum number of rows and columns of the first vector dimension;
[0028] The second vector dimension of the invertible perturbation matrix is determined according to the number of rows and the number of columns of the invertible perturbation matrix.
[0029] Optionally, the processing the word vector data according to the reversible perturbation matrix to obtain transmission data includes:
[0030] The reversible perturbation matrix is multiplied by the word vector data to obtain transmission data.
[0031] Optionally, it also includes:
[0032] Before performing model training, performing clock synchronization on the first data processing node and the second data processing node;
[0033] After the second data processing node receives the transmission data, the transmission data is decrypted based on the current time data of the second data processing node to obtain the word vector data.
[0034] Optionally, the decrypting the transmission data based on the current time data of the second data processing node to obtain the word vector data includes:
[0035] generating an inverse matrix of the reversible disturbance matrix based on the current time data of the second data processing node;
[0036] The transmission data is decrypted according to the inverse matrix to obtain the word vector data.
[0037] A device for transmitting model training data, the device comprising:
[0038] A current time data acquisition module, used to acquire the current time data of the first data processing node when detecting that the word vector data in the model training process will be sent from the first data processing node to the second data processing node;
[0039] A disturbance matrix generation module, used for generating a reversible disturbance matrix based on the current time data;
[0040] A data processing module, used for processing the word vector data according to the reversible perturbation matrix to obtain transmission data;
[0041] A transmission module is used to transmit the transmission data to the second data processing node.
[0042] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned method for transmitting model training data when executed by the processor.
[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for transmitting model training data as described above is implemented.
[0044] The embodiments of the present invention have the following advantages:
[0045] In an embodiment of the present invention, when it is detected that the word vector data in the model training process will be sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained; a reversible perturbation matrix is generated based on the current time data; the word vector data is processed according to the reversible perturbation matrix to obtain transmission data; and the transmission data is transmitted to the second data processing node. The data transmission process of the model processing is processed according to the time parameters to ensure the security of the data transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0047] Figure 1 is a flowchart of a method for model training data provided by one embodiment of the present invention;
[0048] Figure 2 is a flowchart of another method for model training data provided by one embodiment of the present invention;
[0049] Figure 3a This is a schematic diagram of the data training process of a Transformer model in the prior art;
[0050] Figure 3b is a schematic diagram of a data training process of a Transformer model in an embodiment of the present invention;
[0051] Figure 4 It is a schematic diagram of the structure of a device for model training data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] At present, in order to ensure the secure transmission and privacy protection of data during large model training, researchers have proposed various data privacy protection technologies:
[0054] (1) Methods based on secure multi-party computation
[0055] This technology can perform calculations between multiple parties while protecting the privacy of each party's input data. However, the disadvantages are also obvious: secure multi-party computing involves communication and calculations between multiple parties, so it usually brings large, additional computing and communication overhead. In addition, the implementation of secure multi-party computing requires solving various cryptography and distributed computing problems, which is technically difficult.
[0056] (2) Methods based on homomorphic encryption
[0057] In large model training, homomorphic encryption can be used to perform model parameter updates and gradient calculations on encrypted data to protect data privacy. Although homomorphic encryption technology is attractive in theory, it often faces performance and efficiency challenges in practical applications. Homomorphic encryption adds additional complex technologies that lead to low computational efficiency and increase the time and resource consumption of model training. Homomorphic encryption is limited by the fact that its algorithm currently only supports limited and simple computing functions, making it difficult to apply to complex model training tasks.
[0058] (3) Methods based on differential privacy
[0059] In large model training, differential privacy can be applied to protect the privacy of gradient updates and model parameter calculation processes. Similarly, adding random noise increases complexity and transmission consumption, and the introduction of differential privacy mechanisms will lead to a certain degree of degradation in model performance, requiring a trade-off between privacy protection and model accuracy.
[0060] In an embodiment of the present invention, in order to solve the security problem of word vectors during model communication transmission, a reversible vector with random perturbations can be generated based on a time factor. By adding perturbations to the input word vectors, it is ensured that the original word vector data will not be directly leaked during the transmission process. The original data can only be restored after the receiving end solves the inverse matrix using the same time factor, thereby effectively improving the data security during the transmission process.
[0061] Reference Figure 1 , shows a flowchart of a method for model training data provided by an embodiment of the present invention, which may specifically include the following steps:
[0062] Step S101, when it is detected that word vector data in the model training process will be sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained;
[0063] The model may be a data processing model that uses word vectors as input data, or a model that transmits and processes word vectors, for example, a Transformer model.
[0064] Word embeddings is a key technology in natural language processing (NLP), which maps words or phrases in the vocabulary to continuous vector representations in a high-dimensional space. These vectors capture the semantic and grammatical relationships between words, enabling machines to better understand and process text data.
[0065] Word vector data has the following characteristics:
[0066] (1) Distributed representation: Word embedding is a form of distributed representation, which means that the meaning of a word is jointly represented by multiple dimensions in its vector rather than being determined by a single dimension.
[0067] (2) Semantic similarity: In the word vector space, semantically similar words are mapped to similar positions, which enables the vector-based similarity calculation to reflect the semantic relationship between words.
[0068] (3) Context-dependency: Word embeddings are usually learned through contextual information, that is, the vector representation of a word depends on the context in which it appears in the text.
[0069] (4) Computability: Word vectors support various mathematical operations. For example, vector addition and subtraction can be used to capture complex relationships between words, such as “king - man + woman = queen”.
[0070] In practical applications, methods for generating word vector data include counting-based methods and prediction-based methods. Among them, counting methods include latent semantic analysis (LSA), which mainly constructs word vectors by counting the co-occurrence frequency of words in a large corpus. Prediction-based methods can include any of Word2Vec, GloVe (Global Vectors for Word Representation) and FastText, which can learn word vectors by predicting context words or target words.
[0071] In the process of model processing word vector data, multiple data processing nodes can be set, and data can be transmitted from one data processing node to another in a certain order. When data is transmitted between data nodes, word vector data is directly exposed and in an unsafe state. To ensure the safe transmission of word vector data, the current time data can be obtained, and then a reversible perturbation matrix can be generated based on the time data to realize implicit transmission of word vectors and improve transmission security.
[0072] The current time data may include various parts of the current time, such as year, month, day, hour, minute and second.
[0073] Step S102, generating a reversible disturbance matrix based on current time data;
[0074] After obtaining the current time data, a reversible perturbation matrix can be generated based on the current time data. The time data is random, so different random reversible perturbation matrices can be generated at different times.
[0075] In an embodiment of the present invention, the reversible perturbation matrix generated in the first data processing node can be used for data processing at the first data processing node, and in the next data processing node, the word vector data can be restored based on the inverse matrix of the reversible perturbation matrix generated at the same time point to achieve decryption.
[0076] In practical applications, a matrix is reversible, which is called a non-singular matrix or a non-degenerate matrix, which means that there is an inverse matrix of the matrix. For a square matrix (i.e. a matrix with equal number of rows and columns), to satisfy the reversibility condition, the following restrictions need to be met:
[0077] (1) Squareness: Only square matrices can have inverse matrices. This means that the number of rows in the matrix must be equal to the number of columns.
[0078] (2) The determinant is not zero: The determinant of a matrix (denoted as det(A) or |A|) must not be equal to zero. If the determinant is zero, the matrix is singular and there is no inverse matrix.
[0079] (3) Full rank: The rank of a matrix (i.e., the maximum number of linearly independent rows or columns in a matrix) must be equal to the order of the matrix (i.e., the number of rows or columns). In other words, the matrix must be full rank.
[0080] (4) Linearly independent rows / columns: All rows (or columns) of a matrix must be linearly independent. If there exists a set of non-zero coefficients such that the linear combination of these rows (or columns) equals zero rows (or zero columns), then the matrix is not invertible.
[0081] (5) Non-singularity: A matrix cannot be singular. Singular matrices are matrices whose determinant is zero. Singular matrices have no inverse matrix.
[0082] Specifically, you can use the generate time based invertible matrix() function to generate a deterministic matrix. A deterministic matrix means that each element in the matrix is calculated by a fixed rule or formula, rather than being randomly generated. This means that as long as the input parameters are the same, the matrix generated by the function is the same each time.
[0083] Specifically, the process of generating an invertible perturbation matrix is as follows:
[0084] Import numpy as np
[0085] Import time
[0086] Def generate_time_based invertible matrix(n,m):
[0087] “”
[0088] Generate an n×m matrix and ensure its invertibility in the context.
[0089] Parameters:
[0090] n: The number of rows of the matrix
[0091] m: The number of columns of the matrix
[0092] Returns:
[0093] An n×m matrix
[0094] “”
[0095] # Get the current time
[0096] current_time = time.localtime()
[0097] time_parameters =
[0098] current_time.tm_year, current_time.tm_mon, current_time.tm_mday, current_time.tm_hour, current_time.tm_min, current_time.tm_sec
[0099] # Ensure the length of time_parameters is sufficient to fill the n×m matrix
[0100] while len(time_parameters) < n*m: # Repeat time parameters until long enough
[0101] time_parameters.extend(time_parameters)
[0102] # Take the first n*m elements
[0103] time_parameters = time parametersL:nm]
[0104] # Convert to a numpy matrix and reshape to n x m
[0105] matrix = np.array(time parameters).reshape(n, m)
[0106] # Ensure matrix invertibility
[0107] if n == m:
[0108] # If it's a square matrix, check if its determinant is zero
[0109] while np.linalg.det(matrix) == 0: matrix += np.random.rand(n, m) # Add random noise to break singularity
[0110] else:
[0111] # If it's a non-square matrix, check if its rank is min(n, m) while np.linalg.matrix_rank(matrix) < min(n, m): matrix += np.random.rand(n, m) # Add random noise to increase rank
[0112] return matrix
[0113] # Example: Generate a 4x4 invertible matrix
[0114] n = 4
[0115] m = 4
[0116] invertible_matrix = generate_time_based_invertible_matrix(n, m)
[0117] print("Generated matrix:")
[0118] print(invertible_matrix)
[0119] Step S103, process the word vector data according to the invertible perturbation matrix to obtain the transmission data;
[0120] After obtaining the reversible perturbation matrix, the word vector data can be processed according to the reversible perturbation matrix to obtain transmission data. Specifically, matrix operations can be performed based on the reversible perturbation matrix and the word vector data to achieve processing of the word vector data.
[0121] In one example, the word vector data may be multiplied by a reversible perturbation matrix to obtain transmission data, that is, transmission data = word vector data * reversible perturbation matrix.
[0122] Step S104: transmitting the transmission data to the second data processing node.
[0123] After the transmission data is generated, the transmission data can be transmitted to the second data processing node. That is, in the data transmission process between the first data processing node and the second data processing node, what is actually transmitted is the transmission data, not the word vector data. Even if the transmission data can be obtained, it is difficult to parse the word vector data from the transmission data. Therefore, the data transmission scheme does not directly expose the word vector data during the transmission process, and can achieve secure transmission of the word vector data.
[0124] In an embodiment of the present invention, when it is detected that the word vector data in the model training process is sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained; a reversible perturbation matrix is generated based on the current time data; the word vector data is processed according to the reversible perturbation matrix to obtain transmission data; and the transmission data is transmitted to the second data processing node. This realizes processing according to time parameters during the model processing data transmission process to ensure the security of data transmission.
[0125] Reference Figure 2 , shows a flowchart of another method for model training data provided by an embodiment of the present invention, which may specifically include the following steps:
[0126] Step S201, when it is detected that word vector data in the model training process will be sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained;
[0127] Step S202, determining a first vector dimension of the word vector data;
[0128] Among them, the first vector dimension information of the word vector data is the number of rows and columns of the word vector data.
[0129] Step S203, determining a second vector dimension of the reversible perturbation matrix based on the first vector dimension;
[0130] Since the word vector data follows the rules of matrix operations, in order to generate transmission data based on the invertible perturbation matrix and the word vector data, it is necessary to determine the second vector dimension of the invertible perturbation matrix to be generated based on the first vector dimension, so that normal matrix operations can be performed between the invertible perturbation matrix and the word vector data.
[0131] In an embodiment of the present invention, determining the second vector dimension of the invertible perturbation matrix based on the first vector dimension specifically includes: taking the number of columns in the first vector dimension as the number of rows of the invertible perturbation matrix; determining the number of rows of the invertible perturbation matrix based on the minimum value of the number of rows and columns in the first vector dimension; and determining the second vector dimension of the invertible perturbation matrix according to the number of rows and columns of the invertible perturbation matrix.
[0132] For example, A is a matrix composed of input word vectors with a dimension of n*m, B is an invertible perturbation vector matrix with a time factor added, with a dimension of m*k, and C is a matrix generated after adding B to A, with a dimension of n*k. Among them, k << min(n,m). Although the addition of the invertible perturbation matrix increases the redundancy of the technology, it greatly reduces the scale data of the matrix data, changing the vector dimension from dimension (n*m) to (n*k), achieving dimensionality reduction processing of the transmission data, reducing the data transmission volume, and improving the data processing efficiency.
[0133] Step S204, generate an invertible perturbation matrix according to the second vector dimension and the current time data.
[0134] The second vector dimension fixes the number of rows and columns of the invertible perturbation matrix, and the current time data can generate each matrix element within the invertible perturbation matrix.
[0135] In an embodiment of the present invention, generating an invertible perturbation matrix according to the second vector dimension and the current time data includes: determining a preset formula for generating invertible matrix elements; inputting the row and column values corresponding to the second vector dimension and the current time data into the preset formula to determine the matrix elements corresponding to the row and column values in the invertible perturbation matrix; and combining the matrix elements according to the corresponding row and column values to obtain the invertible perturbation matrix.
[0136] Among them, the preset formula can be set according to the actual scenario. The preset formula has two variables, namely the second vector dimension and the current time data. After determining the second vector dimension and the current time data, substituting them into the preset formula can calculate each matrix element.
[0137] In an example, after generating the invertible perturbation matrix, it can be verified whether the invertible perturbation matrix is an invertible matrix. If the invertible perturbation matrix is an invertible matrix, the verification is successful, and step S205 is executed. If the invertible perturbation matrix is a non-invertible matrix, the verification fails, and step 204 is returned to regenerate the invertible perturbation matrix.
[0138] Step S205, processing the word vector data according to the reversible perturbation matrix to obtain transmission data;
[0139] In one embodiment of the present invention, word vector data are processed according to a reversible perturbation matrix to obtain transmission data, including: multiplying the reversible perturbation matrix with the word vector data to obtain transmission data.
[0140] Transmission data = word vector data * reversible perturbation matrix.
[0141] For example: A is the word vector data, B is the reversible perturbation matrix, and C is the transmission data.
[0142]
[0143] Step S206: transmitting the transmission data to the second data processing node.
[0144] In one embodiment of the present invention, before model training, the first data processing node and the second data processing node are clock synchronized so that the second data node and the first data node can generate an interference matrix under the same time parameters. After the second data processing node receives the transmission data, the transmission data can be decrypted based on the inverse matrix of the interference matrix generated by the second data processing node to obtain word vector data.
[0145] In practical applications, by synchronizing the clocks of the first data processing node and the second data processing node, it is possible to avoid a large time difference between the first data processing node and the second data processing node, which makes it difficult to parse the transmission data.
[0146] Specifically, the first data node and the second data node may store the same current time data, and the current time data may be the time when the first data node is triggered to generate a reversible perturbation matrix.
[0147] In one embodiment of the present invention, the transmission data is decrypted based on the current time data of the second data processing node to obtain word vector data, including: generating an inverse matrix of a reversible perturbation matrix based on the current time data of the second data processing node; and decrypting the transmission data according to the inverse matrix to obtain word vector data.
[0148] In practical applications, the clocks of the first data processing node and the second data processing node are synchronized. When a reversible perturbation matrix is generated based on the current time in the first data processing node, the second data processing node can synchronously generate a reversible perturbation matrix based on the current time, and then obtain an inverse matrix based on the generated reversible perturbation matrix, and store the inverse matrix for decrypting the received transmission data.
[0149] For example, assume that the output matrix in the model is the word vector matrix A with dimensions n*m, and the invertible perturbation matrix generated based on the current time is the dimension-reducing invertible perturbation matrix B with dimensions m*k, where k << min(n,m). Denote (A×B) = C, and transmit the matrix C to the second data processing node.
[0150] In the second data processing node, given (A×B) and B, and denoting (A×B) = C, solve for A.
[0151] First, calculate the inverse matrix B of B -1 , if B is invertible, there exists a unique inverse matrix B -1 .
[0153] Furthermore, using (A×B) = C, multiply both sides on the right by B -1 , that is, A×B×B -1 = C×B -1 . Since B×B -1 = I (identity matrix), the above equation can be simplified to:
[0154] A = C×B -1 .
[0155] Based on the above derivation process, the prerequisite for solving the matrix A is that the B matrix is an invertible matrix.
[0156] In practical applications, during the training process of large models, word vector data can be transmitted and protected in the following scenarios:
[0157] Scenario 1: Transmission between devices.
[0158] In distributed training, input data needs to be transmitted between multiple devices (such as servers). By generating invertible perturbation matrices with the same time parameters on each device, the consistency and security of the data during transmission can be ensured.
[0159] Scenario 2: Transmission between modules.
[0160] In a complex neural network model, word vectors need to be passed between different layers or modules. By passing the perturbed word vectors between each module, data leakage can be effectively prevented.
[0161] Scenario 3: Model sharing and collaborative training,
[0162] When multiple parties cooperate in model training, each party can share the perturbed word vector data without exposing the original data, protecting their respective privacy.
[0163] By adding a random perturbation reversible matrix based on a time factor to the word vector, the present invention provides an effective technical means for protecting the privacy and security of data transmission during large model training. This method ensures that the original word vector data will not be directly leaked during the transmission process, and the original data can only be restored after solving the inverse matrix at the receiving end. This technology has broad application prospects in scenarios such as distributed training and multi-party collaborative training, and provides important guarantees for data privacy protection during large model training.
[0164] In an embodiment of the present invention, when it is detected that word vector data in a model training process will be sent from a first data processing node to a second data processing node, the current time data of the first data processing node is obtained; the first vector dimension of the word vector data is determined; based on the first vector dimension, the second vector dimension of the reversible perturbation matrix is determined; according to the second vector dimension and the current time data, a reversible perturbation matrix is generated; the word vector data is processed according to the reversible perturbation matrix to obtain transmission data; the transmission data is transmitted to the second data processing node, thereby realizing the generation of a reversible vector with random perturbations based on a time factor, and then hiding the word vector to be transmitted, thereby ensuring the security of word vector data transmission.
[0165] Reference Figure 3a , shows a schematic diagram of data transmission during a Transformer model training process in the prior art. The Transformer model may include 32 data processing layer blocks for processing input data.
[0166] The word vector (Input Eedding) is input into the Transformer model, and then the word vector is transmitted to each layer block in turn for data processing, and the data output from the 32 layers is normalized. RMSNorm (Root Mean Square Normalization) is a normalization technique used in deep learning, especially in the field of natural language processing (NLP). It is an improvement on layer normalization, aiming to improve the training efficiency and performance of the model. The advantage of RMSNorm is that it is simple and efficient to calculate, because it does not need to calculate the mean, thus reducing the amount of calculation.
[0167] After normalization, the normalized data can be processed linearly. In machine learning and deep learning, the linear processing process involves: Linear layer (fully connected layer): In a neural network, the linear layer performs a weighted sum of the input data and then usually adds a bias term. This is the most basic linear operation. Linear regression: In statistics and machine learning, the linear regression model predicts continuous output values by fitting a linear equation. Linear transformation: In the data preprocessing stage, linear transformations such as scaling and translation are often used to standardize data to better adapt to model training.
[0168] After linear processing of the data, the SoftMax function is executed. The SoftMax function is an activation function commonly used in machine learning and deep learning, mainly used for multi-category classification problems. Its function is to compress a K-dimensional vector containing arbitrary real numbers into a K-dimensional probability distribution, where the value of each element is between (0,1) and the sum of all elements is 1.
[0169] After the activation function is processed, the processing result of the Transformer model is output.
[0170] Among them, during the transmission process from layer1 to layer132, the word vector data is exposed and easy to be tampered with, so the transmission process is not safe.
[0171] Reference Figure 3b , showing a schematic diagram of data transmission during the training of a Transformer model in an embodiment of the present invention. In view of the parallel training characteristics of different large models, the Transformer Block Layer (multi-layer data processing, generally 32 layers) can be flexibly combined and paired, and a time factor perturbation matrix module M can be added to each layer, or M can be added to the middle layer or the last layer of the Transformer Block Layer. Before training the model, it is necessary to synchronize the clocks of each device (the physical server that carries the model). Ensure that the time error is within a controllable range. Because network transmission requires a short interval, the minimum time factor can be selected in minutes or even seconds.
[0172] Specifically, after the word vector data A is output to the Transformer model, the time factor perturbation matrix module M can be used to generate a reversible matrix B based on the current time. Thus, the word vector data and the reversible matrix B are combined to obtain the matrix C, that is, the matrix C = word vector data A*reversible matrix B.
[0173] After the matrix C is safely transmitted to layer 1, in layer 1, the time factor perturbation matrix module M can be used to obtain the inverse matrix B of the reversible matrix B -1 , and then we can use B-1 The matrix C is restored to word vector data A, and then the word vector matrix is processed in layer1.
[0174] When data is transmitted from layer1 to layer2, the time factor perturbation matrix module M can be used again to generate a new reversible matrix B based on the current time, and then data can be transmitted based on the data processed by the reversible matrix. According to the above data processing and data transmission process, the word vector data can be kept hidden during the transmission between modules, and kept decrypted during the data transmission within the module for data transmission.
[0175] In large model training, word vectors are frequently transmitted between model devices and internal modules as input data. In order to protect data privacy during these transmissions, this case proposes a method to generate a reversible vector with random perturbations based on a time factor. By adding perturbations to the input word vectors, data security during transmission can be effectively improved.
[0176] In one embodiment of the present invention, before data transmission, it is also possible to determine whether the next-level data processing node is in the same server as the current data processing node. When the two data processing nodes are nodes on the same server, the word vector data is directly sent to the next data processing node. When the two data processing nodes are nodes on different servers, disturbances can be added to the input word vectors, thereby effectively improving the data security during the transmission process and solving the security issues of word vectors during model communication transmission.
[0177] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0178] Reference Figure 4 , shows a schematic diagram of the structure of a device for generating game voices provided by an embodiment of the present invention, which may specifically include the following modules:
[0179] The current time data acquisition module 401 is used to acquire the current time data of the first data processing node when detecting that the word vector data in the model training process will be sent from the first data processing node to the second data processing node;
[0180] A disturbance matrix generation module 402, used to generate a reversible disturbance matrix based on the current time data;
[0181] A data processing module 403 is used to process the word vector data according to the reversible perturbation matrix to obtain transmission data;
[0182] The transmission module 404 is configured to transmit the transmission data to the second data processing node.
[0183] In one embodiment of the present invention, the reversible perturbation matrix generation module 402 may include:
[0184] A first vector dimension determination submodule, used to determine the first vector dimension of the word vector data;
[0185] A second vector dimension determination submodule, used to determine a second vector dimension of a reversible perturbation matrix based on the first vector dimension;
[0186] The reversible perturbation matrix generating submodule is used to generate a reversible perturbation matrix according to the second vector dimension and the current time data.
[0187] In one embodiment of the present invention, the reversible perturbation matrix generation submodule may include:
[0188] A preset formula determination unit, used to determine a preset formula for generating a reversible matrix element;
[0189] A matrix element determination unit, used for inputting the row and column values corresponding to the second vector dimension and the current time data into the preset formula to determine the matrix elements corresponding to the row and column values in the reversible disturbance matrix;
[0190] The reversible disturbance matrix determination unit is used to combine the matrix elements according to corresponding row and column values to obtain a reversible disturbance matrix.
[0191] In one embodiment of the present invention, the second vector dimension determination submodule may include:
[0192] A row number determination unit, configured to use the number of columns in the first vector dimension as the number of rows of a reversible perturbation matrix;
[0193] A column number determination unit, used to determine the number of columns of the reversible perturbation matrix based on the minimum number of rows and columns of the first vector dimension;
[0194] The second vector dimension determining unit is used to determine the second vector dimension of the reversible perturbation matrix according to the number of rows and columns of the reversible perturbation matrix.
[0195] In one embodiment of the present invention, the data processing module 403 may include:
[0196] The data processing submodule is used to multiply the reversible perturbation matrix with the word vector data to obtain transmission data.
[0197] In one embodiment of the present invention, the device further comprises:
[0198] A clock synchronization module, used for performing clock synchronization on the first data processing node and the second data processing node before model training;
[0199] The word vector recovery module is used to decrypt the transmission data based on the current time data of the second data processing node after the second data processing node receives the transmission data to obtain the word vector data.
[0200] In one embodiment of the present invention, the word vector recovery module may include:
[0201] an inverse matrix determination submodule, configured to generate an inverse matrix of the reversible disturbance matrix based on the current time data of the second data processing node;
[0202] The decryption processing submodule is used to decrypt the transmission data according to the inverse matrix to obtain the word vector data.
[0203] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the method for model training data as described above is implemented.
[0204] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for training model data as described above is implemented.
[0205] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0206] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0207] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0208] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0211] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0212] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0213] The above provides a detailed introduction to the provided method and device for model training data, electronic device, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for transmitting model training data, characterized in that: The method comprises: When it is detected that the word vector data in the model training process will be sent from the first data processing node to the second data processing node, the current time data of the first data processing node is obtained; generating a reversible disturbance matrix based on the current time data; Processing the word vector data according to the reversible perturbation matrix to obtain transmission data; The transmission data is transmitted to the second data processing node.
2. The method according to claim 1, characterized in that The generating a reversible disturbance matrix based on the current time data comprises: Determining a first vector dimension of the word vector data; Based on the first vector dimension, determining a second vector dimension of an invertible perturbation matrix; A reversible perturbation matrix is generated according to the second vector dimension and the current time data.
3. The method according to claim 2, characterized in that The step of generating a reversible disturbance matrix according to the second vector dimension and the current time data comprises: determining a preset formula for generating elements of a reversible matrix; Inputting the row and column values corresponding to the second vector dimension and the current time data into the preset formula to determine the matrix elements corresponding to the row and column values in the reversible disturbance matrix; The matrix elements are combined according to corresponding row and column values to obtain a reversible disturbance matrix.
4. The method according to claim 2, characterized in that: The determining, based on the first vector dimension, a second vector dimension of the reversible perturbation matrix comprises: Using the number of columns in the first vector dimension as the number of rows of the reversible perturbation matrix; Determining the number of columns of the reversible perturbation matrix based on the minimum number of rows and columns of the first vector dimension; The second vector dimension of the invertible perturbation matrix is determined according to the number of rows and columns of the invertible perturbation matrix.
5. The method according to claim 1, characterized in that: The step of processing the word vector data according to the reversible perturbation matrix to obtain transmission data includes: The reversible perturbation matrix is multiplied by the word vector data to obtain transmission data.
6. The method according to claim 2, characterized in that Also includes: Before performing model training, performing clock synchronization on the first data processing node and the second data processing node; After the second data processing node receives the transmission data, the transmission data is processed based on the current time data of the second data processing node to obtain the word vector data.
7. The method according to claim 6, characterized in that The decrypting the transmission data based on the current time data of the second data processing node to obtain the word vector data includes: generating an inverse matrix of the reversible disturbance matrix based on the current time data of the second data processing node; The transmission data is processed according to the inverse matrix to obtain the word vector data.
8. A transmission device for model training data, characterized in that: The device comprises: A current time data acquisition module, used to acquire the current time data of the first data processing node when detecting that the word vector data in the model training process will be sent from the first data processing node to the second data processing node; A disturbance matrix generation module, used for generating a reversible disturbance matrix based on the current time data; A data processing module, used for processing the word vector data according to the reversible perturbation matrix to obtain transmission data; A transmission module is used to transmit the transmission data to the second data processing node.
9. An electronic device, characterized in that: It comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for transmitting model training data as claimed in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for transmitting model training data as described in any one of claims 1 to 7 is implemented.
