Data processing method and device, storage medium and computer program product
By locally training the Transformer layer parameters of the student model in the data processing device, the problem of data leakage in data sensitive scenarios is solved, and the privacy protection of teacher models and training data is achieved, and data security is improved.
Patent Information
- Application Number
- CN202411216116.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-31
AI Technical Summary
In data-sensitive scenarios, vertical field data sets or teacher models are more private, which can easily lead to data leakage when transmitting data, reducing data security.
By receiving the first two layers of Transformer layer data of the teacher model, the first layer Transformer layer parameters of the student model are trained locally in the data processing device based on these data and pre-stored training data, and the K-1 layer Transformer layer parameters are trained in joint training with the first device. , construct student models, avoid transmitting original training data, and protect the privacy of teacher models and training data.
It effectively reduces data leakage, improves data security, and ensures the privacy of teacher models and training data.
Smart Images
Figure CN119227129A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud computing, and in particular to a data processing method and device, a storage medium, and a computer program product. Background Art
[0002] With the continuous development of big data and artificial intelligence technologies, pre-trained large language models have achieved remarkable achievements in the field of natural language processing. When these pre-trained base large models are adapted to downstream tasks, it is often necessary to compress the original models to reduce the requirements for hardware resources during deployment and improve the inference speed. First, use the vertical domain data set related to the downstream task to optimize the base large model to obtain the vertical domain large model, and then regard it as a teacher model to train a small-scale student model. After the training is completed, the student model is the compressed model obtained. During the knowledge distillation operation, the business side of the downstream task is required to provide its vertical domain data set or downstream task data set as the common input data for the teacher model and the student model in the knowledge distillation operation to train the student model.
[0003] For data-sensitive scenarios, when the vertical field datasets or downstream task datasets are highly private, and / or the teacher model data is highly private, the business side of the downstream task inputs the vertical field datasets or downstream task datasets into the teacher model, or transmits the teacher model data to the business side of the downstream task, which will lead to data leakage and reduce data security. Summary of the invention
[0004] The present application provides a data processing method and device, a storage medium, and a computer program product, which can reduce data leakage and improve data security.
[0005] The technical solution of this application is implemented as follows:
[0006] In a first aspect, the present application proposes a data processing method, which is applied to a data processing device, and the method comprises:
[0007] Receive data of the first two Transformer layers in the teacher model sent by the first device;
[0008] Based on the data of the first two Transformer layers and the pre-stored training data, training obtains the parameters of the first Transformer layer of the student model;
[0009] Based on the first Transformer layer obtained through training and the pre-stored training data, the data processing device and the first device jointly train parameters of the last K-1 Transformer layers of the student model, where K is a positive integer greater than or equal to 2;
[0010] The student model is constructed using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layers to process natural language tasks.
[0011] In a second aspect, the present application provides a data processing device, the data processing device comprising:
[0012] A receiving unit, configured to receive data of the first two Transformer layers in the teacher model sent by the first device;
[0013] A training unit, configured to obtain parameters of the first Transformer layer of the student model through training based on the data of the first two Transformer layers and the pre-stored training data on the data processing device; based on the trained first Transformer layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are obtained through joint training by the data processing device and the first device, where K is a positive integer greater than or equal to 2;
[0014] A construction unit is used to construct the student model using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layer to process the natural language task.
[0015] In the third aspect, the present application proposes a data processing device, which includes: a processor, a memory and a communication bus; the communication bus is used to realize the connection and communication between the processor and the memory; the processor implements the above-mentioned data processing method when executing the running program stored in the memory.
[0016] In a fourth aspect, the present application proposes a storage medium on which a computer program is stored, and the computer program implements the above-mentioned data processing method when executed by a processor.
[0017] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which implements the above-mentioned data processing method when executed by a processor.
[0018] The present application provides a data processing method and device, storage medium, and computer program product. The method includes: a data processing device receives data of the first two layers of Transformer layers in a teacher model sent by a first device; N is a positive integer; based on the data of the first two layers of Transformer layers and pre-stored training data, the parameters of the first layer of Transformer layer of a student model are trained; based on the trained first layer of Transformer layer and the pre-stored training data, the parameters of the last K-1 layers of Transformer layers of the student model are jointly trained by the data processing device and the first device, where K is a positive integer greater than or equal to 2; the parameters of the first layer of Transformer layer and the parameters of the last K-1 layers of Transformer layers are used to construct a student model to process natural language tasks. Using the above implementation scheme, the teacher model and the student model are distributedly deployed on different devices. The teacher model only transmits the data of the first two Transformer layers to the data processing device for model training, while the data of the last N-2 Transformer layers of the subject are still on the first device, thereby protecting the privacy of the teacher model data. For the data processing device, the pre-stored training data is used for training locally, and the original pre-stored training data will not be sent to the first device for training, thereby protecting the privacy of the training data, thereby reducing data leakage and improving data security. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0020] Figure 2 A schematic diagram of the structure of an exemplary big data model provided in an embodiment of the present application;
[0021] Figure 3 An exemplary method flow of a teacher model training a student model provided in an embodiment of the present application Figure 1 ;
[0022] Figure 4 An exemplary method flow of a teacher model training a student model provided in an embodiment of the present application Figure 2 ;
[0023] Figure 5 An exemplary method flow of a teacher model training a student model provided in an embodiment of the present application Figure 3 ;
[0024] Figure 6 A schematic diagram of a data processing device provided in an embodiment of the present application Figure 1 ;
[0025] Figure 7A schematic diagram of a data processing device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION
[0026] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0028] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0029] The present application embodiment provides a data processing method, such as Figure 1 As shown, the method may include:
[0030] S101. Receive data of the first two Transformer layers in the teacher model sent by the first device.
[0031] In the embodiment of the present application, the model application type of the teacher model can be a language model. The specific model can be selected according to the actual situation, and the embodiment of the present application does not make specific limitations.
[0032] In one embodiment, data of the first two Transformer layers of the non-main body portion may be transmitted to a data processing device.
[0033] Further, the data of the preprocessing layer and the data of the output layer in the teacher model sent by the first device are received, and the student model directly reuses the data of the preprocessing layer and the data of the output layer in the teacher model.
[0034] In the embodiment of the present application, for the structural diagram of the large language model, see Figure 2, the preprocessing layer can include a tokenizer and an embedding layer. The large language model consists of a tokenizer, an embedding layer, a multi-layer Transformer layer, and a text output layer (output layer).
[0035] In the embodiment of the present application, each Transformer layer is composed of a multi-head attention mechanism part and a position feedforward network part.
[0036] It should be noted that at the beginning of training, the first device will send the parameters of the teacher model's word segmenter, embedding layer, text output layer and the first two Transformer layers to the data processing device, but the other layers of its multiple Transformer layers, that is, the main part, are retained locally and not out of the domain. Therefore, the data privacy of the teacher model is protected as a whole.
[0037] It should be noted that for the Transformer layer, the internal structure of the Transformer layer of the student model is the same as that of the Transformer layer of the teacher model, the number of layers of the student model is smaller than that of the teacher model (assuming that the teacher model has N layers and the student model has K layers, then N is much larger than K), and the Transformer layer of the student model has independent data itself.
[0038] S102: Based on the data of the first two Transformer layers and the pre-stored training data, train and obtain the parameters of the first Transformer layer of the student model.
[0039] In the embodiment of the present application, the process of training the first Transformer layer is completed on the data processing device side.
[0040] In an embodiment of the present application, the parameters of the first Transformer layer of the student model are trained based on the data of the first two Transformer layers, the data of the preprocessing layer, the data of the output layer and the pre-stored training data.
[0041] In the embodiment of the present application, the data processing device first performs word segmentation and embedding operations on the initial pre-stored training data to obtain the above-mentioned pre-stored training data, wherein the word segmentation and embedding operations are performed by the data of the pre-processing layer and the output layer in the teacher model sent by the first device. Then, based on the data of the first two layers of Transformer layers and the pre-stored training data, the parameters of the first layer of Transformer layer are trained.
[0042] Specifically, the pre-stored training data is input into the first two Transformer layers to obtain the first attention value and the first hidden state value corresponding to the second Transformer layer of the teacher model; the pre-stored training data is input into the initial first Transformer layer to obtain the second attention value and the second hidden state value; the parameters of the first Transformer layer are obtained by training according to the first attention value, the first hidden state value, the second attention value and the second hidden state value.
[0043] In an embodiment of the present application, the process of training the parameters of the first Transformer layer according to the first attention value, the first hidden state value, the second attention value and the second hidden state value specifically includes: determining the first loss value according to the first attention value, the first hidden state value, the second attention value and the second hidden state value; reversely updating the parameters of the initial first Transformer layer according to the first loss value until the training stop condition is met, thereby obtaining the parameters of the first Transformer layer.
[0044] In the embodiment of the present application, the process of determining the first loss value according to the first attention value, the first hidden state value, the second attention value and the second hidden state value can be seen in formula (1)-formula (3).
[0045]
[0046] L hidn =MSE(H T ,H S ) (2)
[0047] L total =L attn +L hidn (3)
[0048] In the above formula (1), h represents the number of attention heads in the Transformer layer. represents the attention matrix value (first attention value) in the i-th attention head in the teacher model, represents the attention matrix value (second attention value) in the i-th attention head in the student model, and MSE represents the minimum mean square error operation. For formula (2), H T represents the hidden state value (first hidden state value) in the teacher model, H S represents the hidden state value (second hidden state value) in the teacher model. According to the loss value L calculated by (3) total , the parameters in the first Transformer layer of the student model can be updated in reverse.
[0049] It should be noted that when the preset stop training condition is reached, the training can be terminated to obtain the parameters of the first Transformer layer.
[0050] S103. Based on the trained first Transformer layer and the pre-stored training data, the data processing device and the first device jointly train to obtain parameters of the last K-1 Transformer layers of the student model, where K is a positive integer greater than or equal to 2.
[0051] In the embodiment of the present application, the training of the last K-1 layers of Transformer layers requires the first device and the data processing device to jointly perform distributed training.
[0052] It should be noted that the student model includes K Transformer layers.
[0053] In an embodiment of the present application, based on the trained first Transformer layer, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, the data processing device and the first device jointly train to obtain the parameters of the last K-1 Transformer layers of the student model, where K is a positive integer greater than or equal to 2.
[0054] In the embodiment of the present application, the data processing device also first performs word segmentation and embedding operations on the initial pre-stored training data to obtain the above-mentioned pre-stored training data, wherein the word segmentation and embedding operations are performed by the data of the pre-processing layer and the output layer in the teacher model sent by the first device. Then, based on the first layer of Transformer layer obtained by training and the pre-stored training data, the parameters of the subsequent K-1 layers of Transformer layers are obtained by training.
[0055] It should be noted that the pre-stored training data can be randomly obtained by the data processing device from the local training data set, and the specific one can be selected according to the actual situation, and the embodiment of the present application does not make any specific limitation.
[0056] Specifically, the confidentiality level information of the pre-stored training data is obtained; the first training data with low confidentiality level information in the pre-stored training data is input into the first two Transformer layers of the teacher model to obtain the first hidden state information; the second training data with high confidentiality level information in the pre-stored training data is input into the first Transformer layer of the student model to obtain the second hidden state information; the first hidden state information and the second hidden state information are used to jointly train the data processing device and the first device to obtain the parameters of the last K-1 Transformer layers of the student model.
[0057] In the embodiment of the present application, the process of jointly training the parameters of the last K-1 Transformer layers of the student model by the data processing device and the first device using the first hidden state information and the second hidden state information specifically includes: inputting the first hidden state information and the second hidden state information into the initial last K-1 Transformer layers of the student model to obtain K-1 triple attention values and K-1 third hidden state values; sending the first hidden state information and the second hidden state information to the first device, and receiving K-1 fourth attention values and K-1 fourth hidden state values corresponding to the corresponding layers of the last K-1 Transformer layers of the student model sent by the first device; wherein, the K-1 fourth attention values and K-1 fourth hidden state values corresponding to the corresponding layers of the last K-1 Transformer layers of the student model are obtained by the first device inputting the first hidden state information and the second hidden state information into the 3-Nth Transformer layer of the teacher model; N is a positive integer greater than 2, and N is much larger than K; determining a second loss value using the K-1 third attention values, the K-1 third hidden state values, the K-1 fourth attention values, and the K-1 fourth hidden state values; and reversely updating the parameters of the initial last K-1 Transformer layers according to the second loss value until the stop training condition is satisfied, then obtaining the parameters of the last K-1 Transformer layers.
[0058] In the embodiment of the present application, after the data processing device sends the first hidden state information and the second hidden state information to the first device, the first device inputs the received first hidden state information and the second hidden state information into the last N-2 Transformer layers of the teacher model, and records K-1 fourth attention values and K-1 fourth hidden state values corresponding to the corresponding layers of the last K-1 Transformer layers.
[0059] It should be noted that the corresponding layers of the last K-1 Transformer layers refer to, for the 2nd to kth layers of the student model, in the 3rd to nth layers of the teacher model, selecting k-1 layers f(2)...f(k) by method f such that f(i)≥3 and f(i)≤n and f(i)<f(i + 1). Then the corresponding layer of the ith layer of the student model in the teacher model is the f(i)th layer.
[0060] It should be noted that, for the student model, although data will be sent to the first device when the k-1 layers of Transformer layers are trained after the student model, what is sent is the intermediate data (first hidden state information and second hidden state information) during the forward propagation of the model, rather than the original plaintext data (pre-stored training data), so the privacy of the training data is protected. In particular, for high-level training data, during forward propagation, it will first pass through the first layer of the Transformer layer of the student model and then be sent to the first device, and the first device has no way of knowing the parameters of the first layer of the Transformer layer of the student model, so the first device cannot infer the original plaintext data based on the received intermediate data, thereby further improving the data security of the pre-stored training data on the data processing device side.
[0061] In an embodiment of the present application, the data processing device merges a third attention value, a third hidden state value, a fourth attention value and a fourth hidden state value of each Transformer layer; for each Transformer layer, the loss value of the layer is calculated using formula (1)-formula (2), and finally the loss value of each Transformer layer is summarized to obtain a second loss value.
[0062] It should be noted that when the preset stop training condition is reached, the training can be terminated and the parameters of the last K-1 layers of Transformer layers can be obtained.
[0063] S104. Construct a student model using the parameters of the first Transformer layer and the parameters of the next K-1 Transformer layers to process natural language tasks.
[0064] Furthermore, the student model is constructed using the data of the preprocessing layer, the parameters of the first Transformer layer, the parameters of the next K-1 Transformer layers, and the data of the output layer.
[0065] It can be understood that the teacher model and the student model are distributed and deployed on different devices. The teacher model only transmits the data of the first two Transformer layers to the data processing device for model training, while the data of the last N-2 Transformer layers of the subject are still on the first device, which protects the privacy of the teacher model data. For the data processing device, the pre-stored training data is used for training locally, and the original pre-stored training data will not be sent to the first device for training, which also protects the privacy of the training data, thereby avoiding data leakage and improving data security.
[0066] Based on the above embodiment, a method for a teacher model to train a student model is proposed. Figure 3,include:
[0067] 1. The data processing device receives the data of the word segmenter, the data of the embedding layer, the data of the text output layer, the data of the first transformer layer, and the data of the second transformer layer of the teacher model.
[0068] 2. The data processing device locally trains the parameters of the first Transformer layer of the student model.
[0069] 3. Based on the trained first Transformer layer of the student model, the first device and the data processing device jointly perform distributed training on parameters of other Transformer layers of the student model.
[0070] In the above method of teacher model training student model, the implementation process of 2 is detailed in Figure 4 ,include:
[0071] 1. Perform word segmentation and embedding on the initial pre-stored training data to obtain the pre-stored training data.
[0072] 2. Input the pre-stored training data into the first and second Transformer layers of the teacher model.
[0073] 3. Get the first attention value and the first hidden state value of the second Transformer layer.
[0074] 4. Input the pre-stored training data into the initial first Transformer layer of the student model.
[0075] 5. Get the second attention value and the second hidden state value of the initial first-layer Transformer layer of the student model.
[0076] 6. Compare the first attention value and the second attention value, and the second hidden state value and the first hidden state value, and calculate a first loss value.
[0077] 7. Reverse update the parameters of the first Transformer layer of the student model according to the first loss value. In the above method of training the student model with the teacher model, the implementation process of 3 is detailed in Figure 5 ,include:
[0078] 1. Perform word segmentation and embedding on the initial pre-stored training data to obtain the pre-stored training data.
[0079] 2. Determine the confidentiality level information of the pre-stored training data.
[0080] 3. Input the first training data with a low confidentiality level into the first Transformer layer and the second Transformer layer of the teacher model to obtain the first hidden state information.
[0081] 4. Input the second training data with a high confidentiality level into the first Transformer layer of the student model to obtain the second hidden state information.
[0082] 5. Input the first hidden state information and the second hidden state word information into the student model's initial 2nd-th Transformer layer - Kth Transformer layer to obtain K-1 third attention values and K-1 third hidden state values.
[0083] 6. Send the first hidden state information and the second hidden state word information to the first device.
[0084] 7. The first device inputs the first hidden state information and the second hidden state word information into the 3rd Transformer layer to the Nth Transformer layer of the teacher model.
[0085] 8. The first device records the K-1 fourth attention values and K-1 fourth hidden state values of the corresponding layers of the initial 2nd Transformer layer to the Kth Transformer layer of the student model.
[0086] 9. The first device sends K-1 fourth attention values and K-1 fourth hidden state values to the data processing device.
[0087] 10. The data processing device compares K-1 third attention values and K-1 fourth attention values, and K-1 fourth hidden state values and K-1 third hidden state values to obtain a second loss value.
[0088] 11. The data processing device reversely updates the parameters of the 2-Kth Transformer layer of the student model according to the second loss value.
[0089] The present application embodiment provides a data processing device. Figure 6 As shown, the data processing device 1 includes:
[0090] A receiving unit 10 is used to receive data of the first two Transformer layers in the teacher model sent by the first device;
[0091] The training unit 11 is used to train the parameters of the first Transformer layer of the student model based on the data of the first two Transformer layers and the pre-stored training data on the data processing device; based on the trained first Transformer layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are jointly trained by the data processing device and the first device, where K is a positive integer greater than or equal to 2;
[0092] The construction unit 12 is used to construct the student model using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layer to process the natural language task.
[0093] Optionally, the data processing device further includes: a determination unit;
[0094] The determining unit is used to input the pre-stored training data into the first two Transformer layers to obtain a first attention value and a first hidden state value corresponding to the second Transformer layer of the teacher model; input the pre-stored training data into the initial first Transformer layer to obtain a second attention value and a second hidden state value;
[0095] The training unit 11 is also used to train and obtain parameters of the first Transformer layer according to the first attention value, the first hidden state value, the second attention value and the second hidden state value.
[0096] Optionally, the determining unit is further configured to determine a first loss value according to the first attention value, the first hidden state value, the second attention value, and the second hidden state value;
[0097] The training unit 11 is also used to reversely update the parameters of the initial first Transformer layer according to the first loss value until a stop training condition is met, thereby obtaining the parameters of the first Transformer layer.
[0098] Optionally, the data processing device further includes: an acquisition unit;
[0099] The acquisition unit is used to acquire the confidentiality level information of the pre-stored training data;
[0100] The determining unit is further configured to input the first training data in the pre-stored training data, whose confidentiality level information is a low confidentiality level, into the first two Transformer layers of the teacher model to obtain the first hidden state information; and input the second training data in the pre-stored training data, whose confidentiality level information is a high confidentiality level, into the first Transformer layer of the student model to obtain the second hidden state information;
[0101] The training unit 11 is also used to use the first hidden state information and the second hidden state information to jointly train the data processing device and the first device to obtain the parameters of the last K-1 layers of the Transformer layer of the student model.
[0102] Optionally, the data processing device further includes: a sending unit;
[0103] The determining unit is further used to input the first hidden state information and the second hidden state information into the initial and subsequent K-1 layers of the Transformer layer of the Xue Heng model to obtain K-1 third attention values and K-1 third hidden state values; and determine a second loss value using the K-1 third attention values, the K-1 third hidden state values, the K-1 fourth attention values, and the K-1 fourth hidden state values;
[0104] The sending unit is used to send the first hidden state information and the second hidden state information to the first device, and receive K-1 fourth attention values and K-1 fourth hidden state values of the layer corresponding to the last K-1 layer Transformer layer of the student model sent by the first device; wherein the K-1 fourth attention values and K-1 fourth hidden state values of the layer corresponding to the last K-1 layer Transformer layer of the student model are obtained after the first device inputs the first hidden state information and the second hidden state information into the 3rd to Nth layers Transformer layer of the teacher model; N is a positive integer greater than 2, and N is much greater than K;
[0105] The training unit 11 is also used to reversely update the parameters of the initial K-1-layer Transformer layer according to the second loss value until the training stop condition is met, thereby obtaining the parameters of the K-1-layer Transformer layer.
[0106] Optionally, the receiving unit 10 is further used to receive data of the first two Transformer layers, data of the preprocessing layer, and data of the output layer in the teacher model sent by the first device;
[0107] The training unit 11 is further used to train the parameters of the first Transformer layer of the student model based on the data of the first two Transformer layers, the data of the preprocessing layer, the data of the output layer and the pre-stored training data; based on the trained first Transformer layer, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are jointly trained by the data processing device and the first device, where K is a positive integer greater than or equal to 2;
[0108] The construction unit 12 is also used to construct the student model using the data of the preprocessing layer, the parameters of the first Transformer layer, the parameters of the last K-1 Transformer layer and the data of the output layer.
[0109] An embodiment of the present application provides a data processing device, which receives data of the first two Transformer layers in a teacher model sent by a first device; N is a positive integer; based on the data of the first two Transformer layers and pre-stored training data, parameters of the first Transformer layer of a student model are trained; based on the trained first Transformer layer and the pre-stored training data, parameters of the last K-1 Transformer layers of the student model are jointly trained by the data processing device and the first device, where K is a positive integer greater than or equal to 2; and the student model is constructed using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layers to process natural language tasks. It can be seen that in the data processing device proposed in this embodiment, the teacher model and the student model are distributedly deployed on different devices. The teacher model only transmits the data of the first two Transformer layers to the data processing device for model training, while the data of the last N-2 Transformer layers of the subject are still on the first device, thereby protecting the privacy of the teacher model data. For the data processing device, the pre-stored training data is used for training locally, and the original pre-stored training data will not be sent to the first device for training, thereby protecting the privacy of the training data, thereby avoiding data leakage and improving data security.
[0110] Figure 7 A schematic diagram of the structure of a data processing device 1 provided in an embodiment of the present application Figure 2 In practical applications, based on the same disclosure concept of the above embodiments, Figure 7 As shown, the data processing device 1 of this embodiment includes: a processor 13 , a memory 14 and a communication bus 15 .
[0111] The processor 13 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the processor function may also be other, which is not specifically limited in this embodiment.
[0112] In the embodiment of the present application, the communication bus 15 is used to realize the connection and communication between the processor 13 and the memory 14; when the processor 13 executes the running program stored in the memory 14, the following data processing method is implemented:
[0113] The data of the first two Transformer layers in the teacher model sent by the first device are received, and based on the data of the first two Transformer layers and the pre-stored training data, the parameters of the first Transformer layer of the student model are trained; based on the trained first Transformer layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are jointly trained by the data processing device and the first device, where K is a positive integer greater than or equal to 2; the student model is constructed using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layers to process natural language tasks.
[0114] Furthermore, the processor 13 is also used to input the pre-stored training data into the first two Transformer layers to obtain a first attention value and a first hidden state value corresponding to the second Transformer layer of the teacher model; input the pre-stored training data into the initial first Transformer layer to obtain a second attention value and a second hidden state value; and train the parameters of the first Transformer layer according to the first attention value, the first hidden state value, the second attention value and the second hidden state value.
[0115] Furthermore, the above-mentioned processor 13 is also used to determine a first loss value based on the first attention value, the first hidden state value, the second attention value and the second hidden state value; reversely update the parameters of the initial first layer Transformer layer according to the first loss value until the training stop condition is met, thereby obtaining the parameters of the first layer Transformer layer.
[0116] Furthermore, the processor 13 is also used to obtain confidentiality level information of the pre-stored training data; input the first training data in the pre-stored training data, whose confidentiality level information is a low confidentiality level, into the first two Transformer layers of the teacher model to obtain first hidden state information; input the second training data in the pre-stored training data, whose confidentiality level information is a high confidentiality level, into the first Transformer layer of the student model to obtain second hidden state information; and use the first hidden state information and the second hidden state information to jointly train the data processing device and the first device to obtain parameters of the last K-1 Transformer layers of the student model.
[0117] Furthermore, the processor 13 is further configured to input the first hidden state information and the second hidden state information into the initial K-1 layers of the student model's Transformer layer to obtain K-1 third attention values and K-1 third hidden state values; send the first hidden state information and the second hidden state information to the first device, and receive K-1 fourth attention values and K-1 fourth hidden state values of the layer corresponding to the K-1 layers of the student model's Transformer layer sent by the first device; wherein the K-1 fourth attention values of the layer corresponding to the K-1 layers of the student model's Transformer layer are and K-1 fourth hidden state values are obtained after the first device inputs the first hidden state information and the second hidden state information into the 3rd to Nth Transformer layers of the teacher model; N is a positive integer greater than 2, and N is much greater than K; the second loss value is determined using the K-1 third attention values, the K-1 third hidden state values, the K-1 fourth attention values and the K-1 fourth hidden state values; the parameters of the initial and subsequent K-1 layers of Transformer layers are reversely updated according to the second loss value until the training stop condition is met, thereby obtaining the parameters of the subsequent K-1 layers of Transformer layers.
[0118] Furthermore, the processor 13 is also used to receive the data of the first two Transformer layers, the data of the preprocessing layer and the data of the output layer in the teacher model sent by the first device; based on the data of the first two Transformer layers, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, train the parameters of the first Transformer layer of the student model; based on the trained first Transformer layer, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, the data processing device and the first device jointly train the parameters of the last K-1 Transformer layers of the student model, where K is a positive integer greater than or equal to 2; and construct the student model using the data of the preprocessing layer, the parameters of the first Transformer layer, the parameters of the last K-1 Transformer layers and the data of the output layer.
[0119] An embodiment of the present application provides a storage medium having a computer program stored thereon. The computer-readable storage medium stores one or more programs. The one or more programs can be executed by one or more processors and applied to a data processing device. The computer program implements the data processing method as described above.
[0120] Based on the above embodiments, an embodiment of the present application provides a computer program product, including a computer program, which can be executed by one or more processors, and the computer program implements the data processing method as described above.
[0121] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling an image display device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0123] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. A data processing method, characterized in that: Applied to a data processing device, the method comprises: Receive data of the first two Transformer layers in the teacher model sent by the first device; Based on the data of the first two Transformer layers and the pre-stored training data, training obtains the parameters of the first Transformer layer of the student model; Based on the first Transformer layer obtained through training and the pre-stored training data, the data processing device and the first device jointly train parameters of the last K-1 Transformer layers of the student model, where K is a positive integer greater than or equal to 2; The student model is constructed using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layers to process natural language tasks.
2. The method according to claim 1, characterized in that The training of the parameters of the first Transformer layer of the student model based on the data of the first two Transformer layers and the pre-stored training data includes: Input the pre-stored training data into the first two Transformer layers to obtain a first attention value and a first hidden state value corresponding to the second Transformer layer of the teacher model; Inputting the pre-stored training data into the initial first Transformer layer to obtain a second attention value and a second hidden state value; The parameters of the first Transformer layer are obtained by training according to the first attention value, the first hidden state value, the second attention value and the second hidden state value.
3. The method according to claim 2, characterized in that The training of obtaining parameters of the first Transformer layer according to the first attention value, the first hidden state value, the second attention value, and the second hidden state value includes: determining a first loss value according to the first attention value, the first hidden state value, the second attention value, and the second hidden state value; The parameters of the initial first Transformer layer are reversely updated according to the first loss value until a training stop condition is met, thereby obtaining the parameters of the first Transformer layer.
4. The method according to claim 1, characterized in that: The parameters of the last K-1 layers of the Transformer layer of the student model are obtained by joint training of the data processing device and the first device based on the first layer of the Transformer layer obtained through training and the pre-stored training data, including: Obtaining confidentiality level information of the pre-stored training data; Inputting the first training data of the pre-stored training data, whose confidentiality level information is a low confidentiality level, into the first two Transformer layers of the teacher model to obtain the first hidden state information; Inputting the second training data in the pre-stored training data, whose confidentiality level information is a high confidentiality level, into the first Transformer layer of the student model to obtain second hidden state information; The first hidden state information and the second hidden state information are used to jointly train the data processing device and the first device to obtain parameters of the last K-1 layers of the Transformer layer of the student model.
5. The method according to claim 4, characterized in that The method of using the first hidden state information and the second hidden state information to jointly train the data processing device and the first device to obtain parameters of the last K-1 layers of the Transformer layer of the student model includes: Inputting the first hidden state information and the second hidden state information into the initial and subsequent K-1 layers of the Transformer layer of the student model to obtain K-1 third attention values and K-1 third hidden state values; The first hidden state information and the second hidden state information are sent to the first device, and K-1 fourth attention values and K-1 fourth hidden state values of the layer corresponding to the last K-1 layer Transformer layer of the student model sent by the first device are received; wherein the K-1 fourth attention values and K-1 fourth hidden state values of the layer corresponding to the last K-1 layer Transformer layer of the student model are obtained after the first device inputs the first hidden state information and the second hidden state information into the 3rd to Nth layers Transformer layer of the teacher model; N is a positive integer greater than 2, and N is much greater than K; Determining a second loss value using the K-1 third attention values, the K-1 third hidden state values, the K-1 fourth attention values, and the K-1 fourth hidden state values; The parameters of the initial K-1-layer Transformer layer are reversely updated according to the second loss value until the training stop condition is met, thereby obtaining the parameters of the K-1-layer Transformer layer.
6. The method according to claim 1, characterized in that The method further comprises: Receiving data of the first two Transformer layers, data of the preprocessing layer, and data of the output layer in the teacher model sent by the first device; Based on the data of the first two Transformer layers, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, training obtains the parameters of the first Transformer layer of the student model; Based on the first Transformer layer obtained through training, the data of the preprocessing layer, the data of the output layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are obtained through joint training by the data processing device and the first device, where K is a positive integer greater than or equal to 2; The student model is constructed using the data of the preprocessing layer, the parameters of the first Transformer layer, the parameters of the last K-1 Transformer layers, and the data of the output layer.
7. A data processing device, characterized in that: The data processing device comprises: A receiving unit, configured to receive data of the first two Transformer layers in the teacher model sent by the first device; A training unit, configured to obtain parameters of the first Transformer layer of the student model through training based on the data of the first two Transformer layers and the pre-stored training data on the data processing device; based on the trained first Transformer layer and the pre-stored training data, the parameters of the last K-1 Transformer layers of the student model are obtained through joint training by the data processing device and the first device, where K is a positive integer greater than or equal to 2; A construction unit is used to construct the student model using the parameters of the first Transformer layer and the parameters of the last K-1 Transformer layer to process the natural language task.
8. A data processing device, characterized in that: The data processing device includes: a processor, a memory and a communication bus; the communication bus is used to realize the connection and communication between the processor and the memory; when the processor executes the running program stored in the memory, the method according to any one of claims 1 to 6 is implemented.
9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 6.