Model training methods, devices, storage media, clients, servers, and systems

Abstract

This disclosure provides a model training method, apparatus, storage medium, client, server, and system. By employing a federated learning approach, the server performs weighted aggregation of locally pre-trained language models from various clients to obtain a globally pre-trained language model. On the client side, the globally pre-trained language model sent by the server is first used to generate a pseudo-training sample set corresponding to the client's local training sample set. Then, the local pre-trained language model is updated based on the local training sample subset and the pseudo-training sample set. Finally, the client's current local pre-trained language model is sent to the server so that the server can perform weighted aggregation of the globally pre-trained language model. Ultimately, this allows for the utilization of labeled data from different users, improving the efficiency of labeled data utilization and enhancing the generalization ability of multi-task pre-trained language models.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of machine learning technology, specifically to model training methods, apparatus, storage media, clients, servers, and systems. Background Technology

[0002] In the field of natural language processing, various different tasks can be treated as unified generative tasks, and all tasks can be handled using pre-trained text-to-text language models. Summary of the Invention

[0003] Embodiments of this disclosure provide model training methods, apparatus, storage media, clients, servers, and systems.

[0004] In a first aspect, embodiments of this disclosure provide a global model training method applied to a server. The method includes: performing the following global model update operations: sending a global pre-trained language model to each client; receiving a local pre-trained language model returned by each client in response to the global pre-trained language model after local training; determining the weight coefficients of each local pre-trained language model; and weighting and aggregating each local pre-trained language model according to the corresponding weight coefficients to obtain an updated global pre-trained language model.

[0005] In some optional implementations, the global model update operation further includes: determining whether to end the global model update operation; and continuing to execute the global model update operation in response to determining that it has not ended.

[0006] In some optional implementations, receiving the local pre-trained language model returned by each client in response to the local training of the global pre-trained language model includes: receiving the local pre-trained language model returned by each client in response to the local training of the global pre-trained language model and the loss perturbation gradient of each local training sample in the corresponding local training sample set; and determining the weight coefficient of each of the local pre-trained language models includes: for each local pre-trained language model, determining the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model as the perturbation gradient sum of the local pre-trained language model; for each local pre-trained language model, determining the ratio of the perturbation gradient sum of the local pre-trained language model to the sum of the perturbation gradient sums of all local pre-trained language models as the proportion of the perturbation gradient sum of the local pre-trained language model; and for each local pre-trained language model, determining the weight coefficient of the local pre-trained language model based on the proportion of the perturbation gradient sum of the local pre-trained language model.

[0007] In some optional implementations, determining the weight coefficients of the local pre-trained language model based on the perturbation gradient and proportion of the local pre-trained language model includes: determining the larger of the perturbation gradient and proportion of the local pre-trained language model and a preset non-negative smaller weight coefficient as the weight coefficient of the local pre-trained language model.

[0008] In some optional implementations, determining whether to end the global model update operation includes: determining whether the number of times the global model update operation has been performed is less than a first preset number; in response to determining yes, determining not to end the global model update operation; in response to determining no, determining to end the global model update operation.

[0009] Secondly, embodiments of this disclosure provide a local model training method applied to a client. The method includes: in response to receiving a globally pre-trained language model from a server, inputting local sample data from a local training sample set into the globally pre-trained language model to obtain pseudo-labels corresponding to the local sample data, wherein the local training samples include local sample data and corresponding local labels; generating pseudo-training samples in a pseudo-training sample set using the local sample data from the local training sample set and the corresponding pseudo-labels; determining the globally pre-trained language model as the current local pre-trained language model, and dividing the local training sample set and the corresponding pseudo-training sample set into at least two batches of local training sample subsets and corresponding pseudo-training sample subsets; updating the current local pre-trained language model based on each batch of local training sample subsets and corresponding pseudo-training sample subsets; and sending the current local pre-trained language model to the server.

[0010] In some optional implementations, updating the current local pre-trained language model based on the local training sample subset and the corresponding pseudo-training sample subset includes: calculating a first loss of the current local pre-trained language model for the pseudo-training sample subset according to a first loss function; optimizing the current local pre-trained language model using the first loss to obtain an optimized local pre-trained language model; calculating a second loss of the optimized local pre-trained language model for the local training sample subset according to a second loss function, wherein the second loss function adds a corresponding loss perturbation to the loss of the local training samples; and backpropagating the second loss in the optimized local pre-trained language model to obtain a loss perturbation ladder for each local training sample and the corresponding pseudo-training sample in the local training sample subset. For each pseudo-training sample in the pseudo-training sample subset, the sample weight of the pseudo-training sample with respect to the current local pre-trained language model is determined based on the loss perturbation gradient of the pseudo-training sample. The third loss of the current local pre-trained language model for the local training sample subset and the corresponding pseudo-training sample subset is calculated according to the third loss function, where the third loss function is the sum of the fourth loss function and the first loss function weighted by the sample weights. The first loss function weighted by the sample weights is the loss function obtained by weighting the first loss function according to the sample weights of the corresponding pseudo-training samples. The fourth loss function is the loss of the current local pre-trained language model for the local training samples in the local training sample subset. The model parameters of the current local pre-trained language model are updated according to the third loss.

[0011] In some optional implementations, sending the current local pre-trained language model to the server includes: sending the current local pre-trained language model and the loss perturbation gradients of each pseudo-training sample in the pseudo-training sample set to the server.

[0012] In some optional implementations, determining the sample weight of the pseudo-training sample to the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample for each pseudo-training sample in the subset of pseudo-training samples includes: determining the larger of the preset non-negative smaller sample weight and the inverse of the loss perturbation gradient of the pseudo-training sample as the sample weight of the pseudo-training sample to the current pre-trained language model for each pseudo-training sample in the subset of pseudo-training samples.

[0013] In some optional implementations, determining the sample weight of each pseudo-training sample in the subset of pseudo-training samples based on the loss perturbation gradient of the pseudo-training sample to the current local pre-trained language model further includes: normalizing the sample weight of each pseudo-training sample in the subset of pseudo-training samples to the current pre-trained language model.

[0014] Thirdly, embodiments of this disclosure provide a local model training method applied to a client, the method comprising: performing a target natural language processing task using a locally pre-trained language model, wherein the locally pre-trained language model is trained by any of the local model training methods described in the second aspect.

[0015] Fourthly, embodiments of this disclosure provide a global model training apparatus applied to a server. The apparatus includes a global model update unit configured to perform the following global model update operations: sending a globally pre-trained language model to each client; receiving a locally pre-trained language model returned by each client in response to the globally pre-trained language model after local training; determining the weight coefficients of each of the locally pre-trained language models; and weighting and aggregating each of the locally pre-trained language models according to the corresponding weight coefficients to obtain an updated globally pre-trained language model.

[0016] In some optional implementations, the global model update operation further includes: determining whether to end the global model update operation; and continuing to execute the global model update operation in response to determining that it has not ended.

[0017] In some optional implementations, receiving the local pre-trained language model returned by each client in response to the local training of the global pre-trained language model includes: receiving the local pre-trained language model returned by each client in response to the local training of the global pre-trained language model and the loss perturbation gradient of each local training sample in the corresponding local training sample set; and determining the weight coefficient of each of the local pre-trained language models includes: for each local pre-trained language model, determining the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model as the perturbation gradient sum of the local pre-trained language model; for each local pre-trained language model, determining the ratio of the perturbation gradient sum of the local pre-trained language model to the sum of the perturbation gradient sums of all local pre-trained language models as the proportion of the perturbation gradient sum of the local pre-trained language model; and for each local pre-trained language model, determining the weight coefficient of the local pre-trained language model based on the proportion of the perturbation gradient sum of the local pre-trained language model.

[0018] In some optional implementations, determining the weight coefficients of the local pre-trained language model based on the perturbation gradient and proportion of the local pre-trained language model includes: determining the larger of the perturbation gradient and proportion of the local pre-trained language model and a preset non-negative smaller weight coefficient as the weight coefficient of the local pre-trained language model.

[0019] In some optional implementations, determining whether to end the global model update operation includes: determining whether the number of times the global model update operation has been performed is less than a first preset number; in response to determining yes, determining not to end the global model update operation; in response to determining no, determining to end the global model update operation.

[0020] Fifthly, embodiments of this disclosure provide a local model training apparatus applied to a client. The apparatus includes: a pseudo-label generation unit configured to, in response to receiving a globally pre-trained language model sent by a server, input local sample data from a local training sample set into the globally pre-trained language model to obtain pseudo-labels corresponding to the corresponding local sample data, wherein the local training samples include local sample data and corresponding local labels; a pseudo-sample generation unit configured to generate pseudo-training samples in a pseudo-training sample set using the local sample data from the local training sample set and the corresponding pseudo-labels; a sample batching unit configured to determine the globally pre-trained language model as the current local pre-trained language model, and to divide the local training sample set and the corresponding pseudo-training sample set into at least two batches of local training sample subsets and corresponding pseudo-training sample subsets; a local model updating unit configured to update the current local pre-trained language model based on each batch of local training sample subsets and corresponding pseudo-training sample subsets; and a local model sending unit configured to send the current local pre-trained language model to the server.

[0021] In some optional implementations, the local model update unit is further configured to: calculate a first loss of the current local pre-trained language model for the pseudo-training sample subset according to a first loss function, and optimize the current local pre-trained language model using the first loss to obtain an optimized local pre-trained language model; calculate a second loss of the optimized local pre-trained language model for the local training sample subset according to a second loss function, wherein the second loss function adds a corresponding loss perturbation to the loss of the local training samples, and backpropagate the second loss in the optimized local pre-trained language model to obtain the loss perturbation gradient of each local training sample and corresponding pseudo-training sample in the local training sample subset; for the pseudo-training sample subset... For each pseudo-training sample, the sample weight of the pseudo-training sample to the current local pre-trained language model is determined based on the loss perturbation gradient of the pseudo-training sample. A third loss function is calculated for the current local pre-trained language model for the subset of local training samples and the corresponding subset of pseudo-training samples, where the third loss function is the sum of a fourth loss function and a first loss function weighted by the sample weights. The first loss function weighted by the sample weights is obtained by weighting the first loss function according to the sample weights of the corresponding pseudo-training samples. The fourth loss function is the loss of the current local pre-trained language model for the local training samples in the subset of local training samples. The model parameters of the current local pre-trained language model are updated according to the third loss.

[0022] In some optional implementations, the local model sending unit is further configured to send the current local pre-trained language model and the loss perturbation gradients of each pseudo-training sample in the pseudo-training sample set to the server.

[0023] In some optional implementations, determining the sample weight of the pseudo-training sample to the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample for each pseudo-training sample in the subset of pseudo-training samples includes: determining the larger of the preset non-negative smaller sample weight and the inverse of the loss perturbation gradient of the pseudo-training sample as the sample weight of the pseudo-training sample to the current pre-trained language model for each pseudo-training sample in the subset of pseudo-training samples.

[0024] In some optional implementations, determining the sample weight of each pseudo-training sample in the subset of pseudo-training samples based on the loss perturbation gradient of the pseudo-training sample to the current local pre-trained language model further includes: normalizing the sample weight of each pseudo-training sample in the subset of pseudo-training samples to the current pre-trained language model.

[0025] In a sixth aspect, embodiments of this disclosure provide a task processing apparatus applied to a client, the apparatus comprising: a task processing unit configured to perform a target natural language processing task using a locally pre-trained language model, wherein the locally pre-trained language model is trained using a local model training method as described in any implementation of the second aspect.

[0026] In a seventh aspect, embodiments of this disclosure provide a server, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0027] Eighthly, embodiments of this disclosure provide a client, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the second aspect.

[0028] Ninthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the methods described in any implementation of the first aspect and / or the methods described in any implementation of the second aspect.

[0029] In a tenth aspect, embodiments of this disclosure provide a model training system, including a server as described in any implementation of the seventh aspect and a client as described in any implementation of the eighth aspect.

[0030] Currently, large-scale pre-trained models have achieved good results in multi-task scenarios, and generally, the more tasks and data available, the more significant the improvement. However, in real-world scenarios, different tasks are often distributed across the computing devices of different users / groups of users. Due to data compliance requirements, data cannot be directly shared between different users / tenants, leading to the phenomenon of "data silos." Different users have their own unique data and task types, and they cannot directly communicate with each other. Furthermore, because data labeling is expensive, each task typically has only a small amount of labeled training data, resulting in weak model capabilities and poor generalization.

[0031] The model training method, apparatus, storage medium, client, server, and system provided in the embodiments of this disclosure employ a federated learning method. On the server side, local pre-trained language models from each client are weighted and aggregated to obtain a global pre-trained language model. On the client side, the global pre-trained language model sent by the server is first used to generate a pseudo-training sample set corresponding to the client's local training sample set. Then, the local pre-trained language model is updated based on the local training sample subset and the pseudo-training sample set. Finally, the client's current local pre-trained language model is sent to the server so that the server can perform weighted aggregation of the global pre-trained language model. Ultimately, this allows for the utilization of labeled data from different users, improving the efficiency of labeled data utilization and enhancing the generalization ability of multi-task pre-trained language models. Attached Figure Description

[0032] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0033] Figure 1 This is a system architecture diagram of an embodiment of the model training system according to the present disclosure;

[0034] Figure 2A This is a timing diagram of one embodiment of the model training system according to the present disclosure;

[0035] Figure 2B This is an exploded flowchart of one embodiment of step 205 of this disclosure;

[0036] Figure 2C This is a schematic diagram illustrating an application scenario of an embodiment of the model training system according to the present disclosure;

[0037] Figure 3 This is a flowchart of an embodiment of a global model training method applied to a server according to the present disclosure;

[0038] Figure 4 This is a flowchart of an embodiment of a local model training method applied to a client according to the present disclosure;

[0039] Figure 5 This is a schematic diagram of a structure of an embodiment of a global model training apparatus applied to a server according to the present disclosure;

[0040] Figure 6 This is a schematic diagram of a structure of an embodiment of a local model training device applied to a client according to the present disclosure;

[0041] Figure 7This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure, either as a server or a client. Detailed Implementation

[0042] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] Figure 1 An exemplary system architecture 100 is shown, which is an embodiment of the model training system to which this disclosure can be applied.

[0045] like Figure 1 As shown, the model training system 100 may include clients 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium to provide a communication link between clients 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0046] Users can use clients 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on clients 101, 102, and 103, such as model training applications, natural language processing applications, audio and video conferencing applications, speech recognition applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0047] Clients 101, 102, and 103 can be either hardware or software. When clients 101, 102, and 103 are hardware, they can be various electronic devices with displays and supporting audio and / or video capture, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When clients 101, 102, and 103 are hardware, they can also be a single server or a distributed server cluster consisting of multiple servers providing various natural language processing services. When clients 101, 102, and 103 are software, they can be installed on the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0048] Server 105 can be a server that provides various services, such as a backend server that supports natural language processing applications displayed on clients 101, 102, and 103. The backend server can analyze and process received data such as local pre-trained language models, and feed back the processing results (such as global pre-trained language models) to the clients.

[0049] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0050] It should be noted that the local model training method for the client provided in this disclosure is generally executed by the client 101, 102, and 103, and correspondingly, the local model training device for the client is generally set in the client 101, 102, and 103.

[0051] It should be noted that the global model training method for the server provided in this disclosure is generally executed by the server 105, and correspondingly, the global model training device for the server is generally set in the server 105.

[0052] It should be understood that Figure 1 The number of clients, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of clients, networks, and servers.

[0053] Continue to refer to Figure 2A This illustrates timing 200 of an embodiment of a model training system according to the present disclosure. The model training system in this embodiment may include a client and a server. Timing 200 includes the following steps:

[0054] Step 201: The server sends the global pre-trained language model to each client.

[0055] In this embodiment, the server can send the globally pre-trained language model to each client. Here, each client can be a client that has a model training service binding relationship with the server. Each client trains its own local pre-trained language model based on its local training sample set and the globally pre-trained language model received from the server. The globally pre-trained language model can have the same model structure and model parameter settings as the client's local pre-trained language model, but the values ​​of the model parameters can be the same or different.

[0056] Step 202: In response to receiving the global pre-trained language model from the server, the client inputs the local sample data from the local training sample set into the global pre-trained language model to obtain pseudo-labels corresponding to the local sample data.

[0057] Here, the client can store a local training sample set, which includes local sample data and corresponding local labels. The local labels for the local sample data can be manually annotated. The local sample data from the local training sample set is then input into the global pre-trained language model to obtain pseudo-labels corresponding to the local sample data. These pseudo-labels are used because the global pre-trained language model is a composite of the training results from various clients, i.e., the local pre-trained language models from each client. It may not perform well on a specific client's local training sample set; therefore, the resulting labels can be called pseudo-labels.

[0058] Step 203: The client uses the local sample data in the local training sample set and the corresponding pseudo-labels to generate pseudo-training samples in the pseudo-training sample set.

[0059] Step 204: The client determines the global pre-trained language model as the current local pre-trained language model, and divides the local training sample set and the corresponding pseudo-training sample set into at least two batches of local training sample subsets and corresponding pseudo-training sample subsets.

[0060] Here, the local training sample set and the pseudo training sample set are divided into at least two batches of local training sample subsets and corresponding pseudo training sample subsets.

[0061] Step 205: For each batch of local training sample subsets and corresponding pseudo-training sample subsets, the client updates the current local pre-trained language model based on the local training sample subsets and corresponding pseudo-training sample subsets.

[0062] Here, the client can use various implementation methods to update the current local pre-trained language model based on each batch of local training sample subsets and corresponding pseudo-training sample subsets. After at least two batches of updates to the local pre-trained language model, the model parameters of the client's local pre-trained language model will be optimized.

[0063] Step 206: The client sends the current local pre-trained language model to the server.

[0064] After step 205, the model parameters of the client's local pre-trained language model have been optimized. Therefore, the client can send its current local pre-trained language model to the server so that the server can aggregate it to form a global pre-trained language model.

[0065] Step 207: The server receives the local pre-trained language model returned by each client after the global pre-trained language model has been trained locally.

[0066] Step 208: The server determines the weight coefficients of each local pre-trained language model.

[0067] Here, the server can use various implementation methods to determine the weight coefficients of the local pre-trained language models returned by each client, so that the different local pre-trained language models can be weighted and aggregated in subsequent steps.

[0068] As an example, the server can determine the weight coefficient of the local pre-trained language model returned by each client as the ratio of the number of training samples corresponding to that client to the sum of the number of training samples corresponding to all clients.

[0069] Assume there are K clients in total, u i Let i be the i-th client, where i is a positive integer between 1 and K. Specifically, this can be expressed by the following formula:

[0070]

[0071] Where, n i Let be the number of training samples for the i-th client, and δ i The weight coefficients of the local pre-trained language model returned by the i-th client are calculated.

[0072] Step 209: The server performs weighted aggregation on each local pre-trained language model according to the corresponding weight coefficients to obtain the updated global pre-trained language model.

[0073] Continuing with the above assumptions, and assuming F i For client u i The returned local pre-trained language model can then be represented by the following formula:

[0074]

[0075] Among them, F global This is the globally pre-trained language model obtained through computation.

[0076] After steps 201 to 209, both the globally pre-trained language model stored on the server and the locally pre-trained language model stored on the client are updated and optimized.

[0077] In some alternative implementations, the above process 200 may further include the following step 210:

[0078] Step 210: The server determines whether to end the global model update operation.

[0079] The global model update operation here can include steps 201 to 209. The server can determine whether to end the global model update operation based on various implementation methods according to actual needs. If it is determined that the global model update operation should not be ended, it can proceed to step 201 to continue execution.

[0080] As an example, the server can determine to end the global model update operation if the overall difference between the globally pre-trained language model and the locally pre-trained language models sent by each client is less than a preset difference threshold; otherwise, if the difference is not less than the preset difference threshold, it determines not to end the global model update operation and proceeds to step 201 to continue execution. Here, various methods can be used to calculate the difference between models. For example, suppose models M1 and M2 each have N identical parameters, but the parameter values ​​can be different. The N parameter values ​​of model M1 can be used to form an N-dimensional vector V1, and the N parameter values ​​of model M2 can be used to form an N-dimensional vector V2. Then, the difference between model M1 and model M2 can be transformed into a distance problem between vectors V1 and V2. For example, Euclidean distance can be used to calculate the difference between model M1 and model M2.

[0081] Alternatively, step 210 can also be performed as follows:

[0082] Determine if the number of times the global model update operation has been performed is less than a first preset number. If it is less, then the global model update operation is not terminated, and the process proceeds to step 201. Otherwise, if it is not less, then the global model update operation is terminated. In this way, steps 201 to 209 are executed a first preset number of times, achieving a sufficient number of training iterations for both the global pre-trained language model and each local pre-trained language model.

[0083] In some alternative implementations, step 205 may include, for example: Figure 2B Steps 2051 to 2054 are shown below:

[0084] Step 2051: Calculate the first loss of the current local pre-trained language model for the pseudo-training sample subset according to the first loss function, and optimize the current local pre-trained language model using the first loss to obtain the optimized local pre-trained language model.

[0085] Here, we can assume there are K clients, u i Let i be the i-th client, where i is a positive integer between 1 and K. (X)i ,Y i ) for client u i The local training sample set, where X i For client u i The local sample data set, Y i For client u i A collection of local tags. For client u i The set of pseudo-training samples, where, For client u i A collection of pseudo-tags.

[0086] For client u i The j-th local training sample subset, For client u i and the local training sample subset The corresponding j-th pseudo-training sample subset, F i For client u i The current local pre-trained language model, for and Local sample data in the database.

[0087] Here, step 2051 can be expressed by the following formula:

[0088]

[0089]

[0090]

[0091] in:

[0092] This refers to the subset of pseudo-samples. Local sample data input client u i The current local pre-trained language model F i Here, the actual output of the model is

[0093] This refers to a subset of pseudo-samples. Each sample data in Calculate the sample data Input client u i The current local pre-trained language model F i Actual output results With pseudo-sample subset The first loss function between the corresponding pseudo-labels Then calculate the pseudo-sample subset Each sample data The sum of the corresponding first loss functions b i for and The number of sample data. Here, That is, the client u i The current local pre-trained language model F i For the pseudo-training sample subset The first loss.

[0094] This refers to the first loss obtained using the above method. For client u i The current local pre-trained language model F i Optimization is performed to obtain the optimized local pre-trained language model F′. i It should be noted that here the client u i The current local pre-trained language model F i It itself has not been updated.

[0095] Step 2052: Calculate the second loss of the optimized local pre-trained language model for the local training sample subset according to the second loss function.

[0096] Here, the second loss function adds a corresponding loss perturbation to the loss of the local training samples.

[0097] Continuing with the above assumptions, step 2052 can be expressed by the following formula:

[0098]

[0099]

[0100] in, This refers to a subset of local samples. Local sample data Input client u i The current local pre-trained language model F i Here, the actual output of the model is

[0101] This refers to a subset of local samples. Each sample data in Calculate the sample data Input client u i Optimization of the local pre-trained language model F′i Actual output results with local sample subset The second loss function between the corresponding local labels Then, for the local sample subset Each sample data The corresponding second loss function Perform according to loss disturbance ∈ j We obtain by weighted summation here, That is, the client u i Optimization of the local pre-trained language model F′ i For the local training sample subset The second loss. In practice, the local sample subset Each sample data The corresponding loss perturbation ∈ j These can be the same constant, for example, they can be 1; local sample subset Each sample data The corresponding loss perturbation ∈ j It can also be different values, for example, it can randomly take values ​​between 0 and 1.

[0102] It should be noted that the first loss function and the second loss function here can be various loss functions that are currently known or will be developed in the future, and this disclosure does not make any specific restrictions on them.

[0103] Step 2053: Backpropagate the second loss in the optimized local pre-trained language model to obtain the loss perturbation gradient of each local training sample and the corresponding pseudo-training sample in the local training sample subset.

[0104] Step 2052 can be expressed by the following formula:

[0105]

[0106] in, For a subset of local training samples The loss perturbation gradient for each local training sample and its corresponding pseudo-training sample. That is, There is b i Each loss perturbation gradient.

[0107] Step 2054: For each pseudo-training sample in the subset of pseudo-training samples, determine the sample weight of the pseudo-training sample to the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample.

[0108] In step 2051, client u i The current local pre-trained language model Fi Learning pseudo-sample subsets Chinese counterfeit labels The distribution and obtained the client u i Optimization of the local pre-trained language model F′ i .

[0109] In steps 2052 and 5053, the client u i Optimization of the local pre-trained language model F′ i Local training sample subset Local tags in The training is performed on the above, and the second loss is calculated by calculating F′. i and local tags The differences between them are calculated, and the corresponding loss perturbation gradient is calculated.

[0110] Here, the loss perturbation gradient is lost. Calculated to measure the subset of local training samples Local tags in and pseudo-sample subset Chinese counterfeit labels The similarity between them. And the loss perturbation gradient. The larger the value, the less similar the pseudo-labels are to the local labels; in other words, the pseudo-training samples are less relevant to the client u. i The smaller the gain, the smaller the weight of the corresponding pseudo-training sample should be. Therefore, here, the sample weight of the pseudo-training sample in the current local pre-trained language model is negatively correlated with the loss perturbation gradient of the pseudo-training sample. In practice, various negative correlation methods can be used to determine the sample weight of each pseudo-training sample in the subset of pseudo-training samples in the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample.

[0111] Optionally, step 2054 can be performed as follows: for the subset of pseudo-training samples Each pseudo-training sample The preset non-negative small sample weight c and the pseudo-training sample are used. loss perturbation gradient the opposite number The larger of the two This was identified as a pseudo-training sample. For the current pre-trained language model F i Sample weights

[0112] Here, the preset non-negative smaller sample weights can be preset non-negative values, such as 0.

[0113] Specifically, this optional method can be expressed by the following formula:

[0114]

[0115] Where c is the preset non-negative weight of a small sample. From the above formula, it can be seen that... If the value is non-negative, it indicates that the pseudo-labels and local labels are not similar, meaning they are pseudo-training samples. For client u i The less gain there is, the more likely the pseudo-training samples are to be affected. The samples should have small non-negative weights c; when c is 0, the pseudo-training samples... For client u i The gain is 0, meaning there is no gain.

[0116] exist A negative number indicates that the pseudo-labels and local labels are quite similar, meaning they are pseudo-training samples. For client u i The gain is relatively large, and The smaller the number of pseudo-training samples For client u i The greater the gain.

[0117] Using the above optional methods, the pseudo-training sample subset can be obtained. Each pseudo-training sample Non-negative sample weights

[0118] Optionally, after obtaining this pseudo-training sample subset... Each pseudo-training sample Non-negative sample weights After that, it is also possible to analyze the subset of pseudo-training samples. Each pseudo-training sample For the current pre-trained language model F i Sample weights Normalization is then performed. Specifically, this can be expressed by the following formula:

[0119]

[0120] in, The pseudo-training sample subset obtained after calculation Each pseudo-training sample For the current pre-trained language model F i The sample weights.

[0121] Step 2055: Calculate the third loss of the current local pre-trained language model for the local training sample subset and the corresponding pseudo-training sample subset according to the third loss function.

[0122] The third loss function is the sum of the fourth loss function and the first loss function weighted by the sample weights. The first loss function weighted by the sample weights is the loss function obtained by weighting the first loss function according to the sample weights of the corresponding pseudo-training samples. The fourth loss function is the loss of the current local pre-trained language model for the local training samples in the local training sample subset.

[0123] Specifically, it can be expressed by the following formula:

[0124]

[0125] in, This is the third loss function. This is the fourth loss function. For the current local pre-trained language model F i For this subset of local training samples and the corresponding pseudo-training sample subset The third loss.

[0126] Step 2056: Update the model parameters of the current local pre-trained language model according to the third loss.

[0127] Specifically, this can be expressed by the following formula:

[0128]

[0129] That is, according to the third loss For the current local pre-trained language model F i The model parameters are updated.

[0130] Through the above optional implementation methods, in addition to being able to target client u i The current local pre-trained language model F i In addition to updating the model parameters, it can also obtain the client u i pseudo-training sample set The loss perturbation gradient corresponding to each pseudo-training sample

[0131] Based on the above optional implementation methods, in step 206, the client sends the current local pre-trained language model to the server, which can be performed as follows:

[0132] The client sends the loss perturbation gradient of the current local pre-trained language model and each pseudo-training sample in the pseudo-training sample set to the server.

[0133] Therefore, in addition to receiving client u i The client's current local pre-trained language model F was sent. iIn addition, it can also receive a set of pseudo-training samples from the client. Loss perturbation gradient of each pseudo-training sample

[0134] Accordingly, optionally, the server's determination of the weight coefficients for each local pre-trained language model in step 208 can also be performed as follows:

[0135] First, for each local pre-trained language model, the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model is determined as the perturbation gradient sum of the local pre-trained language model.

[0136] Then, for each local pre-trained language model, the ratio of the perturbation gradient of that local pre-trained language model to the sum of the perturbation gradients of all local pre-trained language models is determined as the proportion of the perturbation gradient of that local pre-trained language model.

[0137] Continuing with the example above, specifically, client u i The returned locally pre-trained language model F i The perturbation gradient and its proportion can be expressed by the following formula:

[0138]

[0139] Where, p i The calculated client u i The returned locally pre-trained language model F i The perturbation gradient and proportion.

[0140] Finally, for each local pre-trained language model, the weight coefficients of the local pre-trained language model are determined based on the perturbation gradient and proportion of the local pre-trained language model.

[0141] Here, various methods can be used to determine the weight coefficients of the local pre-trained language model based on its perturbation gradient and proportion. Here, the weight coefficients of the local pre-trained language model are positively correlated.

[0142] Optionally, to ensure that the weight coefficients of the local pre-trained language model are non-negative and to facilitate subsequent weighted aggregation of the global pre-trained language model, the larger of the perturbation gradient and its proportion of the local pre-trained language model and the preset non-negative smaller weight coefficients can be determined as the weight coefficients of the local pre-trained language model.

[0143] Specifically, it can be expressed by the following formula:

[0144] δ i =max(p i ,wc )

[0145] Among them, w c This is a preset non-negative, relatively small weighting coefficient. For example, the preset non-negative, relatively small weighting coefficient can be zero.

[0146] This ensures that the weight coefficients of the local pre-trained language model are non-negative, and that the minimum weight coefficient is a preset non-negative small weight coefficient.

[0147] Using the optional implementation of step 205 above, pseudo-training samples are generated by using the globally pre-trained language model sent by the server for data augmentation, reducing the requirement for the amount of labeled training sample data and thus reducing the cost of labeling training samples. Furthermore, by re-weighting the generated pseudo-training samples using a meta-learning method, and by adjusting the weights of different client models during global model weighting and aggregation based on the client's loss perturbation gradient, it is possible to weight each client's local pre-trained language model according to the contribution of each client's training samples to the global pre-trained language model.

[0148] In some alternative implementations, process 200 may further include the following step 211:

[0149] Step 211: The client uses a local pre-trained language model to perform the target natural language processing task.

[0150] After training its own local pre-trained language model on the client side, the client can use the trained local pre-trained language model to perform the target natural language processing task. Here, the target natural language processing task is the natural language processing task provided by the client and corresponding to the client's own pre-trained language model.

[0151] It should be noted that the natural language processing tasks for different clients can be the same or different.

[0152] The model training system provided in the above embodiments of this disclosure employs a federated learning method to perform weighted aggregation of local pre-trained language models from various clients on the server side to obtain a global pre-trained language model. On the client side, the global pre-trained language model sent by the server is first used to generate a pseudo-training sample set corresponding to the client's local training sample set. Then, the local pre-trained language model is updated based on the local training sample subset and the pseudo-training sample set. Finally, the client's current local pre-trained language model is sent to the server so that the server can perform weighted aggregation of the global pre-trained language model. Ultimately, this system can utilize the labeled data from different users, improving the efficiency of labeled data utilization and enhancing the generalization ability of multi-task pre-trained language models.

[0153] The following is for reference. Figure 2C, Figure 2C A schematic diagram illustrating an application scenario of an embodiment of the model training system according to this disclosure is shown. For example... Figure 2C As shown, the system comprises 220 servers and K clients 231, ..., 23K. Each client can correspond to a different user (individual user or group user), and data cannot be shared between different clients.

[0154] Server 220 receives locally pre-trained language models F1, F2, ..., F from K clients. K Then, the weight coefficients of each local pre-trained language model are determined, and the global pre-trained language model F is obtained after weighted aggregation. global .

[0155] Each client utilizes the globally pre-trained language model F sent by server 220. global A pseudo-training sample set is generated from the local training sample set. Then, the model is trained based on the local training sample set, the global pre-trained language model, and the pseudo-training sample set to obtain the client's local pre-trained language model, compensating for the lack of labeled training samples. After the local pre-trained language model is updated, the client can send the updated local pre-trained language model to server 220 for weighted aggregation to obtain the global pre-trained language model.

[0156] Optionally, when determining the weight coefficients of the local pre-trained language model, the server 220 can use the loss perturbation sent by the client. This method can make the global pre-trained language model F... global More attention is being paid to locally pre-trained language models with weak performance.

[0157] Continue to refer to Figure 3 The diagram illustrates flow 300 of an embodiment of a global model training method according to this disclosure. This model training method, applied to a server, includes the following steps:

[0158] Step 301: Send the global pre-trained language model to each client.

[0159] Step 302: Receive the local pre-trained language model returned by each client after the global pre-trained language model has been trained locally.

[0160] Step 303: Determine the weight coefficients of each local pre-trained language model.

[0161] Step 304: Weight and aggregate each local pre-trained language model according to the corresponding weight coefficients to obtain the updated global pre-trained language model.

[0162] In this embodiment, the specific operations of steps 301, 302, 303, and 304 and their resulting technical effects are described below. Figure 2A In the embodiments shown, the operations and effects of steps 201, 207, 208 and 209 are basically the same, and will not be described again here.

[0163] In some alternative implementations, the above process 300 may further include the following step 305:

[0164] Step 305: Determine whether to end the global model update operation.

[0165] Here, the specific operation of the optional implementation of step 305 and its resulting technical effects are described in conjunction with... Figure 2A The operations and effects described in step 210 of the illustrated embodiments are basically the same, and will not be repeated here.

[0166] The global model update operation here may include steps 301, 302, 303, and 304. The server can determine whether to end the global model update operation based on various implementation methods according to actual needs. If it is determined that the global model update operation should not be ended, execution can proceed to step 301.

[0167] The method provided in the above embodiments of this disclosure involves the server sending a globally pre-trained language model to the client, and the server performing weighted aggregation of the local pre-trained language models from each client to obtain a globally pre-trained language model. This enables training the globally pre-trained language model using labeled data from different clients, and also allows the local pre-trained language models of each client to utilize labeled data from other clients, thereby improving the generalization ability of both the globally pre-trained language model and the client's local pre-trained language models.

[0168] Continue to refer to Figure 4 The diagram illustrates a flow 400 of an embodiment of a local model training method according to this disclosure. This model training method, applied to a client, includes the following steps:

[0169] Step 401: In response to receiving the global pre-trained language model sent by the server, input the local sample data in the local training sample set into the global pre-trained language model respectively to obtain the pseudo-labels corresponding to the local sample data.

[0170] Step 402: Use the local sample data and corresponding pseudo-labels in the local training sample set to generate pseudo-training samples in the pseudo-training sample set.

[0171] Step 403: Determine the global pre-trained language model as the current local pre-trained language model.

[0172] Step 404: Divide the local training sample set and the corresponding pseudo training sample set into at least two batches of local training sample subsets and corresponding pseudo training sample subsets.

[0173] Step 405: For each batch of local training sample subset and corresponding pseudo training sample subset, update the current local pre-trained language model based on the local training sample subset and corresponding pseudo training sample subset.

[0174] Step 406: Send the current local pre-trained language model to the server.

[0175] In this embodiment, the specific operations of steps 401, 402, 403, 404, 405, and 406 and their resulting technical effects are described below. Figure 2A In the illustrated embodiments, the operations and effects of steps 202, 203, 204, 205, 206, and 207 are basically the same, and will not be described again here.

[0176] The local model training method provided in the above embodiments of this disclosure first uses the globally pre-trained language model sent by the server on the client to generate a pseudo-training sample set corresponding to the client's local training sample set. Then, the local pre-trained language model is updated based on the local training sample subset and the pseudo-training sample set. Finally, the client's current local pre-trained language model is sent to the server so that the server can perform weighted aggregation of the globally pre-trained language model and train the client's own local pre-trained language model. This allows the use of labeled data from different clients, improving the efficiency of labeled data utilization and enhancing the generalization ability of multi-task pre-trained language models.

[0177] This disclosure also provides an embodiment of a task processing method. This task processing method, applied to a client, includes:

[0178] Perform the target natural language processing task using a locally pre-trained language model.

[0179] Here, the client's local pre-trained language model is achieved through methods such as... Figure 4 The local model training method shown in the embodiments and its optional implementations is pre-trained.

[0180] Here, the target natural language processing task is the natural language processing task provided by the client that corresponds to the client's locally pre-trained language model. For example, the target natural language processing task could be a machine translation task.

[0181] It should be noted that the natural language processing tasks for different clients can be the same or different.

[0182] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a global model training device, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various servers.

[0183] like Figure 5 As shown, the global model training device 500 of this embodiment includes a global model update unit 501, configured to perform the following global model update operations: sending a global pre-trained language model to each client; receiving a local pre-trained language model returned by each client in response to the local training of the global pre-trained language model; determining the weight coefficients of each local pre-trained language model; and weighting and aggregating each local pre-trained language model according to the corresponding weight coefficients to obtain an updated global pre-trained language model.

[0184] In this embodiment, the specific processing of the global model update unit 501 of the global model training device 500 and the resulting technical effects can be referred to separately. Figure 3 The relevant descriptions of steps 301, 302, 303 and 304 in the corresponding embodiments will not be repeated here.

[0185] In some optional implementations, the global model update operation may further include: determining whether to end the global model update operation; and continuing to execute the global model update operation in response to determining that it has not ended.

[0186] In some optional implementations, receiving the local pre-trained language model returned by each client in response to the local training of the global pre-trained language model may include: receiving the local pre-trained language model and the loss perturbation gradient of each local training sample in the corresponding local training sample set returned by each client in response to the local training of the global pre-trained language model; and determining the weight coefficient of each of the local pre-trained language models may include: for each local pre-trained language model, determining the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model as the perturbation gradient sum of the local pre-trained language model; for each local pre-trained language model, determining the ratio of the perturbation gradient sum of the local pre-trained language model to the sum of the perturbation gradient sums of all local pre-trained language models as the proportion of the perturbation gradient sum of the local pre-trained language model; and for each local pre-trained language model, determining the weight coefficient of the local pre-trained language model based on the proportion of the perturbation gradient sum of the local pre-trained language model.

[0187] In some optional implementations, determining the weight coefficients of the local pre-trained language model based on the perturbation gradient and proportion of the local pre-trained language model may include: for each local pre-trained language model, performing the following weight coefficient determination operation: determining whether the perturbation gradient and proportion of the local pre-trained language model are positive; in response to determining yes, determining the perturbation gradient and proportion of the local pre-trained language model as the weight coefficients of the local pre-trained language model; in response to determining no, determining a preset non-negative smaller weight coefficient as the weight coefficients of the local pre-trained language model.

[0188] In some optional implementations, determining whether to end the global model update operation may include: determining whether the number of times the global model update operation has been performed is less than a first preset number; in response to determining yes, determining not to end the global model update operation; in response to determining no, determining to end the global model update operation.

[0189] It should be noted that the implementation details and technical effects of each unit in the global model training device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0190] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a local model training device, which is similar to... Figure 4 Corresponding to the method embodiments shown, the device can be specifically applied to various clients.

[0191] like Figure 6As shown, the local model training device 600 in this embodiment includes: a pseudo-label generation unit 601, a pseudo-sample generation unit 602, a sample batching unit 603, a local model update unit 604, and a local model sending unit 605. The system includes the following components: a pseudo-label generation unit 601, configured to, in response to receiving a global pre-trained language model from a server, input local sample data from a local training sample set into the global pre-trained language model to obtain pseudo-labels corresponding to the local sample data, wherein the local training samples include local sample data and corresponding local labels; a pseudo-sample generation unit 602, configured to use the local sample data from the local training sample set and the corresponding pseudo-labels to generate pseudo-training samples in a pseudo-training sample set; a sample batching unit 603, configured to determine the global pre-trained language model as the current local pre-trained language model, and to divide the local training sample set and the corresponding pseudo-training sample set into at least two batches of local training sample subsets and corresponding pseudo-training sample subsets; a local model update unit 604, configured to update the current local pre-trained language model based on each batch of local training sample subsets and corresponding pseudo-training sample subsets; and a local model sending unit 605, configured to send the current local pre-trained language model to the server.

[0192] In this embodiment, the specific processing of the pseudo-label generation unit 601, pseudo-sample generation unit 602, sample batching unit 603, local model update unit 604, and local model sending unit 605 of the model training device 600, and the resulting technical effects, can be found in reference to [the relevant documentation]. Figure 4 The relevant descriptions of steps 401, 402, 403, 404 and 405 in the corresponding embodiments will not be repeated here.

[0193] In some optional implementations, the local model update unit 604 may be further configured to: calculate a first loss of the current local pre-trained language model for the pseudo-training sample subset according to a first loss function, and optimize the current local pre-trained language model using the first loss to obtain an optimized local pre-trained language model; calculate a second loss of the optimized local pre-trained language model for the local training sample subset according to a second loss function, wherein the second loss function adds a corresponding loss perturbation to the loss of the local training samples, and backpropagates the second loss in the optimized local pre-trained language model to obtain the loss perturbation gradient of each local training sample and corresponding pseudo-training sample in the local training sample subset; for the pseudo-training sample subset... For each pseudo-training sample, the sample weight of the pseudo-training sample in the current local pre-trained language model is determined based on the loss perturbation gradient of the pseudo-training sample. A third loss function is calculated for the current local pre-trained language model for the subset of local training samples and the corresponding subset of pseudo-training samples, where the third loss function is the sum of a fourth loss function and a first loss function weighted by the sample weights. The first loss function weighted by the sample weights is the loss function obtained by weighting the first loss function according to the sample weights of the corresponding pseudo-training samples. The fourth loss function is the loss of the current local pre-trained language model for the local training samples in the subset of local training samples. The model parameters of the current local pre-trained language model are updated according to the third loss.

[0194] In some optional implementations, the local model sending unit 605 may be further configured to send the loss perturbation gradient of the current local pre-trained language model and each pseudo-training sample in the pseudo-training sample set to the server.

[0195] In some optional implementations, determining the sample weight of the pseudo-training sample to the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample for each pseudo-training sample in the subset of pseudo-training samples may include: determining the larger of the preset non-negative smaller sample weight and the inverse of the loss perturbation gradient of the pseudo-training sample as the sample weight of the pseudo-training sample to the current pre-trained language model for each pseudo-training sample in the subset of pseudo-training samples.

[0196] In some optional implementations, determining the sample weight of each pseudo-training sample in the subset of pseudo-training samples based on the loss perturbation gradient of the pseudo-training sample to the current local pre-trained language model may further include: normalizing the sample weight of each pseudo-training sample in the subset of pseudo-training samples to the current pre-trained language model.

[0197] It should be noted that the implementation details and technical effects of each unit in the model training device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0198] This disclosure also provides an embodiment of a task processing apparatus. The task processing apparatus, applied to a client, includes: a task processing unit configured to perform a target natural language processing task using a locally pre-trained language model, wherein the locally pre-trained language model is obtained through, for example... Figure 4 The local model is trained using the embodiments shown and their alternative implementations described in the example.

[0199] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing embodiments of the present disclosure as a client or server. Figure 7 The computer system 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0200] like Figure 7 As shown, the computer system 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing device 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0201] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows computer system 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 A computer system 700 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0202] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0203] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0204] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0205] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 3The embodiments shown and their alternative implementations illustrate global model training methods, and / or, as... Figure 4 The embodiments shown and their optional implementations illustrate local model training methods, and / or the task processing methods shown in the embodiments and their optional implementations of this disclosure.

[0206] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0207] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0208] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not necessarily limiting in certain circumstances; for example, a local model sending unit can also be described as "a unit that sends the current local pre-trained language model to the server".

[0209] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A global model training method, comprising: Perform the following global model update operations: Send the global pre-trained language model to each client; The system receives local pre-trained language models returned by each client after the global pre-trained language model has been trained locally. The local pre-trained language model is trained based on a local training sample set and a corresponding pseudo training sample set. The local training samples in the local training sample set are labeled, and the pseudo training sample set is generated based on local sample data and the pseudo labels corresponding to the local sample data. Determine the weight coefficients of each of the local pre-trained language models; The local pre-trained language models are weighted and aggregated according to their respective weight coefficients to obtain the updated global pre-trained language model. The receipt of the locally pre-trained language model returned by each client after local training of the globally pre-trained language model includes: Receive the loss perturbation gradient of each client in response to the local pre-trained language model returned after local training of the global pre-trained language model; and The determination of the weight coefficients of each of the local pre-trained language models includes: For each of the local pre-trained language models, the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model is determined as the perturbation gradient sum of the local pre-trained language model. For each of the local pre-trained language models, the ratio of the perturbation gradient of the local pre-trained language model to the sum of the perturbation gradients of all the local pre-trained language models is determined as the proportion of the perturbation gradient of the local pre-trained language model. For each local pre-trained language model, the weight coefficients of the local pre-trained language model are determined based on the perturbation gradient and proportion of the local pre-trained language model.

2. The method according to claim 1, wherein, The global model update operation also includes: Determine whether to end the global model update operation; In response to the determination that the process has not ended, the global model update operation continues.

3. The method according to claim 1, wherein, The step of determining the weight coefficients of the local pre-trained language model based on the perturbation gradient and proportion of the local pre-trained language model includes: The larger of the perturbation gradient and its proportion of the local pre-trained language model and the preset non-negative smaller weight coefficient is determined as the weight coefficient of the local pre-trained language model.

4. The method according to claim 2, wherein, The step of determining whether to end the global model update operation includes: Determine whether the number of times the global model update operation is performed is less than a first preset number; In response to the determination that the global model update operation is not terminated; If the condition is not met, the global model update operation is terminated.

5. A local model training method, comprising: In response to receiving a global pre-trained language model from the server, local sample data from the local training sample set are input into the global pre-trained language model to obtain pseudo-labels corresponding to the local sample data. The local training samples include local sample data and corresponding local labels, and the corresponding local labels are obtained by annotation. Using the local sample data and corresponding pseudo-labels in the local training sample set, generate pseudo-training samples in the pseudo-training sample set; The global pre-trained language model is determined as the current local pre-trained language model, and the local training sample set and the corresponding pseudo training sample set are divided into at least two batches of local training sample subsets and corresponding pseudo training sample subsets. For each batch of local training sample subset and corresponding pseudo training sample subset, the current local pre-trained language model is updated based on the local training sample subset and corresponding pseudo training sample subset; Send the current local pre-trained language model to the server. The step of updating the current local pre-trained language model based on the local training sample subset and the corresponding pseudo-training sample subset includes: According to the first loss function, calculate the first loss of the current local pre-trained language model for the pseudo training sample subset, and use the first loss to optimize the current local pre-trained language model to obtain an optimized local pre-trained language model. According to the second loss function, the second loss of the optimized local pre-trained language model for the local training sample subset is calculated, wherein the second loss function adds a corresponding loss perturbation to the loss of the local training samples, and the second loss is backpropagated in the optimized local pre-trained language model to obtain the loss perturbation gradient of each local training sample and corresponding pseudo-training sample in the local training sample subset. The third loss of the current local pre-trained language model for the local training sample subset and the corresponding pseudo training sample subset is calculated according to the third loss function, wherein the third loss function is determined based on the first loss function; The model parameters of the current local pre-trained language model are updated according to the third loss.

6. The method according to claim 5, wherein, The step of updating the current local pre-trained language model based on the local training sample subset and the corresponding pseudo-training sample subset further includes: For each pseudo-training sample in the subset of pseudo-training samples, the sample weight of the pseudo-training sample to the current local pre-trained language model is determined based on the loss perturbation gradient of the pseudo-training sample. Wherein, the third loss function is the sum of the fourth loss function and the first loss function weighted by the sample weights, the first loss function weighted by the sample weights is the loss function obtained by weighting the first loss function according to the sample weights of the corresponding pseudo-training samples, and the fourth loss function is the loss of the current local pre-trained language model for the local training samples in the local training sample subset.

7. The method according to claim 6, wherein, Sending the current locally pre-trained language model to the server includes: The loss perturbation gradients of the current local pre-trained language model and each pseudo-training sample in the pseudo-training sample set are sent to the server.

8. The method according to claim 6, wherein, For each pseudo-training sample in the subset of pseudo-training samples, determining the sample weight of the pseudo-training sample with respect to the current local pre-trained language model based on the loss perturbation gradient of the pseudo-training sample includes: For each pseudo-training sample in the subset of pseudo-training samples, the larger of the preset non-negative smaller sample weight and the negative of the loss perturbation gradient of the pseudo-training sample is determined as the sample weight of the pseudo-training sample to the current pre-trained language model.

9. The method according to claim 6, wherein, The step of determining the sample weight of each pseudo-training sample in the subset of pseudo-training samples, based on the loss perturbation gradient of the pseudo-training sample, with respect to the current local pre-trained language model, further includes: The sample weights of each pseudo-training sample in the subset of pseudo-training samples are normalized for the current pre-trained language model.

10. A task processing method, comprising: The target natural language processing task is performed using a local pre-trained language model, wherein the local pre-trained language model is trained using the local model training method as described in any one of claims 5-9.

11. A global model training device, applied to a server, comprising: The global model update unit is configured to perform the following global model update operation: send the global pre-trained language model to each client; The system receives local pre-trained language models returned by each client after local training of the global pre-trained language model. These local pre-trained language models are trained based on a local training sample set and a corresponding pseudo-training sample set. The local training samples in the local training sample set are labeled, and the pseudo-training sample set is generated based on local sample data and its corresponding pseudo-labels. The system then determines the weight coefficients of each local pre-trained language model and performs weighted aggregation on each local pre-trained language model according to its corresponding weight coefficients to obtain the updated global pre-trained language model. The step of receiving the local pre-trained language model returned by each client after the global pre-trained language model has been trained locally includes: Receive the loss perturbation gradient of each client in response to the local pre-trained language model returned after local training of the global pre-trained language model; and The determination of the weight coefficients of each of the local pre-trained language models includes: For each of the local pre-trained language models, the sum of the loss perturbation gradients of each local training sample corresponding to the local pre-trained language model is determined as the perturbation gradient sum of the local pre-trained language model. For each of the local pre-trained language models, the ratio of the perturbation gradient of the local pre-trained language model to the sum of the perturbation gradients of all the local pre-trained language models is determined as the proportion of the perturbation gradient of the local pre-trained language model. For each local pre-trained language model, the weight coefficients of the local pre-trained language model are determined based on the perturbation gradient and proportion of the local pre-trained language model.

12. A local model training device, applied to a client, comprising: The pseudo-label generation unit is configured to, in response to receiving a global pre-trained language model sent by the server, input local sample data from the local training sample set into the global pre-trained language model respectively to obtain pseudo-labels corresponding to the local sample data. The local training samples include local sample data and corresponding local labels, and the corresponding local labels are obtained by annotation. The pseudo-sample generation unit is configured to generate pseudo-training samples in the pseudo-training sample set using local sample data in the local training sample set and corresponding pseudo-labels. The sample batching unit is configured to determine the global pre-trained language model as the current local pre-trained language model, and to divide the local training sample set and the corresponding pseudo-training sample set into at least two batches of local training sample subsets and corresponding pseudo-training sample subsets. The local model update unit is configured to update the current local pre-trained language model based on each batch of local training sample subset and corresponding pseudo-training sample subset. The local model sending unit is configured to send the current locally pre-trained language model to the server. The local model update unit is further configured as follows: According to the first loss function, calculate the first loss of the current local pre-trained language model for the pseudo training sample subset, and use the first loss to optimize the current local pre-trained language model to obtain an optimized local pre-trained language model. According to the second loss function, the second loss of the optimized local pre-trained language model for the local training sample subset is calculated, wherein the second loss function adds a corresponding loss perturbation to the loss of the local training samples, and the second loss is backpropagated in the optimized local pre-trained language model to obtain the loss perturbation gradient of each local training sample and corresponding pseudo-training sample in the local training sample subset. The third loss of the current local pre-trained language model for the local training sample subset and the corresponding pseudo training sample subset is calculated according to the third loss function, wherein the third loss function is determined based on the first loss function; The model parameters of the current local pre-trained language model are updated according to the third loss.

13. A task processing apparatus, comprising: The task processing unit is configured to perform a target natural language processing task using a local pre-trained language model, wherein the local pre-trained language model is trained using the local model training method as described in any one of claims 5-9.

14. A server, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-4.

15. A client, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 5-9.

16. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1-4 and / or the method as described in any one of claims 5-9.

17. A model training system, comprising a server as described in claim 14 and a client as described in claim 15.

Images