Communication and memory efficient large model distributed training method, text classification method

Abstract

The embodiment of the present application provides a large-model distributed training method and a text classification method with high communication and memory efficiency, which belongs to the field of artificial intelligence technology. The method includes: performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter to obtain a first reference classification model and a second reference classification model, performing text classification on the language text by the first reference classification model to obtain a first text category, performing text classification on the language text by the second reference classification model to obtain a second text category, calculating the loss difference according to the text category label, the first text category and the second text category, and obtaining the original symbol of the loss difference, sending the original symbol to the server, and receiving the target symbol obtained by the server by aggregating the original symbol, updating the original local classification model according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model, which can improve the efficiency of model training.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large-model distributed training method and a text classification method with high communication and memory efficiency. Background Art

[0002] In related technologies, in order to obtain a deep learning model that performs well on a specific task, such as a text classification task, a model pre-trained on a large dataset is usually downloaded from a server and used as a training starting point for training on a local task-specific dataset on the terminal device. During the training process, the terminal device needs to execute the gradient descent algorithm one or more times, which places high demands on the terminal device's memory and computing power. In addition, the terminal device and the server often need to go through thousands of communication rounds, and the terminal device will send a large number of model parameters to the server in each communication round, which makes the communication overhead of model training relatively high. These factors seriously affect the training of the local model of the terminal device, making the model training efficiency low. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a large-model distributed training method and text classification method with efficient communication and memory, aiming to improve the efficiency of model training.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a communication and memory efficient large model distributed training method, applied to a terminal device, the method comprising:

[0005] Obtaining a language text and a text category label of the language text;

[0006] Receive a random seed sent by the server, and generate a model perturbation vector according to the random seed;

[0007] Obtaining perturbation hyperparameters, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameters, to obtain a first reference classification model and a second reference classification model;

[0008] Performing text classification on the language text using the first reference classification model to obtain a first text category;

[0009] Performing text classification on the language text using the second reference classification model to obtain a second text category;

[0010] Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference;

[0011] Sending the original symbol to the server, and receiving a target symbol obtained by aggregating the original symbol by the server;

[0012] The original local classification model is updated according to the target symbol, the model perturbation vector and a preset learning rate to obtain a target classification model.

[0013] In some embodiments, calculating the loss difference based on the text category label, the first text category, and the second text category includes:

[0014] Calculating a loss based on the text category label and the first text category to obtain a first loss;

[0015] Calculating a loss based on the text category label and the second text category to obtain a second loss;

[0016] A difference between the first loss and the second loss is calculated to obtain the loss difference.

[0017] In some embodiments, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter to obtain a first reference classification model and a second reference classification model includes:

[0018] Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector;

[0019] Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model;

[0020] A subtraction calculation is performed on the model parameters of the original local classification model and the reference disturbance vector to obtain the second reference classification model.

[0021] In some embodiments, after updating the original local classification model according to the target symbol, the model perturbation vector, and the preset learning rate to obtain the target classification model, the communication and memory efficient large model distributed training method further includes:

[0022] Get the test text;

[0023] Performing model evaluation on the target classification model according to the test text to obtain model generalization data;

[0024] Acquire the number of texts of the language text according to the model generalization data;

[0025] The target classification model is updated according to the amount of text.

[0026] In some embodiments, updating the target classification model according to the amount of text includes:

[0027] If the amount of text is less than a preset threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label, and the sample category label;

[0028] If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

[0029] In some embodiments, updating the target classification model according to the category proportions includes:

[0030] If the category proportion is less than a preset proportion threshold, receiving the original global model sent by the server, performing text classification on the language text using the target classification model to obtain a local category label, performing text classification on the language text using the original global model to obtain a global category label, and calculating a third loss based on the local category label, the global category label, and the text category label;

[0031] Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model;

[0032] If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a text classification method, which is applied to a terminal device. The method includes:

[0034] Get the target text;

[0035] The target text is classified by a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method described in the first aspect.

[0036] To achieve the above-mentioned purpose, a third aspect of the embodiments of the present application provides a text classification device, which is applied to a terminal device. The device includes:

[0037] Acquisition module, used to obtain target text;

[0038] A classification module is used to perform text classification on the target text through a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method described in the first aspect.

[0039] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the communication and memory-efficient large-model distributed training method described in the first aspect or the text classification method described in the second aspect.

[0040] To achieve the above-mentioned objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the communication and memory-efficient large-model distributed training method described in the first aspect or the text classification method described in the second aspect.

[0041] The communication and memory efficient large model distributed training method, text classification method, text classification device, electronic device, and computer-readable storage medium of the embodiment of the present application complete the training of the local model of the terminal device by obtaining the language text and the text category label of the language text, and using the language text and the text category label as training samples. In order to enable the model to better adapt to unknown samples and improve the generalization ability and tolerance of the model to noise, a random seed sent by the server is received, and a model perturbation vector is generated based on the random seed. In order to reduce communication overhead, the server does not send the model pre-trained on a large data set to the terminal device. In order to train the original local classification model and determine the impact of the model perturbation on the original local classification model, perturbation hyperparameters are obtained, and the original local classification model is perturbed according to the model perturbation vector and the perturbation hyperparameters to obtain a first reference classification model and a second reference classification model. For model training, the first reference classification model and the second reference classification model are used to perform task reasoning respectively. The language text is classified by the first reference classification model to obtain a first text category, and the language text is classified by the second reference classification model to obtain a second text category. The loss difference is calculated based on the text category label, the first text category, and the second text category to determine the difference between the two inferences of the model. It consumes more communication resources for the terminal device to send model parameters to the server. In order to reduce communication overhead, the original symbol of the loss difference is obtained and the original symbol is sent to the server, so that the communication overhead of the terminal device is only one bit. In order to optimize the local model of the terminal device, the receiving server aggregates the original symbols to obtain the target symbol. By aggregating the original symbols, the model performance is avoided due to the large heterogeneity of local data sets of different terminal devices. The original local classification model is updated according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model, so that the terminal device can update the model without executing the gradient descent algorithm, reducing the memory and computing power requirements of the terminal device, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a communication and memory efficient large model distributed training method provided by an embodiment of the present application;

[0043] Figure 2 yes Figure 1 Flowchart of step S130 in FIG.

[0044] Figure 3 yes Figure 1 Flowchart of step S160 in FIG.

[0045] Figure 4 is another flow chart of the communication and memory efficient large model distributed training method provided by an embodiment of the present application;

[0046] Figure 5 yes Figure 4 Flowchart of step S440 in FIG.

[0047] Figure 6 yes Figure 5 Flowchart of step S520 in FIG.

[0048] Figure 7 is a flowchart of the text classification method provided in an embodiment of the present application;

[0049] Figure 8 is a structural diagram of a text classification device provided in an embodiment of the present application;

[0050] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] In related technologies, in order to obtain a deep learning model that performs well on a specific task, such as a text classification task, a model pre-trained on a large dataset is usually downloaded from a server and used as a training starting point for training on a local task-specific dataset on the terminal device. During the training process, the terminal device needs to execute the gradient descent algorithm one or more times, which places high demands on the terminal device's memory and computing power. In addition, the terminal device and the server often need to go through thousands of communication rounds, and the terminal device will send a large number of model parameters to the server in each communication round, which makes the communication overhead of model training relatively high. These factors seriously affect the training of the local model of the terminal device, making the model training efficiency low.

[0055] Based on this, the embodiments of the present application provide a large-model distributed training method, text classification method, text classification device, electronic device, and computer-readable storage medium with efficient communication and memory, aiming to improve the efficiency of model training.

[0056] The communication and memory-efficient large-model distributed training method, text classification method, text classification device, electronic device, and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the communication and memory-efficient large-model distributed training method in the embodiments of the present application is described.

[0057] The communication and memory efficient large model distributed training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The communication and memory efficient large model distributed training method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the communication and memory efficient large model distributed training method, etc., but is not limited to the above forms.

[0058] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0059] Figure 1 This is an optional flowchart of the communication and memory efficient large model distributed training method provided in an embodiment of the present application, which is applied to a terminal device and may include but is not limited to steps S110 to S180.

[0060] Step S110, obtaining a language text and a text category label of the language text;

[0061] Step S120: receiving a random seed sent by the server, and generating a model perturbation vector according to the random seed;

[0062] Step S130, obtaining perturbation hyperparameters, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameters, to obtain a first reference classification model and a second reference classification model;

[0063] Step S140, classifying the language text using a first reference classification model to obtain a first text category;

[0064] Step S150, performing text classification on the language text using a second reference classification model to obtain a second text category;

[0065] Step S160, calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original symbol of the loss difference;

[0066] Step S170, sending the original symbols to the server, and receiving the target symbols obtained by aggregating the original symbols from the server;

[0067] Step S180, updating the original local classification model according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model.

[0068] In step S110 of some embodiments, based on the text classification task, the terminal device collects text data from different data sources such as social media, databases, etc. to obtain language text, and classifies the language text to obtain text category labels, and trains the local model of the terminal device based on the language text and the text category labels of the language text. The language text can be text materials such as product reviews, social media posts, articles, etc., and the text category labels can be set based on the text classification task. If the text classification task is emotion recognition, the text category labels are emotion categories, such as positive, negative, or neutral. If the text classification task is intent recognition, the text category labels are intent categories, such as booking tickets, checking the weather, etc. If the text classification task is topic classification, the text category labels are topic categories, such as technology, sports, etc.

[0069] In step S120 of some embodiments, the server generates a random integer and sends it to the terminal device as a random seed. If the random seed is generated by the terminal device, there may be a risk of the random seed being predicted or maliciously manipulated. Furthermore, the terminal device has limited computing resources, and generating a high-quality random seed requires relatively high computing power. Furthermore, if the random seed is generated by the terminal device, directly using the sign of the difference in loss values for majority voting can cause model training to fail to converge due to excessive randomness. To improve the security of model training, enable model training to converge, and reduce the computational burden on the terminal device, embodiments of the present application generate a random seed by the server, and estimate the model gradient of the original local classification model based on the random seed for model training. Furthermore, by sending the random seed instead of the perturbation vector, the consumption of server communication resources is reduced. The terminal device calculates the random seed using the random number generation algorithm built into the pseudo-random number generator to obtain the model perturbation vector. The random number generation algorithm can be a Philox algorithm, a linear feedback shift register (LFSR) algorithm, a linear congruential generator (LCG) algorithm, or the like. The model perturbation vector is used to perturb the original local classification model to increase diversity during model training and enable the model to better generalize to different data distributions.

[0070] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S230:

[0071] Step S210, multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector;

[0072] Step S220, adding the model parameters of the original local classification model and the reference disturbance vector to obtain a first reference classification model;

[0073] Step S230 , performing a subtraction calculation on the model parameters of the original local classification model and the reference disturbance vector to obtain a second reference classification model.

[0074] In step S210 of some embodiments, a perturbation hyperparameter is obtained. The perturbation hyperparameter is used to control the degree of perturbation of the model by the model perturbation vector. The perturbation hyperparameter is a number greater than 0. If the model perturbation vector is z and the perturbation hyperparameter is e, then the reference perturbation vector is e*z, where * represents a multiplication operation.

[0075] In step S220 of some embodiments, in the related art, during the model training process under the federated learning framework, the terminal device often needs to send the model gradient to the server. When the number of model parameters is large, the uplink communication overhead of the terminal device to the server will be large. In order to reduce the communication overhead, the embodiment of the present application decomposes the model gradient into a model perturbation vector and a corresponding gradient projection. In order to make the model training process converge as quickly as possible and improve the training efficiency of the model, it is necessary to calculate the gradient projection. In order to calculate the gradient projection, the reference perturbation vector is used to perform forward perturbation and backward perturbation on the original local classification model respectively. The model parameters of the original local classification model and the reference perturbation vector are added to perform forward perturbation on the original local classification model to obtain a first reference classification model. The first reference classification model is expressed as w+e*z.

[0076] The original local classification model is a text classification model located locally on the terminal device. It can be a single model or a composite model integrated with multiple models. The original local classification model can be selected based on actual needs, such as the BERT model or the Transformer model. To obtain a deep model that performs well on local language text on the terminal device, you can download a pre-trained large language model and use it as the original local classification model to fine-tune the original local classification model. Text can be input into the original local classification model for text classification, resulting in text category labels.

[0077] For the positive and negative sentiment classification problem, the input language text is "<text to be classified>It was" (this is...), and the response template of the large language model is "Terrible." (bad) or "Great." (good) to determine the text category label.

[0078] For intent recognition, the input language text is “<text to be classified>In the previous sentense, does the pronoun <span2>refer to <span1>? Yes or No? (In the previous sentence, the pronoun <span2>Whether the reference is an entity <span1>? ), the response template of the large language model is "Yes." or "No." to determine the text category label.

[0079] For topic classification, the input language text is "<Text to be classified>Question: this belongs to which subject? I found this answer<Category>. Is that correct? Yes or No?" The large language model's response template is Yes or No to determine the text category label.

[0080] In step S230 of some embodiments, the model parameters of the original local classification model are subtracted from the reference perturbation vector to perform backward perturbation on the original local classification model to obtain a second reference classification model. The second reference classification model is represented as we*z.

[0081] In the above steps S210 to S230 , random perturbations are added to the original local classification model to estimate the gradient projection, thereby improving the efficiency of model training.

[0082] In step S140 of some embodiments, in order to measure the model classification performance of the first reference classification model, the first reference classification model is used to perform text classification on the language text to obtain a first text category. The first text category is the text category predicted by the first reference classification model for the language text.

[0083] In step S150 of some embodiments, in order to measure the model classification performance of the second reference classification model, the second reference classification model is used to perform text classification on the language text to obtain a second text category. The second text category is the text category predicted by the second reference classification model for the language text.

[0084] See also Figure 3 In some embodiments, step S160 may include but is not limited to steps S310 to S330:

[0085] Step S310, calculating a loss based on the text category label and the first text category to obtain a first loss;

[0086] Step S320, calculating a loss based on the text category label and the second text category to obtain a second loss;

[0087] Step S330: Calculate the difference between the first loss and the second loss to obtain a loss difference.

[0088] In step S310 of some embodiments, based on the cross-entropy loss function, loss calculation is performed according to the text category label and the first text category to measure the difference between the predicted text category output by the first reference classification model and the true text category to obtain a first loss.

[0089] In step S320 of some embodiments, based on the cross entropy loss function, loss calculation is performed according to the text category label and the second text category to measure the difference between the predicted text category output by the second reference classification model and the true text category to obtain a second loss.

[0090] In step S330 of some embodiments, the first loss is subtracted from the second loss to obtain a difference between the first loss and the second loss, thereby obtaining a loss difference. The loss difference is used to measure the difference in text classification performance between the first reference classification model and the second reference classification model.

[0091] Through the above steps S310 to S330, the loss difference value of the model's two text inferences can be obtained to guide the training of the local model of the terminal device based on the loss difference value.

[0092] In step S170 of some embodiments, under the federated learning framework, multiple terminal devices do not expose local private data, and under the coordination of the server, each terminal device jointly trains a common deep learning model. During the model training process, the terminal device needs to go through thousands of communication rounds with the server to converge the local model, so that the terminal device needs to send the model parameters to the server thousands of times, which usually consumes a lot of communication resources. In order to save communication resources, the original sign of the gradient projection is obtained. The gradient projection is the projection of the model gradient on the model perturbation vector. The gradient projection is used to indicate the magnitude of the model parameter update, and the original sign is used to indicate the direction of the model parameter update. The loss difference is divided by the perturbation hyperparameter to determine the proportional relationship between the two and obtain the gradient projection. The perturbation hyperparameter is used to control the size of the gradient projection, thereby affecting the efficiency of the model update to accelerate model convergence. Since the perturbation hyperparameter is a positive number, the original sign of the gradient projection is the original sign of the loss difference.

[0093] The original symbol of the loss difference is obtained, and the original symbol is sent to the server instead of the model parameter. The original symbol is used to indicate that the loss difference is greater than or equal to 0 or less than 0, so that the terminal device only needs to consume one bit of communication resources in each communication round. The server screens the original symbols sent by each terminal device, selects the original symbol with the highest frequency of occurrence as the target symbol, and sends the target symbol to each terminal device respectively. For example, the original symbol is set to include + or -, + indicates that the loss difference is greater than or equal to 0, and - indicates that the loss difference is less than 0. The original symbols received by the server for the three terminal devices participating in this update are s1, s2, and s3, respectively. The original symbols of s1 and s2 are both +, and the original symbol of s3 is -. The frequency of occurrence of + is greater than the frequency of occurrence of -, and the target symbol is +.

[0094] In step S180 of some embodiments, the terminal device needs to execute one or more gradient descent algorithms to update the local model, which places high demands on the memory and computing power of the terminal device. In order to reduce the memory and computing power requirements of the terminal device for model training and improve the efficiency of model training, the embodiment of the present application multiplies the preset learning rate, the target symbol and the model perturbation vector, and subtracts the model parameters of the original local classification model from the calculation result of the multiplication operation without executing the gradient descent algorithm to obtain the target classification model. The preset learning rate is the learning rate used for the text classification task. It is a hyperparameter and is usually set to a small positive number. For example, the preset learning rate can be in the range of [1e-7, 1e-6]. The target symbol is used to indicate the direction of the model parameter update, and the model perturbation vector is used to indicate the magnitude of the model parameter update. If the target symbol is s, the model perturbation vector is z, the preset learning rate is a, and the model parameters of the original local classification model are w, then the model parameters of the target classification model are expressed as w-asz.

[0095] In a federated learning framework, each terminal device sends model parameters or gradient parameters to a server. The server aggregates the model parameters or gradient parameters of each terminal device and determines the aggregated parameters. Each terminal device then trains its local model based on the aggregated parameters sent by the server. However, this model training approach presents two issues, which can lead to performance degradation or even failure of the final fine-tuned model. First, if a malicious terminal device sends incorrect model parameters or gradient parameters to the server, the aggregated parameters will be distorted, causing the local models constructed by each terminal device based on the aggregated parameters to be vulnerable to Byzantine attacks. Second, if the heterogeneity between the local datasets of each terminal device is too great, the local model's personalization capabilities will be insufficient. Related technologies use gradient information uploaded by the terminal device to modify the aggregated parameters. However, due to concerns about data privacy and security, as well as local computing costs and server communication costs, each terminal device does not exchange its local gradient information, making it impossible for the server to determine the authenticity and accuracy of each gradient information. In this embodiment, the terminal device uploads the symbols of the gradient projection and trains the local model based on the aggregated symbols, making the local model resistant to Byzantine attacks and data heterogeneity.

[0096] See also Figure 4 In some embodiments, after step S180, the communication and memory efficient large model distributed training method may further include but is not limited to steps S410 to S440:

[0097] Step S410, obtaining a test text;

[0098] Step S420, performing model evaluation on the target classification model based on the test text to obtain model generalization data;

[0099] Step S430, obtaining the text quantity of the language text according to the model generalization data;

[0100] Step S440: Update the target classification model according to the amount of text.

[0101] In step S410 of some embodiments, the terminal device obtains test text. The test text is text data used to test the model performance of the terminal device's local target classification model. The test text has a test category label that indicates the text category of the test text. The test text and the language text are texts for the same text classification task.

[0102] In step S420 of some embodiments, a test text is input into a target classification model for text classification, obtaining a test text category. The test text category and the test category label are compared, and a first number of test text categories and test category labels that are identical are recorded. The number of test texts is obtained to obtain a second number. The ratio of the first number to the second number is used as model generalization data.

[0103] In step S430 of some embodiments, if the model generalization data is greater than or equal to a preset generalization threshold, indicating that the target classification model can accurately classify the test text locally on the terminal device and that the target classification model has good personalized performance, the target classification model is not updated. If the model generalization data is less than the preset generalization threshold, indicating that the generalization performance of the target classification model is insufficient and that the target classification model needs to be updated, the number of language texts used to train the target classification model is obtained to obtain the number of texts.

[0104] In step S440 of some embodiments, an update method of the target classification model is determined according to the amount of text, and the target classification model is updated according to the update method.

[0105] In the above steps S410 to S440, by evaluating the generalization of the target classification model, the target classification model is updated when the generalization is insufficient, so that the updated target classification model has better text classification performance on local text data, thereby improving the individual performance of the model.

[0106] See also Figure 5 In some embodiments, step S440 may include but is not limited to step S510 or step S520:

[0107] Step S510: If the number of texts is less than a preset threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained. The target classification model is updated based on the language text, the sample text, the text category label, and the sample category label.

[0108] Step S520: If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language texts is obtained, the category proportions are determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportions.

[0109] In step S510 of some embodiments, if the number of texts is less than a preset threshold, indicating that the number of samples used to train the target classification model is insufficient, resulting in low model generalization, the language text is expanded by synonym replacement, random insertion, random deletion, back translation, text generation, etc. to obtain sample text. The sample text is categorized to obtain a sample category label, or the text category label of the language text corresponding to the sample text is used as the sample category label. A new data set is constructed based on the language text, sample text, text category label, and sample category label, and the target classification model is updated using the new data set with reference to steps S110 to S180.

[0110] It should be noted that synonym replacement refers to randomly selecting words from a text and replacing them with synonyms. Random insertion refers to inserting words or phrases at random locations in a text. Random deletion refers to randomly deleting words from a text. Back translation refers to translating a text from a source language to a target language and then back from the target language to the source language. Text generation refers to generating new text data using a language model.

[0111] In step S520 of some embodiments, if the number of texts is greater than or equal to a preset number threshold, indicating that the number of samples is sufficient, the number of language texts under each text category is obtained to obtain the number of categories. The number of each category is divided by the number of texts to obtain the category proportion of each category. The update method of the target classification model is determined according to the category proportion, and the target classification model is updated according to the update method. For example, the target classification model can classify three text categories, which are c1, c2 and c3 respectively. The number of categories with text category c1 is s1, the number of categories with text category c2 is s2, and the number of categories with text category c3 is 0. Then the category proportion of text category c1 is s1 / (s1+s2), the category proportion of text category c2 is s2 / (s1+s2), and the category proportion of text category c2 is 0.

[0112] Through the above steps S510 to S520, the update method of the target classification model is differentiated according to different situations, so that the model can be updated accurately and efficiently according to the specific update method, thereby improving the text classification accuracy of the model on local text data.

[0113] See also Figure 6 In some embodiments, step S520 may include but is not limited to steps S610 to S630:

[0114] Step S610: If the category proportion is less than the preset proportion threshold, the original global model sent by the server is received, and the language text is classified using the target classification model to obtain a local category label. The language text is also classified using the original global model to obtain a global category label. A third loss is calculated based on the local category label, the global category label, and the text category label.

[0115] Step S620: Adjust the model parameters of the target classification model according to the third loss to obtain a candidate classification model, send the candidate classification model to the server, receive an aggregated classification model obtained by aggregating the candidate classification models, and update the candidate classification model according to the aggregated classification model.

[0116] Step S630: If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0117] In some embodiments, in step S610, if the category share of at least one category is less than a preset share threshold, this indicates that the model generalization is low due to an imbalance in the number of samples across categories. To improve the model generalization, sample text from other terminal devices is needed. Specifically, the server initializes the model, obtains an original global model, and broadcasts the original global model to each terminal device. The terminal device receives the original global model sent by the server, inputs the language text into the target classification model for text classification, obtains a local category label, and then inputs the language text into the original global model for text classification, obtains a global category label. Based on the cross-entropy loss function, a loss is calculated based on the local category label and the text category label to obtain a first sub-loss, which is used to measure the model classification performance of the target classification model. Based on the cross-entropy loss function, a loss is calculated based on the global category label and the text category label to obtain a second sub-loss, which is used to measure the model classification performance of the original global model. Based on the cross-entropy loss function, a loss is calculated based on the global category label and the local category label to obtain a third sub-loss, which is used to measure the classification performance difference between the original global model and the target classification model, so that the target classification model can learn the text classification knowledge of the original global model. The first sub-loss, the second sub-loss, and the third sub-loss are summed to obtain the third loss.

[0118] In step S620 of some embodiments, the third loss is minimized, the model parameters of the target classification model are adjusted, and a candidate classification model is obtained. The model parameters of the candidate classification model are sent to the server, and the server calculates the mean of the model parameters of each terminal device to obtain an aggregated classification model. The original global model is updated according to the aggregated classification model, and the updated original global model is sent to the terminal device, and the target classification model is updated according to step S610. The original global model can capture the data distribution characteristics of different terminal devices, reduce the model performance degradation caused by data distribution differences, and use the original global model and the local language text of the terminal device to update the target classification model, so that the target classification model can not only learn the global classification knowledge captured by the original global model, but also learn the local classification knowledge implicit in the local text data, thereby improving the generalization of the target classification model.

[0119] In step S630 of some embodiments, if the category proportion of each category is greater than or equal to the preset proportion threshold, it means that the local text data distribution of the terminal device is relatively balanced, and then one or more network layers are randomly selected from the network layer of the target classification model as the target network layer, and a low-rank adaptation layer is added after the target network layer. The language text is input into the target classification model with the low-rank adaptation layer added for text classification to obtain a predicted text category. Based on the cross-entropy loss function, the loss of the predicted text category and the text category label is calculated to obtain a fourth loss. Minimize the fourth loss, fine-tune the network parameters of the low-rank adaptation layer, and keep the network parameters of other network layers unchanged. By fine-tuning the model structure of the target classification model, the generalization of the target classification model for text classification tasks can be improved.

[0120] Through the above steps S610 to S630, the generalization of the target classification model can be improved, so that the target classification model can accurately classify the local text data of the terminal device, thereby improving the accuracy of text classification.

[0121] Each terminal device stores a local private model and a local private data set. Each private text in the local private data set has a text category label and shares a pseudo-random number generator. The communication and memory efficient large model distributed training method of the embodiment of the present application includes: in the current communication round, each terminal device receives a random seed sent by the server, inputs the random seed into the pseudo-random number generator to generate a model perturbation vector, and sets the perturbation hyperparameters. The local private model is forward perturbed according to the perturbation hyperparameters and the model perturbation vector to obtain a first perturbation model. If the perturbation hyperparameter is e, the model perturbation vector is z, and the model parameter of the local private model is w, then the first perturbation model is expressed as w+ez. The local private model is backward perturbed according to the perturbation hyperparameters and the model perturbation vector to obtain a second perturbation model, and the second perturbation model is expressed as w-ez. Each terminal device selects a data batch in the local private data set, performs text reasoning on the data batch through the first perturbation model, and obtains a first text category. The data batch is text reasoning through the second perturbation model to obtain a second text category. Based on the cross-entropy loss function, the loss is calculated based on the text category label and the first text category to obtain the first loss. Based on the cross-entropy loss function, the loss is calculated based on the text category label and the second text category to obtain the second loss. The first loss is subtracted from the second loss to obtain the loss difference, and the original symbol of the loss difference is obtained. Each terminal device sends its own original symbol to the server. The server selects the majority of the original symbols as the aggregate symbol s and forwards the aggregate symbol s to all terminal devices. Each terminal device updates the model parameter w of the local private model based on the aggregate symbol s, the model perturbation vector z, and the preset learning rate, that is, w = w-asz, and enters the next communication round until the communication round reaches the preset round.

[0122] See also Figure 7 The present application also provides a text classification method, which is applied to a terminal device. The text classification method may include but is not limited to steps S710 to S720:

[0123] Step S710, obtaining target text;

[0124] Step S720: classify the target text using the target classification model to obtain the target text category.

[0125] In step S710 of some embodiments, the terminal device obtains target text, which is text data to be classified.

[0126] In step S720 of some embodiments, in order to obtain the text category of the target text, the target text is input into a target classification model for text classification to obtain the target text category.

[0127] In the above steps S710 to S720, the model training efficiency of the target classification model is relatively high, and when the target classification model is used for text classification, the efficiency of text classification is greatly improved.

[0128] See also Figure 8 The present application also provides a text classification device, which is applied to a terminal device and can implement the above-mentioned text classification method. The text classification device includes:

[0129] An acquisition module 810 is used to acquire a target text;

[0130] The classification module 820 is used to perform text classification on the target text using a target classification model to obtain a target text category.

[0131] In some embodiments, the text classification device is further configured to:

[0132] Obtaining a language text and a text category label of the language text;

[0133] Receive a random seed sent by the server, and generate a model perturbation vector according to the random seed;

[0134] Obtaining perturbation hyperparameters, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameters, to obtain a first reference classification model and a second reference classification model;

[0135] Performing text classification on the language text using the first reference classification model to obtain a first text category;

[0136] Performing text classification on the language text using the second reference classification model to obtain a second text category;

[0137] Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference;

[0138] Sending the original symbol to the server, and receiving a target symbol obtained by aggregating the original symbol by the server;

[0139] The original local classification model is updated according to the target symbol, the model perturbation vector and a preset learning rate to obtain a target classification model.

[0140] In some embodiments, the text classification device is further configured to:

[0141] Calculating a loss based on the text category label and the first text category to obtain a first loss;

[0142] Calculating a loss based on the text category label and the second text category to obtain a second loss;

[0143] A difference between the first loss and the second loss is calculated to obtain the loss difference.

[0144] In some embodiments, the text classification device is further configured to:

[0145] Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector;

[0146] Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model;

[0147] A subtraction calculation is performed on the model parameters of the original local classification model and the reference disturbance vector to obtain the second reference classification model.

[0148] In some embodiments, the text classification device is further configured to:

[0149] Get the test text;

[0150] Performing model evaluation on the target classification model according to the test text to obtain model generalization data;

[0151] Acquire the number of texts of the language text according to the model generalization data;

[0152] The target classification model is updated according to the amount of text.

[0153] In some embodiments, the text classification device is further configured to:

[0154] If the amount of text is less than a preset threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label, and the sample category label;

[0155] If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

[0156] In some embodiments, the text classification device is further configured to:

[0157] If the category proportion is less than a preset proportion threshold, receiving the original global model sent by the server, performing text classification on the language text using the target classification model to obtain a local category label, performing text classification on the language text using the original global model to obtain a global category label, and calculating a third loss based on the local category label, the global category label, and the text category label;

[0158] Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model;

[0159] If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0160] The specific implementation of the text classification device is basically the same as the specific embodiment of the above-mentioned text classification method, and will not be repeated here.

[0161] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the aforementioned communication- and memory-efficient large-model distributed training method or text classification method. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0162] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0163] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0164] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the communication and memory efficient large model distributed training method or text classification method of the embodiments of this application;

[0165] Input / output interface 930, used to implement information input and output;

[0166] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0167] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0168] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0169] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned communication and memory-efficient large-model distributed training method or text classification method.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0172] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in this application, "at least one (item)" means one or more, and "more" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A communication and memory efficient distributed training method for large models, characterized by: Applied to a terminal device, the method includes: Obtaining a language text and a text category label of the language text; Receive a random seed sent by the server, and generate a model perturbation vector according to the random seed; Obtaining perturbation hyperparameters, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameters, to obtain a first reference classification model and a second reference classification model; Performing text classification on the language text using the first reference classification model to obtain a first text category; Performing text classification on the language text using the second reference classification model to obtain a second text category; Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference; the original sign is used to indicate that the loss difference is greater than or equal to 0 or less than 0; Sending the original symbols to the server, and receiving a target symbol obtained by aggregating the original symbols by the server; the target symbol is the original symbol with the highest frequency of occurrence screened by the server from the original symbols sent by each terminal device; A multiplication operation is performed on the target symbol, the model perturbation vector and a preset learning rate, and a calculation result of the multiplication operation is subtracted from the model parameters of the original local classification model to obtain a target classification model.

2. The method according to claim 1, characterized in that The calculating the loss difference according to the text category label, the first text category, and the second text category includes: Calculating a loss based on the text category label and the first text category to obtain a first loss; Calculating a loss based on the text category label and the second text category to obtain a second loss; A difference between the first loss and the second loss is calculated to obtain the loss difference.

3. The method according to claim 1, characterized in that The performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter to obtain a first reference classification model and a second reference classification model includes: Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector; Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model; A subtraction calculation is performed on the model parameters of the original local classification model and the reference disturbance vector to obtain the second reference classification model.

4. The method according to any one of claims 1 to 3, characterized in that After the original local classification model is updated according to the target symbol, the model perturbation vector, and the preset learning rate to obtain the target classification model, the communication and memory efficient large model distributed training method further includes: Get the test text; Performing model evaluation on the target classification model according to the test text to obtain model generalization data; Acquire the number of texts of the language text according to the model generalization data; The target classification model is updated according to the amount of text.

5. The method according to claim 4, characterized in that The updating of the target classification model according to the amount of text includes: If the amount of text is less than a preset threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label, and the sample category label; If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

6. The method according to claim 5, characterized in that The updating of the target classification model according to the category proportions includes: If the category proportion is less than a preset proportion threshold, receiving the original global model sent by the server, performing text classification on the language text using the target classification model to obtain a local category label, performing text classification on the language text using the original global model to obtain a global category label, and calculating a third loss based on the local category label, the global category label, and the text category label; Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model; If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

7. A text classification method, characterized in that: Applied to a terminal device, the method includes: Get the target text; The target text is classified by a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method according to any one of claims 1 to 6.

8. A text classification device, characterized in that: Applied to a terminal device, the device includes: Acquisition module, used to obtain target text; A classification module is used to perform text classification on the target text through a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the communication and memory efficient large model distributed training method described in any one of claims 1 to 6 or the text classification method described in claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the communication and memory efficient large model distributed training method according to any one of claims 1 to 6 or the text classification method according to claim 7.

Images