Communication and memory efficient large model distributed training method and text classification method

Abstract

The embodiment of the invention provides a communication and memory efficient large model distributed training method and a text classification method, and belongs to the technical field of artificial intelligence. The method comprises the steps of performing model disturbance on an original local classification model according to a model disturbance vector and a disturbance hyper-parameter to obtain a first reference classification model and a second reference classification model, performing text classification on language texts through the first reference classification model to obtain a first text category, and performing text classification on the language texts through the second reference classification model; performing text classification on the language text through a second reference classification model to obtain a second text category, calculating a loss difference value according to the text category label, the first text category and the second text category, obtaining an original symbol of the loss difference value, and sending the original symbol to a server, and receiving the target symbol obtained by aggregating the original symbols by the server, and updating the original local classification model according to the target symbol, the model disturbance vector and the preset learning rate to obtain the target classification model, so that the model training efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large-model distributed training method and a text classification method with high communication and memory efficiency. Background Art

[0002] In related technologies, in order to obtain a deep learning model that performs well on specific tasks such as text classification tasks, a model pre-trained on a large data set is usually downloaded from a server, and this is used as a training starting point to train on a data set of a specific task on a local terminal device. During the training process, the terminal device needs to execute the gradient descent algorithm once or multiple times, which places high demands on the memory and computing power of the terminal device. In addition, the terminal device and the server often need to go through thousands of communication rounds, and the terminal device will send a large number of model parameters to the server in each communication round, which makes the communication overhead of model training large. These factors seriously affect the training of the local model of the terminal device, making the efficiency of model training low. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a large-model distributed training method and text classification method with efficient communication and memory, aiming to improve the efficiency of model training.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a large model distributed training method with high communication and memory efficiency, which is applied to a terminal device. The method includes: Obtaining a language text and a text category label of the language text; Receiving a random seed sent by a server, and generating a model perturbation vector according to the random seed; Obtaining a perturbation hyperparameter, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter, to obtain a first reference classification model and a second reference classification model; Performing text classification on the language text by using the first reference classification model to obtain a first text category; Performing text classification on the language text by using the second reference classification model to obtain a second text category; Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference; Sending the original symbol to the server, and receiving a target symbol obtained by aggregating the original symbol by the server; The original local classification model is updated according to the target symbol, the model perturbation vector and a preset learning rate to obtain a target classification model.

[0005] In some embodiments, the calculating the loss difference according to the text category label, the first text category, and the second text category includes: Calculate the loss according to the text category label and the first text category to obtain a first loss; Calculate the loss according to the text category label and the second text category to obtain a second loss; The difference between the first loss and the second loss is calculated to obtain the loss difference.

[0006] In some embodiments, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter to obtain a first reference classification model and a second reference classification model includes: Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector; Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model; The model parameters of the original local classification model and the reference disturbance vector are subtracted to obtain the second reference classification model.

[0007] In some embodiments, after the original local classification model is updated according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model, the communication and memory efficient large model distributed training method further includes: Get the test text; Performing model evaluation on the target classification model according to the test text to obtain model generalization data; Acquire the number of texts of the language text according to the model generalization data; The target classification model is updated according to the amount of text.

[0008] In some embodiments, updating the target classification model according to the amount of text includes: If the amount of the text is less than a preset amount threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label and the sample category label; If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

[0009] In some embodiments, updating the target classification model according to the category proportions includes: If the category proportion is less than the preset proportion threshold, the original global model sent by the server is received, the language text is classified by the target classification model to obtain a local category label, the language text is classified by the original global model to obtain a global category label, and a third loss is calculated according to the local category label, the global category label and the text category label; Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model; If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0010] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a text classification method, which is applied to a terminal device. The method includes: Get the target text; The target text is classified by a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method described in the first aspect.

[0011] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a text classification device, which is applied to a terminal device, and the device includes: An acquisition module is used to acquire the target text; A classification module is used to classify the target text through a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method described in the first aspect.

[0012] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the communication and memory efficient large model distributed training method described in the first aspect or the text classification method described in the second aspect.

[0013] To achieve the above-mentioned objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the communication and memory-efficient large model distributed training method described in the first aspect or the text classification method described in the second aspect.

[0014] The communication and memory efficient large model distributed training method, text classification method, text classification device, electronic device, and computer readable storage medium of the embodiment of the present application, by acquiring language text and text category labels of the language text, the language text and the text category labels are used as training samples to complete the training of the local model of the terminal device. In order to enable the model to better adapt to unknown samples and improve the generalization ability and tolerance of the model to noise, the random seed sent by the server is received, and the model perturbation vector is generated according to the random seed. In order to reduce the communication overhead, the server does not send the model pre-trained on the large data set to the terminal device. In order to perform model training on the original local classification model and determine the impact of the model perturbation on the original local classification model, the perturbation hyperparameter is obtained, and the original local classification model is perturbed according to the model perturbation vector and the perturbation hyperparameter to obtain the first reference classification model and the second reference classification model. For model training, the first reference classification model and the second reference classification model are used to perform task reasoning respectively, and the language text is classified by the first reference classification model to obtain the first text category, and the language text is classified by the second reference classification model to obtain the second text category. The loss difference is calculated based on the text category label, the first text category and the second text category to determine the difference between the two inferences of the model. It will consume more communication resources for the terminal device to send model parameters to the server. In order to reduce communication overhead, the original symbol of the loss difference is obtained and sent to the server, so that the communication overhead of the terminal device is only one bit. In order to optimize the local model of the terminal device, the receiving server aggregates the original symbols to obtain the target symbol. By aggregating the original symbols, the model performance is avoided due to the large heterogeneity of local data sets of different terminal devices. The original local classification model is updated according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model, so that the terminal device can update the model without executing the gradient descent algorithm, reducing the memory and computing power requirements of the terminal device, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a communication and memory efficient large model distributed training method provided in an embodiment of the present application; Figure 2 yes Figure 1 Flow chart of step S130 in FIG. Figure 3 yes Figure 1 Flow chart of step S160 in FIG. Figure 4 is another flow chart of the communication and memory efficient large model distributed training method provided by an embodiment of the present application; Figure 5 yes Figure 4 Flowchart of step S440 in FIG. Figure 6 yes Figure 5 Flow chart of step S520 in FIG. Figure 7 is a flowchart of a text classification method provided in an embodiment of the present application; Figure 8 is a structural diagram of a text classification device provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0019] In related technologies, in order to obtain a deep learning model that performs well on specific tasks such as text classification tasks, a model pre-trained on a large data set is usually downloaded from a server, and this is used as a training starting point to train on a data set of a specific task on a local terminal device. During the training process, the terminal device needs to execute the gradient descent algorithm once or multiple times, which places high demands on the memory and computing power of the terminal device. In addition, the terminal device and the server often need to go through thousands of communication rounds, and the terminal device will send a large number of model parameters to the server in each communication round, which makes the communication overhead of model training large. These factors seriously affect the training of the local model of the terminal device, making the efficiency of model training low.

[0020] Based on this, the embodiments of the present application provide a large-model distributed training method, text classification method, text classification device, electronic device, and computer-readable storage medium with efficient communication and memory, aiming to improve the efficiency of model training.

[0021] The communication and memory efficient large model distributed training method, text classification method, text classification device, electronic device, and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the communication and memory efficient large model distributed training method in the embodiments of the present application is described.

[0022] The communication and memory efficient large model distributed training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The communication and memory efficient large model distributed training method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements communication and memory efficient large model distributed training methods, etc., but is not limited to the above forms.

[0023] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0024] Figure 1 It is an optional flowchart of the communication and memory efficient large model distributed training method provided in an embodiment of the present application, which is applied to a terminal device and may include but is not limited to steps S110 to S180.

[0025] Step S110, obtaining a language text and a text category label of the language text; Step S120, receiving a random seed sent by the server, and generating a model perturbation vector according to the random seed; Step S130, obtaining perturbation hyperparameters, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameters, and obtaining a first reference classification model and a second reference classification model; Step S140, classifying the language text using a first reference classification model to obtain a first text category; Step S150, classifying the language text by using a second reference classification model to obtain a second text category; Step S160, calculating the loss difference according to the text category label, the first text category and the second text category, and obtaining the original symbol of the loss difference; Step S170, sending the original symbol to the server, and receiving the target symbol obtained by aggregating the original symbol by the server; Step S180, updating the original local classification model according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model.

[0026] In step S110 of some embodiments, based on the text classification task, the terminal device collects text data from different data sources such as social media, databases, etc., obtains language text, and classifies the language text to obtain text category labels, and trains the local model of the terminal device based on the language text and the text category labels of the language text. The language text can be text materials such as product reviews, social media posts, articles, etc., and the text category labels can be set based on the text classification task. If the text classification task is emotion recognition, the text category labels are emotion categories, such as positive, negative or neutral. If the text classification task is intent recognition, the text category labels are intent categories, such as booking tickets, checking the weather, etc. If the text classification task is topic classification, the text category labels are topic categories, such as technology, sports, etc.

[0027] In step S120 of some embodiments, the server generates a random integer and sends the random integer as a random seed to the terminal device. If the random seed is generated by the terminal device, there may be a risk that the random seed is predicted or maliciously manipulated, and the computing resources of the terminal device are limited, and generating high-quality random seeds requires relatively high computing power. In addition, if the random seed is generated by the terminal device, the aggregation mode of directly using the sign of the difference in loss value for majority voting will cause the model training to fail to converge due to excessive randomness. In order to improve the security of model training, enable model training to converge, and reduce the computing burden of the terminal device, the embodiment of the present application generates a random seed by the server to estimate the model gradient of the original local classification model based on the random seed for model training. In addition, by sending a random seed instead of a perturbation vector, the consumption of server communication resources is reduced. The terminal device calculates the random seed through the random number generation algorithm built into the pseudo-random number generator to obtain a model perturbation vector. The random number generation algorithm can be a philox algorithm, a linear feedback shift register (LFSR) algorithm, a linear congruential generator (LCG) algorithm, and the like. The model perturbation vector is used to perturb the original local classification model to increase diversity during model training so that the model can better generalize to different data distributions.

[0028] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S230: Step S210, multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector; Step S220, adding the model parameters of the original local classification model and the reference disturbance vector to obtain a first reference classification model; Step S230, performing subtraction calculation on the model parameters of the original local classification model and the reference disturbance vector to obtain a second reference classification model.

[0029] In step S210 of some embodiments, a disturbance hyperparameter is obtained, and the disturbance hyperparameter is used to control the degree of disturbance of the model by the model disturbance vector, and the disturbance hyperparameter is a number greater than 0. If the model disturbance vector is z and the disturbance hyperparameter is e, the reference disturbance vector is e*z, where * represents a multiplication operation.

[0030] In step S220 of some embodiments, in the related art, during the model training process under the federated learning framework, the terminal device often needs to send the model gradient to the server. When the number of model parameters is large, the uplink communication overhead of the terminal device to the server will be large. In order to reduce the communication overhead, the embodiment of the present application decomposes the model gradient into a model perturbation vector and a corresponding gradient projection. In order to make the model training process converge as quickly as possible and improve the training efficiency of the model, it is necessary to calculate the gradient projection. In order to calculate the gradient projection, the reference perturbation vector is used to perform forward perturbation and backward perturbation on the original local classification model respectively. The model parameters of the original local classification model and the reference perturbation vector are added to forward perturb the original local classification model to obtain a first reference classification model. The first reference classification model is represented as w+e*z.

[0031] The original local classification model is a text classification model located locally on the terminal device. It can be a single model or a composite model that integrates multiple models. The original local classification model can be selected according to the actual situation, such as the BERT model, transformer model, etc. In order to obtain a deep model that performs well on the local language text of the terminal device, you can download a pre-trained large language model and use the large language model as the original local classification model to fine-tune the original local classification model. The language text can be input into the original local classification model for text classification to obtain the text category label.

[0032] For the positive and negative sentiment classification problem, the input language text is "<text to be classified> It was" (this is...), and the response template of the large language model is "Terrible." (terrible) or "Great." (good) to determine the text category label.

[0033] For intent recognition, the input language text is "<text to be classified> In the previous sentense, does the pronoun <span2>refer to <span1>? Yes or No?" (In the previous sentence, the pronoun <span2>Whether the object is an entity <span1>? ), the response template of the large language model is "Yes." or "No." to determine the text category label.

[0034] For topic classification, the input language text is "<Text to be classified> Question: this belongs to which subject? I found this answer <Some category>. Is that correct? Yes or No?" The response template of the large language model is Yes or No to determine the text category label.

[0035] In step S230 of some embodiments, the model parameters of the original local classification model are subtracted from the reference disturbance vector to perform backward perturbation on the original local classification model to obtain a second reference classification model. The second reference classification model is represented as we*z.

[0036] In the above steps S210 to S230, random perturbations are added to the original local classification model to estimate the gradient projection, thereby improving the efficiency of model training.

[0037] In step S140 of some embodiments, in order to measure the model classification performance of the first reference classification model, the language text is classified by the first reference classification model to obtain a first text category. The first text category is the text category predicted by the first reference classification model for the language text.

[0038] In step S150 of some embodiments, in order to measure the model classification performance of the second reference classification model, the second reference classification model is used to classify the language text to obtain a second text category. The second text category is the text category predicted by the second reference classification model for the language text.

[0039] See also Figure 3 In some embodiments, step S160 may include but is not limited to steps S310 to S330: Step S310, calculating the loss according to the text category label and the first text category to obtain a first loss; Step S320, calculating the loss according to the text category label and the second text category to obtain a second loss; Step S330, calculating the difference between the first loss and the second loss to obtain the loss difference.

[0040] In step S310 of some embodiments, based on the cross entropy loss function, loss calculation is performed according to the text category label and the first text category to measure the difference between the predicted text category output by the first reference classification model and the actual text category to obtain a first loss.

[0041] In step S320 of some embodiments, based on the cross entropy loss function, loss calculation is performed according to the text category label and the second text category to measure the difference between the predicted text category output by the second reference classification model and the actual text category to obtain a second loss.

[0042] In step S330 of some embodiments, the first loss is subtracted from the second loss to obtain a difference between the first loss and the second loss, thereby obtaining a loss difference. The loss difference is used to measure the difference in text classification performance between the first reference classification model and the second reference classification model.

[0043] Through the above steps S310 to S330, the loss difference value of the model's two text inferences can be obtained to guide the training of the local model of the terminal device based on the loss difference value.

[0044] In step S170 of some embodiments, under the federated learning framework, multiple terminal devices do not expose local private data, and under the coordination of the server, each terminal device jointly trains a common deep learning model. During the model training process, the terminal device needs to go through thousands of communication rounds with the server to converge the local model, so that the terminal device needs to send thousands of model parameters to the server, which usually consumes a lot of communication resources. In order to save communication resources, the original sign of the gradient projection is obtained, and the gradient projection is the projection of the model gradient on the model perturbation vector. The gradient projection is used to indicate the magnitude of the model parameter update, and the original sign is used to indicate the direction of the model parameter update. Divide the loss difference by the perturbation hyperparameter, determine the proportional relationship between the two, and obtain the gradient projection. The perturbation hyperparameter is used to control the size of the gradient projection, thereby affecting the efficiency of the model update to accelerate the convergence of the model. Since the perturbation hyperparameter is a positive number, the original sign of the gradient projection is the original sign of the loss difference.

[0045] Get the original symbol of the loss difference, and send the original symbol instead of the model parameter to the server. The original symbol is used to indicate that the loss difference is greater than or equal to 0 or less than 0, so that the terminal device only needs to consume one bit of communication resources in each communication round. The server screens the original symbols sent by each terminal device, selects the original symbol with the highest frequency as the target symbol, and sends the target symbol to each terminal device respectively. For example, the original symbol is set to include + or -, + indicates that the loss difference is greater than or equal to 0, and - indicates that the loss difference is less than 0. The original symbols received by the server for the three terminal devices participating in the update are s1, s2, and s3, respectively. The original symbols of s1 and s2 are both +, and the original symbol of s3 is -. The frequency of + is greater than the frequency of -, and the target symbol is +.

[0046] In step S180 of some embodiments, the terminal device needs to execute one or more gradient descent algorithms to update the local model, which places high demands on the memory and computing power of the terminal device. In order to reduce the memory and computing power requirements of the terminal device for model training and improve the efficiency of model training, the embodiment of the present application multiplies the preset learning rate, the target symbol and the model perturbation vector, and subtracts the model parameters of the original local classification model from the calculation result of the multiplication operation without executing the gradient descent algorithm to obtain the target classification model. The preset learning rate is the learning rate used for the text classification task, which is a hyperparameter and is usually set to a small positive number. For example, the preset learning rate can be in the interval [1e-7, 1e-6]. The target symbol is used to indicate the direction of the model parameter update, and the model perturbation vector is used to indicate the magnitude of the model parameter update. If the target symbol is s, the model perturbation vector is z, the preset learning rate is a, and the model parameters of the original local classification model are w, then the model parameters of the target classification model are expressed as w-asz.

[0047] Under the federated learning framework, each terminal device sends model parameters or gradient parameters to the server, and the server aggregates the model parameters or gradient parameters of each terminal device to determine the aggregate parameters. Each terminal device trains the local model according to the aggregate parameters sent by the server. However, this model training method has the following two problems, which cause the performance of the final fine-tuning model to degrade or even collapse. First, if a malicious terminal device sends an incorrect model parameter or gradient parameter to the server, the aggregate parameters will be distorted, causing the local models constructed by each terminal device based on the aggregate parameters to be subject to Byzantine attacks. Secondly, if the heterogeneity between the local data sets of each terminal device is too large, the local model personalization capability of the terminal device will be insufficient. In the related art, the terminal device uploads gradient information to correct the aggregate parameters. However, based on data privacy and security, as well as local computing costs and server communication costs, each terminal device does not exchange its own local gradient information, making it impossible for the server to determine the authenticity and accuracy of each gradient information. In the embodiment of the present application, the terminal device uploads the symbol of the gradient projection, and trains the local model according to the aggregated symbol, so that the local model has resistance to Byzantine attacks and data heterogeneity.

[0048] See also Figure 4 In some embodiments, after step S180, the communication and memory efficient large model distributed training method may also include but is not limited to steps S410 to S440: Step S410, obtaining a test text; Step S420, performing model evaluation on the target classification model according to the test text to obtain model generalization data; Step S430, obtaining the text quantity of the language text according to the model generalization data; Step S440, updating the target classification model according to the amount of text.

[0049] In step S410 of some embodiments, the terminal device obtains a test text, which is text data used to test the model performance of the local target classification model of the terminal device. The test text has a test category label, which is used to indicate the text category of the test text. The test text and the language text are texts for the same text classification task.

[0050] In step S420 of some embodiments, the test text is input into the target classification model for text classification to obtain the test text category. The test text category and the test category label are compared, and the first number of the test text category and the test category label are recorded. The number of test texts is obtained to obtain the second number. The ratio of the first number to the second number is used as the model generalization data.

[0051] In step S430 of some embodiments, if the model generalization data is greater than or equal to the preset generalization threshold, it means that the target classification model can accurately classify the test text locally on the terminal device, and the target classification model has good personalized performance, and the target classification model is not updated. If the model generalization data is less than the preset generalization threshold, it means that the generalization performance of the target classification model is insufficient and the target classification model needs to be updated, then the number of language texts used to train the target classification model is obtained to obtain the number of texts.

[0052] In step S440 of some embodiments, the update method of the target classification model is determined according to the amount of text, and the target classification model is updated according to the update method.

[0053] In the above steps S410 to S440, by evaluating the generalization of the target classification model, the target classification model is updated when the generalization is insufficient, so that the updated target classification model has better text classification performance on local text data, thereby improving the individual performance of the model.

[0054] See also Figure 5 In some embodiments, step S440 may include but is not limited to step S510 or step S520: Step S510, if the number of texts is less than a preset number threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label and the sample category label; Step S520, if the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

[0055] In step S510 of some embodiments, if the number of texts is less than a preset number threshold, indicating that the number of samples used to train the target classification model is insufficient, resulting in low model generalization, the language text is expanded by synonym replacement, random insertion, random deletion, back translation, text generation, etc. to obtain a sample text. The sample text is categorized to obtain a sample category label, or the text category label of the language text corresponding to the sample text is used as the sample category label. A new data set is constructed based on the language text, sample text, text category label, and sample category label, and the target classification model is updated using the new data set with reference to steps S110 to S180.

[0056] It should be noted that synonym replacement refers to randomly selecting some words from the language text and replacing them with synonyms. Random insertion refers to inserting words or phrases at random positions in the language text. Random deletion refers to randomly deleting some words in the language text. Back translation refers to translating the language text from the source language to the target language and then translating from the target language back to the source language. Text generation refers to generating new text data using a language model.

[0057] In step S520 of some embodiments, if the number of texts is greater than or equal to the preset number threshold, indicating that the number of samples is sufficient, the number of language texts under each text category is obtained to obtain the number of categories. The number of each category is divided by the number of texts to obtain the category proportion of each category, and the update method of the target classification model is determined according to the category proportion, and the target classification model is updated according to the update method. For example, the target classification model can classify 3 text categories, and these 3 text categories are c1, c2 and c3. The number of categories with text category c1 is s1, the number of categories with text category c2 is s2, and the number of categories with text category c3 is 0, then the category proportion of text category c1 is s1 / (s1+s2), the category proportion of text category c2 is s2 / (s1+s2), and the category proportion of text category c2 is 0.

[0058] Through the above steps S510 to S520, the update method of the target classification model is differentiated according to different situations, so as to accurately and efficiently update the model according to a specific update method, thereby improving the text classification accuracy of the model on local text data.

[0059] See also Figure 6 In some embodiments, step S520 may include but is not limited to steps S610 to S630: Step S610, if the category proportion is less than the preset proportion threshold, the original global model sent by the server is received, the language text is classified by the target classification model to obtain a local category label, the language text is classified by the original global model to obtain a global category label, and the third loss is calculated according to the local category label, the global category label and the text category label; Step S620, adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model; Step S630: If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0060] In step S610 of some embodiments, if the category proportion of at least one category is less than the preset proportion threshold, it means that the generalization of the model is low due to the imbalance of the number of samples of each category. In order to improve the generalization of the model, it is necessary to use sample texts of other terminal devices. Specifically, the server initializes the model, obtains the original global model, and broadcasts the original global model to each terminal device. The terminal device receives the original global model sent by the server, inputs the language text into the target classification model for text classification, obtains the local category label, inputs the language text into the original global model for text classification, and obtains the global category label. Based on the cross entropy loss function, the loss is calculated according to the local category label and the text category label to obtain the first sub-loss, and the first sub-loss is used to measure the model classification performance of the target classification model. Based on the cross entropy loss function, the loss is calculated according to the global category label and the text category label to obtain the second sub-loss, and the second sub-loss is used to measure the model classification performance of the original global model. Based on the cross entropy loss function, the loss is calculated according to the global category label and the local category label to obtain the third sub-loss, and the third sub-loss is used to measure the classification performance difference between the original global model and the target classification model, so that the target classification model can learn the text classification knowledge of the original global model. The first sub-loss, the second sub-loss and the third sub-loss are summed to obtain the third loss.

[0061] In step S620 of some embodiments, the third loss is minimized, the model parameters of the target classification model are adjusted, and a candidate classification model is obtained. The model parameters of the candidate classification model are sent to the server, and the server calculates the mean of the model parameters of each terminal device to obtain an aggregated classification model. The original global model is updated according to the aggregated classification model, and the updated original global model is sent to the terminal device, and the target classification model is updated according to step S610. The data distribution characteristics of different terminal devices can be captured by the original global model, and the degradation of model performance caused by data distribution differences can be reduced. The target classification model is updated using the original global model and the local language text of the terminal device, so that the target classification model can not only learn the global classification knowledge captured by the original global model, but also learn the local classification knowledge implied by the local text data, thereby improving the generalization of the target classification model.

[0062] In step S630 of some embodiments, if the category proportion of each category is greater than or equal to the preset proportion threshold, it means that the local text data distribution of the terminal device is relatively balanced, then one or more network layers are randomly selected from the network layer of the target classification model as the target network layer, and a low-rank adaptation layer is added after the target network layer. The language text is input into the target classification model with the low-rank adaptation layer added for text classification to obtain a predicted text category. Based on the cross entropy loss function, the loss of the predicted text category and the text category label is calculated to obtain a fourth loss. Minimize the fourth loss, fine-tune the network parameters of the low-rank adaptation layer, and keep the network parameters of other network layers unchanged. By fine-tuning the model structure of the target classification model, the generalization of the target classification model for text classification tasks can be improved.

[0063] Through the above steps S610 to S630, the generalization of the target classification model can be improved, so that the target classification model can accurately classify the local text data of the terminal device, thereby improving the accuracy of text classification.

[0064] Each terminal device stores a local private model and a local private data set, each private text in the local private data set has a text category label, and shares a pseudo-random number generator. The communication and memory efficient large model distributed training method of the embodiment of the present application includes: in the current communication round, each terminal device receives a random seed sent by the server, inputs the random seed into the pseudo-random number generator to generate a model perturbation vector, and sets the perturbation hyperparameter. The local private model is forward perturbed according to the perturbation hyperparameter and the model perturbation vector to obtain a first perturbation model. If the perturbation hyperparameter is e, the model perturbation vector is z, and the model parameter of the local private model is w, then the first perturbation model is represented as w+ez. The local private model is backward perturbed according to the perturbation hyperparameter and the model perturbation vector to obtain a second perturbation model, and the second perturbation model is represented as w-ez. Each terminal device selects a data batch in the local private data set, performs text reasoning on the data batch through the first perturbation model, and obtains a first text category. The data batch is textually reasoned through the second perturbation model to obtain a second text category. Based on the cross entropy loss function, the loss is calculated according to the text category label and the first text category to obtain the first loss. Based on the cross entropy loss function, the loss is calculated according to the text category label and the second text category to obtain the second loss. The first loss is subtracted from the second loss to obtain the loss difference, and the original symbol of the loss difference is obtained. Each terminal device sends its original symbol to the server, and the server selects most of the original symbols as the aggregate symbol s and forwards the aggregate symbol s to all terminal devices. Each terminal device updates the model parameter w of the local private model according to the aggregate symbol s, the model perturbation vector z and the preset learning rate, that is, w=w-asz, and enters the next communication round until the communication round reaches the preset round.

[0065] See also Figure 7 The present application also provides a text classification method, which is applied to a terminal device. The text classification method may include but is not limited to steps S710 to S720: Step S710, obtaining the target text; Step S720, classify the target text using the target classification model to obtain the target text category.

[0066] In step S710 of some embodiments, the terminal device obtains a target text, which is text data to be classified.

[0067] In step S720 of some embodiments, in order to obtain the text category of the target text, the target text is input into a target classification model for text classification to obtain the target text category.

[0068] In the above steps S710 to S720, the model training efficiency of the target classification model is relatively high, and when the target classification model is used for text classification, the efficiency of text classification is greatly improved.

[0069] See also Figure 8 The present application also provides a text classification device, which is applied to a terminal device and can implement the above-mentioned text classification method. The text classification device includes: An acquisition module 810 is used to acquire a target text; The classification module 820 is used to perform text classification on the target text through the target classification model to obtain the target text category.

[0070] In some embodiments, the text classification device is further used to: Obtaining a language text and a text category label of the language text; Receiving a random seed sent by a server, and generating a model perturbation vector according to the random seed; Obtaining a perturbation hyperparameter, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter, to obtain a first reference classification model and a second reference classification model; Performing text classification on the language text by using the first reference classification model to obtain a first text category; Performing text classification on the language text by using the second reference classification model to obtain a second text category; Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference; Sending the original symbol to the server, and receiving a target symbol obtained by aggregating the original symbol by the server; The original local classification model is updated according to the target symbol, the model perturbation vector and a preset learning rate to obtain a target classification model.

[0071] In some embodiments, the text classification device is further used to: Calculate the loss according to the text category label and the first text category to obtain a first loss; Calculate the loss according to the text category label and the second text category to obtain a second loss; The difference between the first loss and the second loss is calculated to obtain the loss difference.

[0072] In some embodiments, the text classification device is further used to: Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector; Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model; The model parameters of the original local classification model and the reference disturbance vector are subtracted to obtain the second reference classification model.

[0073] In some embodiments, the text classification device is further used to: Get the test text; Performing model evaluation on the target classification model according to the test text to obtain model generalization data; Acquire the number of texts of the language text according to the model generalization data; The target classification model is updated according to the amount of text.

[0074] In some embodiments, the text classification device is further used to: If the amount of the text is less than a preset amount threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label and the sample category label; If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

[0075] In some embodiments, the text classification device is further used to: If the category proportion is less than the preset proportion threshold, the original global model sent by the server is received, the language text is classified by the target classification model to obtain a local category label, the language text is classified by the original global model to obtain a global category label, and a third loss is calculated according to the local category label, the global category label and the text category label; Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model; If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

[0076] The specific implementation of the text classification device is basically the same as the specific implementation of the above-mentioned text classification method, and will not be repeated here.

[0077] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned communication and memory efficient large model distributed training method or text classification method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0078] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the communication and memory efficient large model distributed training method or text classification method of the embodiments of this application; Input / output interface 930, used to implement information input and output; Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 ); The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0079] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned communication and memory-efficient large-model distributed training method or text classification method.

[0080] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0082] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0083] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0085] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0086] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0091] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A communication and memory efficient large model distributed training method, characterized in that Applied to a terminal device, the method comprises: Obtaining a language text and a text category label of the language text; Receiving a random seed sent by a server, and generating a model perturbation vector according to the random seed; Obtaining a perturbation hyperparameter, performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter, to obtain a first reference classification model and a second reference classification model; Performing text classification on the language text by using the first reference classification model to obtain a first text category; Performing text classification on the language text by using the second reference classification model to obtain a second text category; Calculating a loss difference according to the text category label, the first text category, and the second text category, and obtaining an original sign of the loss difference; Sending the original symbol to the server, and receiving a target symbol obtained by aggregating the original symbol by the server; The original local classification model is updated according to the target symbol, the model perturbation vector and a preset learning rate to obtain a target classification model.

2. The method according to claim 1, characterized in that The calculating the loss difference according to the text category label, the first text category and the second text category includes: Calculate the loss according to the text category label and the first text category to obtain a first loss; Calculate the loss according to the text category label and the second text category to obtain a second loss; The difference between the first loss and the second loss is calculated to obtain the loss difference.

3. The method according to claim 1, characterized in that The step of performing model perturbation on the original local classification model according to the model perturbation vector and the perturbation hyperparameter to obtain a first reference classification model and a second reference classification model includes: Multiplying the model disturbance vector and the disturbance hyperparameter to obtain a reference disturbance vector; Adding the model parameters of the original local classification model and the reference disturbance vector to obtain the first reference classification model; The model parameters of the original local classification model and the reference disturbance vector are subtracted to obtain the second reference classification model.

4. The method according to any one of claims 1 to 3, characterized in that: After the original local classification model is updated according to the target symbol, the model perturbation vector and the preset learning rate to obtain the target classification model, the communication and memory efficient large model distributed training method further includes: Get the test text; Performing model evaluation on the target classification model according to the test text to obtain model generalization data; Acquire the number of texts of the language text according to the model generalization data; The target classification model is updated according to the amount of text.

5. The method according to claim 4, characterized in that The updating of the target classification model according to the amount of text includes: If the amount of the text is less than a preset amount threshold, the language text is expanded to obtain a sample text, and a sample category label of the sample text is obtained, and the target classification model is updated according to the language text, the sample text, the text category label and the sample category label; If the number of texts is greater than or equal to the preset number threshold, the number of categories of the language text is obtained, the category proportion is determined according to the number of categories and the number of texts, and the target classification model is updated according to the category proportion.

6. The method according to claim 5, characterized in that The updating of the target classification model according to the category proportions includes: If the category proportion is less than the preset proportion threshold, the original global model sent by the server is received, the language text is classified by the target classification model to obtain a local category label, the language text is classified by the original global model to obtain a global category label, and a third loss is calculated according to the local category label, the global category label and the text category label; Adjusting the model parameters of the target classification model according to the third loss to obtain a candidate classification model, sending the candidate classification model to the server, receiving an aggregated classification model obtained by aggregating the candidate classification models by the server, and updating the candidate classification model according to the aggregated classification model; If the category proportion is greater than or equal to the preset proportion threshold, a low-rank adaptation layer is added to the target classification model, and the low-rank adaptation layer is fine-tuned according to the language text and text category label.

7. A text classification method, characterized in that: Applied to a terminal device, the method comprises: Get the target text; The target text is classified by a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method according to any one of claims 1 to 6.

8. A text classification device, characterized in that: Applied to a terminal device, the device comprises: An acquisition module is used to acquire the target text; A classification module is used to perform text classification on the target text through a target classification model to obtain a target text category; the target classification model is trained according to the communication and memory efficient large model distributed training method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the communication and memory efficient large model distributed training method described in any one of claims 1 to 6 or the text classification method described in claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the communication and memory efficient large model distributed training method described in any one of claims 1 to 6 or the text classification method described in claim 7.

Images