Model federated fine-tuning methods, text classification methods, devices, media and equipment

Abstract

This disclosure relates to a model federated fine-tuning method, a text classification method, an apparatus, a medium, and a device. The pre-trained target model includes a first model deployed on a server and second models deployed on at least one client. The fine-tuning method applied to the target client includes: segmenting text samples into words; generating multiple embedding vectors corresponding to each word segment using the second model; determining a target word from the multiple word segments that has classification utility for the text category labeled by the classification tag; subjecting the embedding vectors of the other word segments (excluding the target word) to noise perturbation to obtain perturbation vectors; and fine-tuning the model parameters of the first model collaboratively with the server and other clients based on the perturbation vectors, the embedding vectors corresponding to the target word, and the classification tag. This method can effectively improve the usability of the target model in classification tasks while strengthening client data privacy protection.

Description

Technical Field

[0001] This disclosure relates to the field of privacy protection, specifically to a model federated fine-tuning method, a text classification method, an apparatus, a medium, and an device. Background Technology

[0002] In recent years, pre-trained language models (PLMs), represented by Bidirectional Encoder Representation from Transformers (BERT) and Generative Pre-Training (GPT) models, have demonstrated powerful text learning capabilities and have been widely applied in various fields such as finance, law, and healthcare. To improve the usability of pre-trained language models in downstream applications, a common approach is to fine-tune the models on datasets relevant to the downstream tasks. However, due to resource or technical limitations, many users cannot independently obtain and fine-tune pre-trained language models. This has given rise to a new business scenario combining Language Models (LMs) with the Models-as-a-Service (MaaS) ecosystem. In MaaS, servers with ample computing resources and technical expertise provide rich pre-trained models, service resources, and core functionalities, while clients can access a one-stop MaaS platform to fine-tune, deploy, and invoke models using their own private datasets, thereby customizing language models to meet their specific needs.

[0003] However, while this solution provides clients with efficient and customizable LM services, it also introduces risks of server-side model privacy leaks and client-side data privacy leaks. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides a model federated fine-tuning method, wherein the pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client, and the method is applied to the target client among the at least one client, including:

[0006] Obtain text samples and their corresponding classification labels;

[0007] The text sample is segmented into multiple words, and the embedding vectors corresponding to each of the multiple words are generated by the second model deployed on the target client.

[0008] From the plurality of word segments, determine the target word segment that has classification effect on the text category labeled by the classification tag;

[0009] The embedding vectors corresponding to the other words in the plurality of word segments, excluding the target word, are subjected to noise perturbation processing to obtain perturbation vectors;

[0010] Based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label, the model parameters of the first model are fine-tuned in collaboration with the server and other clients among the at least one client.

[0011] Secondly, this disclosure provides a text classification method, including:

[0012] Get the text to be categorized;

[0013] Based on the text to be classified, a classification prediction result for the text to be classified is obtained through a pre-tuned target model, wherein the target model is fine-tuned according to the model federated fine-tuning method provided in the first aspect of this disclosure.

[0014] Thirdly, this disclosure provides a model federated fine-tuning apparatus, wherein the pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client, and the apparatus is applied to the target client among the at least one client, comprising:

[0015] The first acquisition module is used to acquire text samples and the classification labels corresponding to the text samples;

[0016] The local computing module is used to segment the text sample to obtain multiple segments, and generate the embedding vectors corresponding to each of the multiple segments through the second model deployed on the target client;

[0017] The target word segmentation determination module is used to determine, from the plurality of word segments, the target word that has classification effect on the text category labeled by the classification tag;

[0018] The perturbation module is used to perform noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments besides the target word to obtain perturbation vectors;

[0019] The fine-tuning module is used to fine-tune the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label.

[0020] Fourthly, this disclosure provides a text classification device, comprising:

[0021] The second acquisition module is used to acquire the text to be classified.

[0022] A classification module is used to obtain a classification prediction result of the text to be classified by using a pre-tuned target model, wherein the target model is obtained by fine-tuning the model federated fine-tuning method provided in the first aspect of this disclosure.

[0023] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the model federated fine-tuning method provided in the first aspect of this disclosure or the steps of the text classification method provided in the second aspect of this disclosure.

[0024] Sixthly, this disclosure provides an electronic device, comprising:

[0025] A storage device on which computer programs are stored;

[0026] A processing device is configured to execute the computer program in the storage device to implement the steps of the model federated fine-tuning method provided in the first aspect of this disclosure or the steps of the text classification method provided in the second aspect of this disclosure.

[0027] In the above technical solution, the pre-trained target model includes a first model deployed on the server and a second model deployed on at least one client. This means that only a portion of the target model (the second model) needs to be disclosed to the clients participating in the federated model tuning, thus protecting server-side model privacy to a certain extent. Furthermore, noise perturbation is applied only to segmented words corresponding to multiple text samples that lack classification utility for the text categories labeled with the classification tags. This reduces perturbation to target segmented words that do have classification utility for the text categories labeled with the classification tags, where the target segmented words have a significant impact on model classification performance. This adaptive perturbation mechanism provides a more subtle trade-off between the usability of the target model and client data privacy, effectively improving the usability of the target model in downstream classification tasks while strengthening client data privacy protection.

[0028] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0030] Figure 1 This is a flowchart illustrating a model federation fine-tuning method according to an exemplary embodiment.

[0031] Figure 2 This is a schematic diagram illustrating a model federation fine-tuning process according to an exemplary embodiment.

[0032] Figure 3 This is a flowchart illustrating a text classification method according to an exemplary embodiment.

[0033] Figure 4 This is a block diagram illustrating a model federation fine-tuning device according to an exemplary embodiment.

[0034] Figure 5 This is a block diagram illustrating a text classification device according to an exemplary embodiment.

[0035] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0036] As discussed in the background section, while MaaS provides clients with efficient and customizable Modeling Library (LM) services, it also introduces risks of server-side model privacy breaches and client-side data privacy breaches. Specifically, on the one hand, because the pre-training process consumes significant resources, the weights of the Modeling Library (PLM) are typically considered proprietary server-side data and cannot be directly disclosed. On the other hand, client-side text data often contains personally identifiable information, trade secrets, etc., and directly disclosing the raw data to the server would lead to serious privacy breaches, which undoubtedly hinders privacy-conscious clients from using customized services.

[0037] In related technologies, to protect model privacy, the server deploys the core of the PLM (Programmable Module) as a black box on a cloud server, exposing only the embedding blocks to the client. To protect data privacy, the client adds noise perturbation to the embedding vector of the input text and sends the perturbated embedding vector to the server for subsequent model fine-tuning. Since the noise perturbation introduced by the client inevitably leads to a decrease in the model's usability in downstream tasks, a good trade-off between model usability and data privacy protection cannot be achieved.

[0038] In view of this, this disclosure provides a model federated fine-tuning method, a text classification method, an apparatus, a medium, and an equipment.

[0039] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0040] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0043] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0044] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0045] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0046] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0047] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0048] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0049] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0050] Figure 1 This is a flowchart illustrating a model federated fine-tuning method according to an exemplary embodiment. In this disclosure, to improve model training efficiency, a target model can be pre-trained on a server. Then, the pre-trained target model can be fine-tuned using a Split-and-Privatize (SAP) federated fine-tuning framework. The SAP federated fine-tuning framework may include the aforementioned server and at least one client. The pre-trained target model is divided into a first model deployed on the server and second models deployed on each client (i.e., each client in the SAP federated fine-tuning framework deploys a second model). The first model is a top-level model, and the second model is a bottom-level model; that is, the output of the second model serves as the input of the first model. In one possible implementation, the target model can be a language model.

[0051] For example, if the SAP Federated Fine-Tuning Framework includes a server and a client, then the first model is deployed on the server and the second model is deployed on the client.

[0052] For example, if the SAP Federated Fine-Tuning Framework includes a server and three clients, then the first model is deployed on the server, and the second model is deployed on each of the three clients, and the model structure and model parameters of the second model deployed on each client are the same.

[0053] In this disclosure, the segmentation position of the target model can be determined based on the actual application scenario, so as to divide the model into a first model and a second model. The second model of the target client includes at least an embedding block, and the embedding block includes multiple embedding layers.

[0054] In one implementation, the second model of the target client includes an embedding block for converting the word segments corresponding to the text into embedding vectors (i.e., text representations). The embedding block may include multiple concatenated embedding layers. The first model may include a concatenated encoder and an output layer. The encoder includes multiple concatenated encoding modules. The first encoding module of the multiple concatenated encoding modules is connected to the last embedding layer in the embedding block, and the last encoding module of the multiple concatenated encoding modules is connected to the output layer. The encoder is used to learn a generalized representation of the text. The output layer is constructed based on the attributes of the downstream task and processes the generalized representation output by the encoder into the model output (i.e., classification prediction result) according to the specific task requirements.

[0055] In another implementation, the second model of the target client includes concatenated embedding blocks and at least one encoding module. The embedding blocks are used to convert the word segments corresponding to the text into embedding vectors (i.e., text representations). The embedding blocks may include multiple concatenated embedding layers. The encoding modules are used to learn generalized representations of the text. When the second model of the target client includes multiple encoding modules, the multiple encoding modules are concatenated, and the first encoding module among the multiple encoding modules is connected to the last embedding layer of the embedding block. The first model may include a concatenated encoder and an output layer. The encoder includes multiple concatenated encoding modules, the last encoding module among the multiple concatenated encoding modules is connected to the output layer, and the first encoding module among the multiple concatenated encoding modules is connected to the last encoding module of the second model. The encoder is used to further learn generalized representations of the text. The output layer is constructed according to the attributes of the downstream task and processes the generalized representation output by the encoder into the model output (i.e., classification prediction results) according to the specific task requirements.

[0056] The above-described model federated fine-tuning method can be applied to a target client within at least one client in the SAP federated fine-tuning framework. The target client can be any client among the at least one client, or a client among the at least one client that meets preset conditions. For example... Figure 1 As shown, the federal fine-tuning method for this model may include the following S101 to S105.

[0057] In S101, obtain the text sample and the corresponding classification label of the text sample.

[0058] In this disclosure, for client data privacy reasons, both the text samples and category labels can be located locally on the target client; that is, the text samples and corresponding category labels do not leave the target client's local storage (i.e., ...). Figure 2 The local data shown can also be located on other storage devices.

[0059] In S102, the text sample is segmented into multiple words, and the embedding vectors corresponding to each word are generated by the second model deployed on the target client.

[0060] In this disclosure, text samples can be segmented according to a vocabulary to obtain multiple segments; then, each segment is input into a second model on the target client to obtain the embedding vectors corresponding to each segment (e.g., ...). Figure 2 (As shown).

[0061] In S103, target segmentation words that have classification utility for the text category labeled by the classification tag are determined from multiple segmentation words.

[0062] In this disclosure, classification utility is an indicator of classification quality, aiming to identify the word segment that contributes most to the utility objective (i.e., classification utility) in each class of text samples. Words that appear frequently in text samples of the text category labeled by the classification label, but appear less frequently in text samples of other categories, are considered to help distinguish the text category labeled by the classification label from other categories, and these words are considered to have classification utility for the text category labeled by the classification label.

[0063] In S104, noise perturbation is applied to the embedding vectors of the other words besides the target word in multiple word segments to obtain perturbation vectors.

[0064] In this disclosure, when the second model only contains embedding blocks, the computational burden on the client is relatively small, but the server can easily recover the corresponding text content from the transmitted embedding vectors through nearest neighbor search. Therefore, after generating the embedding vectors corresponding to multiple word segments, they are not directly sent to the server. Instead, these embedding vectors need to be perturbed with noise to avoid the risk of client data privacy leakage caused by the server recovering the corresponding text content from the transmitted embedding vectors. However, while introducing noise perturbation enhances data privacy protection, it also leads to a certain loss in model performance, thus presenting a trade-off between model utility and data privacy protection. To improve this trade-off, word segments that have a significant impact on the model's classification performance are first selected from multiple word segments, i.e., target word segments that have classification utility for the text categories labeled by the classification tags. Then, such as Figure 2As shown, noise perturbation (adding noise) is applied only to the word segments corresponding to the text samples that do not have classification utility for the text categories labeled by the classification labels. This reduces the perturbation of the target word segments that have classification utility for the text categories labeled by the classification labels. In order to adaptively apply the privacy protection mechanism to perturb the embedding vectors corresponding to each word, a more subtle trade-off is made between the usability of the target model and data privacy.

[0065] Furthermore, if the second model contains more encoding modules, and the text identifiers generated based on the second model are more abstract and general, then it becomes more difficult to recover the original text content from them.

[0066] In S105, the model parameters of the first model are fine-tuned in collaboration with the server and at least one other client, based on the perturbation vector, the embedding vector corresponding to the target word, and the classification label.

[0067] In this disclosure, such as Figure 2 As shown, during the process of fine-tuning the model parameters of the first model using the SAP Federated Fine-Tuning Framework, the model parameters of the second model remain unchanged. Furthermore, when fine-tuning the first model, in order to reduce computational overhead, a parameter-efficient fine-tuning method can be chosen, namely, freezing some parameters of the first model (i.e.,... Figure 2 The frozen module in the model only adjusts the unfrozen parameters of the first model (i.e., Figure 2 (Adjustable module in the middle).

[0068] In the above technical solution, the pre-trained target model includes a first model deployed on the server and a second model deployed on at least one client. This means that only a portion of the target model (the second model) needs to be disclosed to the clients participating in the federated model tuning, thus protecting server-side model privacy to a certain extent. Furthermore, noise perturbation is applied only to segmented words corresponding to multiple text samples that lack classification utility for the text categories labeled with the classification tags. This reduces perturbation to target segmented words that do have classification utility for the text categories labeled with the classification tags, where the target segmented words have a significant impact on model classification performance. This adaptive perturbation mechanism provides a more subtle trade-off between the usability of the target model and client data privacy, effectively improving the usability of the target model in downstream classification tasks while strengthening client data privacy protection.

[0069] The following is a detailed explanation of the specific implementation method for determining the target word that has classification effect on the text category labeled by the classification tag in S103 above. Specifically, it can be achieved through the following steps (1) to (2):

[0070] Step (1): Obtain the category utility words corresponding to the text categories marked by the category labels.

[0071] In this disclosure, the classification utility terms are the K reference words in the vocabulary that have the highest utility importance (UI) for the text category labeled by the classification tag. The vocabulary is used to segment the text samples, and K is greater than 1. The classification result of the text classification model can include multiple preset categories, wherein the classification utility terms corresponding to each preset category can be pre-established.

[0072] Step (2): Among multiple segmented words, the segmented words that belong to the category utility words are identified as the target segmented words.

[0073] The following details the method for determining the category utility terms corresponding to the text categories marked by the category labels. Specifically, this can be achieved through the following steps (a1) to (a3):

[0074] Step (a1): For each reference word in the vocabulary, obtain the frequency of the reference word in the text samples under each preset category.

[0075] Step (a2): Determine the utility importance of the reference word to the text category labeled by the classification tag based on the frequency of the reference word in the text samples under each preset category.

[0076] In this disclosure, the training set on the target client includes multiple text samples, which are divided into multiple categories according to text type. For each reference word in the vocabulary, the frequency of the reference word in the text samples under each preset category can be obtained through statistical analysis; then, based on the frequency of the reference word in the text samples under each preset category, the utility importance of the reference word to the text category labeled by the classification tag is determined.

[0077] Step (a3): Identify the K reference words in the vocabulary that have the highest utility importance for the text categories labeled with the classification tags as classification utility words.

[0078] The following is a detailed explanation of the specific implementation method for determining the utility importance of the reference word to the text category labeled by the classification tag based on the frequency of the reference word in the text samples under each preset category in step (a2). Specifically, it can be achieved through the following steps (a21) and (a22):

[0079] Step (a21): For each of the multiple preset categories other than the text category labeled by the classification label, determine the ratio of the frequency of the reference word appearing in the text samples of the text category labeled by the classification label to the frequency of the reference word appearing in the text samples of the other category.

[0080] Step (a22): Determine the utility importance of the reference word to the text category labeled by the classification tag based on the ratio corresponding to each other category.

[0081] In one possible implementation, the sum of the logarithms of the ratios corresponding to each other category can be used to determine the utility importance of the reference word to the text category labeled by the classification tag.

[0082] For example, the utility importance of the reference word to the text category labeled by the classification tag can be determined by the following equation based on the ratio corresponding to each other category:

[0083]

[0084] Among them, UI mc For the reference word t in the vocabulary list m The utility importance of the text category c labeled by the classification tag; p(t=t m |y=c) is the reference word t m The frequency of occurrence in text samples under category c; p(t=t m |y=c') is the reference word t m Frequency of occurrence in text samples under other categories c'; t represents the reference word, y represents the text category.

[0085] For example, the aforementioned preset categories include C1, C2, and C3. If the text category labeled in the above text sample is C1, then a certain reference word t in the vocabulary... m Utility importance for category C1

[0086] The following details the specific implementation method for noise perturbation processing of the embedding vectors corresponding to the segmented words other than the target segmented word in S104 above, to obtain the perturbation vector. Specifically, it can be achieved through the following steps (b1) to (b3):

[0087] Step (b1): Perform noise perturbation processing on the embedding vectors corresponding to the other words in the multiple word segments, excluding the target word.

[0088] In this disclosure, such as Figure 2As shown, random noise can be added to the embedding vectors of each segment other than the target segment in multiple segmentations, based on differential privacy or Probably Approximately Correct (PAC) privacy mechanisms, to perform noise perturbation. For example, when performing noise perturbation based on differential privacy, a Gaussian mechanism or a random mechanism satisfying dχ-privacy (a variant of local differential privacy) can be used to add random noise to the embedding vectors of each other segment.

[0089] For example, random noise can be added to the corresponding embedding vector using a random mechanism that satisfies dχ-privacy, according to the following equation: in, Add a noise vector to the embedding vector φ(x) The resulting vector (i.e., the embedding vector obtained after noise perturbation processing) Noise vector The probability density function is denoted by η; η is a parameter in the random mechanism that satisfies dχ-privacy and is used to control the noise level. The smaller η is, the larger the corresponding noise variance and the stronger the privacy protection capability.

[0090] Step (b2): For each embedding vector obtained after noise perturbation processing, determine whether the reference embedding vector corresponding to each reference word in the vocabulary contains that embedding vector.

[0091] Step (b3): ​​If the reference embedding vector corresponding to each reference word in the vocabulary does not contain the embedding vector, then the reference embedding vector that is closest to the embedding vector is determined as the perturbation vector.

[0092] In this disclosure, the reference embedding vectors corresponding to each reference word in the vocabulary constitute an embedding vector space. The embedding vectors obtained after noise perturbation may not fall within this space. Therefore, it is necessary to replace and adjust the embedding vectors that do not fall within the embedding vector space. Specifically, for each embedding vector obtained after noise perturbation, first determine whether the embedding vector is contained in the embedding vector space. If the embedding vector is contained in the space, it indicates that the embedding vector falls within the space, and in this case, there is no need to replace or adjust it; the embedding vector can be directly determined as the corresponding perturbation vector. If the embedding vector is not contained in the space, it indicates that the embedding vector does not fall within the space, and in this case, ... Figure 2 As shown, the embedding vector can be replaced by the nearest reference embedding vector (i.e., the nearest neighbor reference embedding vector) in the embedding vector space, that is, the nearest reference embedding vector is determined as the corresponding perturbation vector.

[0093] The following provides a detailed explanation of the specific implementation method for fine-tuning the model parameters of the first model in collaboration with the server and at least one other client in step S105, based on the perturbation vector, the embedding vector corresponding to the target word, and the classification label. Specifically, this can be achieved through the following steps (c1) and (c2).

[0094] Step (c1): Send the perturbation vector and the embedding vector corresponding to the target word to the server. The server then uses the first model to generate the classification prediction result of the text sample based on the perturbation vector and the embedding vector corresponding to the target word, and sends it to the target client.

[0095] Step (c2): Based on the classification prediction results and classification labels of the text samples, determine the first gradient information corresponding to the output layer of the first model, and send the first gradient information to the server. The server then updates the model parameters based on the first gradient information and the second gradient information corresponding to the output layer of the first model sent by other clients.

[0096] Specifically, such as Figure 2 As shown, the target client can send each perturbation vector and the embedding vector corresponding to the target word to the server. The server receives each perturbation vector and the embedding vector corresponding to the target word, then inputs them into the local first model to obtain the classification prediction result of the text sample, and sends the classification prediction result of the text sample to the target client. The target client determines the model loss of the first model based on the received classification prediction result and the classification label of the text sample, and then determines the first gradient information corresponding to the output layer of the first model based on the model loss, and sends the first gradient information to the server. When the SAP Federated Fine-Tuning Framework includes multiple clients, in addition to receiving the first gradient information sent by the target client, the server also receives the second gradient information corresponding to the output layer of the first model sent by each of the other clients (excluding the target client). In this case, the server can perform backpropagation in the first model based on the average of the first gradient information and the second gradient information to update the model parameters of the first model.

[0097] Figure 3 This is a flowchart illustrating a text classification method according to an exemplary embodiment, wherein the method can be applied to a target client among the at least one of the aforementioned clients. Figure 3 As shown, the text classification method may include S201 and S202.

[0098] In S201, obtain the text to be classified.

[0099] In S202, the classification prediction result of the text to be classified is obtained by using a pre-tuned target model based on the text to be classified.

[0100] In this disclosure, the same forward computation steps as the aforementioned federated fine-tuning method can be used to obtain the classification prediction result for the text to be classified. Specifically, after word segmentation, embedding vector calculation, and adaptive perturbation of the embedding vector, the target client sends the perturbated vector to the server, and the server generates the classification prediction result for the text to be classified using the first model. The target word segmentation in the adaptive perturbation mechanism is the target word segmentation determined in the aforementioned fine-tuning process, and the first model on the server is the first model after the aforementioned fine-tuning.

[0101] In the above technical solution, the pre-trained target model includes a first model deployed on the server and a second model deployed on at least one client. This means that only a portion of the target model (the second model) needs to be disclosed to the clients participating in the federated model tuning, thus protecting server-side model privacy to a certain extent. Furthermore, noise perturbation is applied only to segmented words corresponding to multiple text samples that lack classification utility for the text categories labeled with the classification tags. This reduces perturbation to target segmented words that do have classification utility for the text categories labeled with the classification tags, where the target segmented words have a significant impact on model classification performance. This adaptive perturbation mechanism provides a more subtle trade-off between the usability of the target model and client data privacy, effectively improving the usability of the target model in downstream classification tasks while strengthening client data privacy protection.

[0102] Figure 4 This is a block diagram illustrating a model federated fine-tuning device according to an exemplary embodiment. The pre-trained model includes a first model deployed on a server and second models deployed on at least one client. The model federated fine-tuning device 300 is applied to a target client among the at least one client, including:

[0103] The first acquisition module 301 is used to acquire text samples and corresponding classification labels;

[0104] The local computing module 302 is used to segment the text sample to obtain multiple segments, and generate the embedding vectors corresponding to each of the multiple segments through the second model deployed on the target client;

[0105] The target word segmentation determination module 303 is used to determine, from the plurality of word segments, the target word that has classification effect on the text category marked by the classification label;

[0106] The perturbation module 304 is used to perform noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments except the target word to obtain a perturbation vector;

[0107] The fine-tuning module 305 is used to fine-tune the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label.

[0108] In the above technical solution, the pre-trained target model includes a first model deployed on the server and a second model deployed on at least one client. This means that only a portion of the target model (the second model) needs to be disclosed to the clients participating in the federated model tuning, thus protecting server-side model privacy to a certain extent. Furthermore, noise perturbation is applied only to segmented words corresponding to multiple text samples that lack classification utility for the text categories labeled with the classification tags. This reduces perturbation to target segmented words that do have classification utility for the text categories labeled with the classification tags, where the target segmented words have a significant impact on model classification performance. This adaptive perturbation mechanism provides a more subtle trade-off between the usability of the target model and client data privacy, effectively improving the usability of the target model in downstream classification tasks while strengthening client data privacy protection.

[0109] Optionally, the target word segmentation determination module 303 includes:

[0110] The acquisition submodule is used to acquire the category utility words corresponding to the text category labeled by the category label. The category utility words are the K reference words in the vocabulary that have the highest utility importance to the text category labeled by the category label. The vocabulary is used to segment the text sample, and K is greater than 1.

[0111] The first determining submodule is used to determine the segment that belongs to the category utility word among the plurality of segmented words as the target segment.

[0112] Optionally, the classification utility terms are determined by a utility term determination device, which may include:

[0113] The third acquisition module is used to acquire the frequency of each reference word in the text samples under each preset category for each reference word in the vocabulary; the utility importance determination module is used to determine the utility importance of the reference word to the text category labeled by the classification label based on the frequency of the reference word in the text samples under each preset category.

[0114] The classification utility term determination module is used to determine the K reference words in the vocabulary that have the highest utility importance to the text category labeled by the classification tag as the classification utility terms.

[0115] Optionally, the utility importance determination module includes:

[0116] The second determining submodule is used to determine, for each of the multiple preset categories other than the text category marked by the classification label, the ratio of the frequency of the reference word appearing in the text sample under the text category marked by the classification label to the frequency of the reference word appearing in the text sample under the other category.

[0117] The third determining submodule is used to determine the utility importance of the reference word to the text category labeled by the classification tag based on the ratio corresponding to each of the other categories.

[0118] Optionally, the disturbance module 304 includes:

[0119] The processing submodule is used to perform noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments besides the target word;

[0120] The fourth determination submodule is used to determine whether the reference embedding vector corresponding to each reference word in the vocabulary contains each embedding vector for each embedding vector obtained after noise perturbation processing, wherein the vocabulary is used to segment the text sample.

[0121] The fifth determination submodule is used to determine the reference embedding vector that is closest to the embedding vector as the perturbation vector if the reference embedding vector corresponding to each reference word in the vocabulary does not contain the embedding vector.

[0122] Optionally, the fine-tuning module 305 includes:

[0123] The sending submodule is used to send the perturbation vector and the embedding vector corresponding to the target word to the server, so that the server can generate the classification prediction result of the text sample through the first model based on the perturbation vector and the embedding vector corresponding to the target word and send it to the target client;

[0124] The sixth determining submodule is used to determine the first gradient information corresponding to the output layer of the first model based on the classification prediction result of the text sample and the classification label, and send the first gradient information to the server so that the server can update the model parameters based on the first gradient information and the second gradient information corresponding to the output layer sent by the other clients.

[0125] Optionally, the second model of the target client includes at least an embedding block, which includes multiple embedding layers.

[0126] Optionally, the second model of the target client includes an embedding block and at least one encoding module.

[0127] Optionally, both the text sample and the classification label are located locally on the target client.

[0128] It should be noted that the aforementioned utility term determination device can be integrated into the aforementioned model federation fine-tuning device 300, or it can be independent of the aforementioned model federation fine-tuning device 300. This disclosure does not make any specific limitations.

[0129] Figure 5 This is a block diagram illustrating a text classification device according to an exemplary embodiment. Figure 5 As shown, the text classification device 400 includes:

[0130] The second acquisition module 401 is used to acquire the text to be classified.

[0131] The classification module 402 is used to obtain the classification prediction result of the text to be classified by using a pre-tuned target model, wherein the target model is fine-tuned according to the above-mentioned model federated fine-tuning method provided in this disclosure.

[0132] In the above technical solution, the pre-trained target model includes a first model deployed on the server and a second model deployed on at least one client. This means that only a portion of the target model (the second model) needs to be disclosed to the clients participating in the federated model tuning, thus protecting server-side model privacy to a certain extent. Furthermore, noise perturbation is applied only to segmented words corresponding to multiple text samples that lack classification utility for the text categories labeled with the classification tags. This reduces perturbation to target segmented words that do have classification utility for the text categories labeled with the classification tags, where the target segmented words have a significant impact on model classification performance. This adaptive perturbation mechanism provides a more subtle trade-off between the usability of the target model and client data privacy, effectively improving the usability of the target model in downstream classification tasks while strengthening client data privacy protection.

[0133] The following is for reference. Figure 6The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0134] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0135] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0136] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0137] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0138] In some implementations, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0139] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0140] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: It acquires a text sample and a corresponding classification label for the text sample; the pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client; it segments the text sample to obtain multiple segments, and generates embedding vectors corresponding to each of the multiple segments using the second model deployed on the target client; it determines a target segment from the multiple segments that has classification utility for the text category labeled by the classification label; it performs noise perturbation processing on the embedding vectors corresponding to the other segments besides the target segment, obtaining perturbation vectors; and, based on the perturbation vectors, the embedding vectors corresponding to the target segment, and the classification label, it collaborates with the server and other clients in the at least one client to fine-tune the model parameters of the first model.

[0141] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire text to be classified; and, based on the text to be classified, obtain a classification prediction result of the text to be classified using a pre-tuned target model, wherein the target model is obtained by fine-tuning according to the model federated fine-tuning method provided in this disclosure.

[0142] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, the second acquisition module can also be described as a "module for acquiring text to be classified".

[0145] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] According to one or more embodiments of this disclosure, Example 1 provides a model federated fine-tuning method, wherein the pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client, and the method is applied to the target client among the at least one client, including:

[0148] Obtain text samples and their corresponding classification labels;

[0149] The text sample is segmented into multiple words, and the embedding vectors corresponding to each of the multiple words are generated by the second model deployed on the target client.

[0150] From the plurality of word segments, determine the target word segment that has classification effect on the text category labeled by the classification tag;

[0151] The embedding vectors corresponding to the other words in the plurality of word segments, excluding the target word, are subjected to noise perturbation processing to obtain perturbation vectors;

[0152] Based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label, the model parameters of the first model are fine-tuned in collaboration with the server and other clients among the at least one client.

[0153] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein determining the target word from the plurality of word segments that has classification utility for the text category labeled by the classification tag includes:

[0154] Obtain the category utility words corresponding to the text category labeled by the category label, wherein the category utility words are the K reference words in the vocabulary that have the highest utility importance to the text category labeled by the category label, and the vocabulary is used to segment the text sample, and K is greater than 1;

[0155] The segment that belongs to the category of utility words among the multiple segmented words is determined as the target segmented word.

[0156] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein the classification utility terms are determined in the following manner:

[0157] For each reference word in the vocabulary, the frequency of the reference word in text samples under each preset category is obtained; based on the frequency of the reference word in text samples under each preset category, the utility importance of the reference word to the text category labeled by the classification tag is determined.

[0158] The K reference words in the vocabulary that have the highest utility importance for the text category labeled by the classification tag are determined as the classification utility words.

[0159] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, wherein determining the utility importance of the reference word to the text category labeled by the classification tag based on the frequency of the reference word in text samples under each preset category includes:

[0160] For each of the multiple preset categories other than the text category labeled by the classification label, determine the ratio of the frequency of the reference word appearing in the text samples under the text category labeled by the classification label to the frequency of the reference word appearing in the text samples under the other category;

[0161] The utility importance of the reference word to the text category labeled by the classification tag is determined based on the ratio corresponding to each of the other categories.

[0162] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 1, wherein the embedding vectors corresponding to the other words in the plurality of word segments, excluding the target word, are subjected to noise perturbation processing to obtain perturbation vectors, including:

[0163] Noise perturbation processing is applied to the embedding vectors corresponding to the other segmented words besides the target segmented word in the plurality of segmented words;

[0164] For each embedding vector obtained after noise perturbation processing, it is determined whether the embedding vector is contained in the reference embedding vector corresponding to each reference word in the vocabulary, wherein the vocabulary is used to segment the text sample.

[0165] If the reference embedding vector corresponding to each reference word in the vocabulary does not contain the embedding vector, then the reference embedding vector that is closest to the embedding vector is determined as the perturbation vector.

[0166] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 1, wherein the step of fine-tuning the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label, includes:

[0167] The perturbation vector and the embedding vector corresponding to the target word are sent to the server, so that the server can generate the classification prediction result of the text sample through the first model based on the perturbation vector and the embedding vector corresponding to the target word and send it to the target client;

[0168] Based on the classification prediction results of the text samples and the classification labels, the first gradient information corresponding to the output layer of the first model is determined, and the first gradient information is sent to the server so that the server can update the model parameters based on the first gradient information and the second gradient information corresponding to the output layer sent by the other clients.

[0169] According to one or more embodiments of this disclosure, Example 7 provides a method as described in any one of Examples 1-6, wherein the second model of the target client includes at least an embedding block, the embedding block including multiple embedding layers.

[0170] According to one or more embodiments of this disclosure, Example 8 provides the method of Example 7, wherein the second model of the target client includes an embedding block and at least one encoding module.

[0171] According to one or more embodiments of this disclosure, Example 9 provides a method as described in any one of Examples 1-6, wherein both the text sample and the classification label are located locally on the target client.

[0172] According to one or more embodiments of this disclosure, Example 10 provides a text classification method, including:

[0173] Get the text to be categorized;

[0174] Based on the text to be classified, the classification prediction result of the text to be classified is obtained by a pre-tuned target model, wherein the target model is fine-tuned according to any one of Examples 1-9.

[0175] According to one or more embodiments of this disclosure, Example 11 provides a model federated fine-tuning apparatus, wherein a pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client, the apparatus being applied to a target client among the at least one client, comprising:

[0176] The first acquisition module is used to acquire text samples and the classification labels corresponding to the text samples;

[0177] The local computing module is used to segment the text sample to obtain multiple segments, and generate the embedding vectors corresponding to each of the multiple segments through the second model deployed on the target client;

[0178] The target word segmentation determination module is used to determine, from the plurality of word segments, the target word that has classification effect on the text category labeled by the classification tag;

[0179] The perturbation module is used to perform noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments besides the target word to obtain perturbation vectors;

[0180] The fine-tuning module is used to fine-tune the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label.

[0181] According to one or more embodiments of this disclosure, Example 12 provides a text classification apparatus, comprising:

[0182] The second acquisition module is used to acquire the text to be classified.

[0183] The classification module is used to obtain the classification prediction result of the text to be classified by using a pre-tuned target model, wherein the target model is fine-tuned according to any one of Examples 1-9.

[0184] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-10.

[0185] According to one or more embodiments of this disclosure, Example 14 provides an electronic device comprising:

[0186] A storage device on which computer programs are stored;

[0187] A processing device for executing the computer program in the storage device to implement the steps of any one of the methods in Examples 1-10.

[0188] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0189] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0190] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A model federated fine-tuning method, characterized in that, The pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client. The method is applied to the target client among the at least one client, including: Obtain text samples and their corresponding classification labels; The text sample is segmented into multiple words, and the embedding vectors corresponding to each of the multiple segmented words are generated by a second model deployed on the target client. From the plurality of word segments, determine the target word segment that has classification effect on the text category labeled by the classification tag; The embedding vectors corresponding to the other words in the plurality of word segments, excluding the target word, are subjected to noise perturbation processing to obtain perturbation vectors; Based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label, the model parameters of the first model are fine-tuned in collaboration with the server and other clients among the at least one client. The step of determining the target word from the plurality of word segments that has classification utility for the text category labeled by the classification tag includes: Obtain the category utility terms corresponding to the text categories labeled by the category tags; The segment that belongs to the category of utility words among the multiple segmented words is determined as the target segmented word.

2. The method according to claim 1, characterized in that, The classification utility words are the K reference words in the vocabulary that have high utility importance to the text category labeled by the classification tag. The vocabulary is used to segment the text sample, and K is greater than 1.

3. The method according to claim 2, characterized in that, The classification utility terms are determined in the following way: For each reference word in the vocabulary, the frequency of the reference word in text samples under each preset category is obtained; based on the frequency of the reference word in text samples under each preset category, the utility importance of the reference word to the text category labeled by the classification tag is determined. The K reference words in the vocabulary that have the highest utility importance for the text category labeled by the classification tag are determined as the classification utility words.

4. The method according to claim 3, characterized in that, The step of determining the utility importance of the reference word to the text category labeled by the classification tag based on the frequency of the reference word in the text samples under each preset category includes: For each of the multiple preset categories other than the text category labeled by the classification label, determine the ratio of the frequency of the reference word appearing in the text samples under the text category labeled by the classification label to the frequency of the reference word appearing in the text samples under the other category; The utility importance of the reference word to the text category labeled by the classification tag is determined based on the ratio corresponding to each of the other categories.

5. The method according to claim 1, characterized in that, The step of performing noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments besides the target word to obtain perturbation vectors includes: Noise perturbation processing is applied to the embedding vectors corresponding to the other segmented words besides the target segmented word in the plurality of segmented words; For each embedding vector obtained after noise perturbation processing, it is determined whether the embedding vector is contained in the reference embedding vector corresponding to each reference word in the vocabulary, wherein the vocabulary is used to segment the text sample. If the reference embedding vector corresponding to each reference word in the vocabulary does not contain the embedding vector, then the reference embedding vector that is closest to the embedding vector is determined as the perturbation vector.

6. The method according to claim 1, characterized in that, The step of fine-tuning the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label, includes: The perturbation vector and the embedding vector corresponding to the target word are sent to the server, so that the server can generate the classification prediction result of the text sample through the first model based on the perturbation vector and the embedding vector corresponding to the target word and send it to the target client; Based on the classification prediction results of the text samples and the classification labels, the first gradient information corresponding to the output layer of the first model is determined, and the first gradient information is sent to the server so that the server can update the model parameters based on the first gradient information and the second gradient information corresponding to the output layer sent by the other clients.

7. The method according to any one of claims 1-6, characterized in that, The second model of the target client includes at least an embedding block, which includes multiple embedding layers.

8. The method according to claim 7, characterized in that, The second model of the target client includes an embedding block and at least one encoding module.

9. The method according to any one of claims 1-6, characterized in that, Both the text sample and the classification label are located locally on the target client.

10. A text classification method, characterized in that, include: Get the text to be categorized; Based on the text to be classified, a classification prediction result for the text to be classified is obtained through a pre-tuned model, wherein the model is obtained by the model federated fine-tuning method according to any one of claims 1-9.

11. A model federation fine-tuning device, characterized in that, The pre-trained target model includes a first model deployed on a server and a second model deployed on at least one client. The apparatus is applied to the target client among the at least one client, and includes: The first acquisition module is used to acquire text samples and the classification labels corresponding to the text samples; The local computing module is used to segment the text sample to obtain multiple segments, and generate the embedding vectors corresponding to each of the multiple segments through the second model deployed on the target client; The target word segmentation determination module is used to determine, from the plurality of word segments, the target word that has classification effect on the text category labeled by the classification tag; The perturbation module is used to perform noise perturbation processing on the embedding vectors corresponding to the other words in the plurality of word segments besides the target word to obtain perturbation vectors; The fine-tuning module is used to fine-tune the model parameters of the first model in collaboration with the server and other clients among the at least one client, based on the perturbation vector, the embedding vector corresponding to the target word segmentation, and the classification label. The target word segmentation determination module includes: The acquisition submodule is used to acquire the category utility words corresponding to the text categories labeled by the category tags; The first determining submodule is used to determine the segment that belongs to the category utility word among the plurality of segmented words as the target segment.

12. A text classification device, characterized in that, include: The second acquisition module is used to acquire the text to be classified. The classification module is used to obtain the classification prediction result of the text to be classified by using a pre-tuned target model, wherein the target model is obtained by the model federated fine-tuning method according to any one of claims 1-9.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the steps of the method described in any one of claims 1-10.

14. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-10.

Images