A federal text open intention classification method based on particle swarm calculation

Abstract

The application discloses a kind of federal text open intention classification methods based on granular computing, including sequentially completed client training step, server aggregation step, client classification step, and the obtained pre-training language model parameter and local granular ball knowledge base are uploaded to server in client training step;Global model and global granular ball knowledge base are constructed in server aggregation step, and are issued to each client;Global feature is obtained by inputting the sample to be classified into global model in client classification step;And global feature is compared with global granular ball knowledge base, and intention classification is realized.The application constructs granular ball structured knowledge in client to realize efficient representation learning, and aggregation model and knowledge are supported in server side to support global reasoning and open identification suitable for privacy protection scene of multi-client, multi-source heterogeneous text data, under the premise that user data cannot be centrally summarized, still can realize accurate identification to known intention and effective rejection to unknown intention.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric data processing, and in particular to a federated text open intent classification method based on granular ball calculation. BACKGROUND

[0002] In the field of natural language processing, intent classification is one of the key tasks to realize human-computer semantic understanding, and is widely used in intelligent customer service, voice assistants, dialogue robots and other systems. Traditional intent classification methods usually assume that all intent categories in the training set will appear in the reasoning stage, that is, the so-called "closed-world assumption". However, in actual application scenarios, users may at any time propose new types of requests that have not been seen in the training process, thereby causing the model to be unable to correctly identify or misclassify, affecting user experience and system reliability.

[0003] To solve the above problems, researchers have proposed the open intent classification (OIC) task, which aims to identify known intents while detecting unknown intents and rejecting them (i.e., judging as "unknown"). Existing OIC methods can be mainly divided into two categories: one class introduces pseudo-unknown samples in the training stage, which are treated as the "K+1" class to train a multi-classification model; the other class only uses known classes for training, and uses threshold, boundary or distance mechanism to reject those input samples that fall outside the boundary of known classes in the reasoning stage. These methods usually perform well in experimental environments, but are mostly based on centralized training, that is, assuming that all training data can be uniformly stored and accessed.

[0004] However, in real-world applications, user data is often stored in multiple devices or organizations, and centralized data aggregation is not feasible due to privacy protection, data sovereignty, legal regulations and other considerations. For example, in financial, medical, educational and other scenarios, user data owned by different institutions or terminals are isolated from each other and cannot be shared at will. Under this background, the dependence of traditional OIC methods on centralized data seriously limits their actual landing ability.

[0005] Therefore, there is an urgent need for an open intent classification method that can meet the following requirements: (1) compatible with distributed and privacy-limited training scenarios; (2) effectively fuse heterogeneous client knowledge representation; (3) build a controllable and structured open space model to improve the recognition ability of unknown intents. SUMMARY

[0006] The present application aims to overcome the deficiencies of the prior art, and provide a federal text open intent classification method based on granular computing.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] In a first aspect, the present application provides a federal text open intent classification method based on granular computing, comprising sequentially completed client training steps, server aggregation steps, and client classification steps, wherein the client training steps comprise the following sub-steps:

[0009] S11: Extracting sample features of the local training set using a pre-trained language model to obtain a real sample set containing feature vectors of samples and class labels of samples;

[0010] S12: Modeling the real sample set using unsupervised granular ball clustering to obtain a first granular ball set as an approximate expression of class distribution;

[0011] S13: Constructing pseudo-unknown samples based on the first granular ball set, and assigning the class labels of the pseudo-unknown samples to the unknown intent class to obtain a pseudo-unknown sample set;

[0012] S14: Local training of the pre-trained language model using the real sample set and the pseudo-unknown sample set, and uploading the obtained pre-trained language model parameters and the first granular ball set as a local granular ball knowledge base to the server after training;

[0013] The server aggregation steps comprise the following sub-steps:

[0014] S21: Aggregating the pre-trained language model parameters uploaded by each client to construct a global model;

[0015] S22: Aggregating the local granular ball knowledge bases uploaded by each client to construct a global granular ball knowledge base;

[0016] S23: Distributing the global model and the global granular ball knowledge base to each client;

[0017] The client classification steps comprise the following sub-steps:

[0018] S31: Inputting the sample to be classified into the global model to obtain global features;

[0019] S32: Comparing the global features with the global granular ball knowledge base to realize intent classification.

[0020] Further, in step S11, the pre-trained language model is used to extract sample features of the local training set to obtain a real sample set containing feature vectors of samples and class labels of samples, comprising:

[0021] Client The local training set on the client is denoted as ; where denotes the number of local intent text samples of the client , i denotes the i-th sample, denotes the i-th intent text sample of the client , and denotes the class label of the i-th intent text sample of the client .

[0022] The features of the i-th intent text sample of the client are extracted using the pre-trained language model, to obtain the feature vector of the i-th intent text sample of the client , and further obtain the real sample set .

[0023] Further, in step S12, the first set of granules approximating the class distribution is obtained, and the j-th granule has the following attributes: granule size : the number of samples contained in the granule; granule centroid : the mean of all sample features in the granule; granule mean radius : the average Euclidean distance of all samples in the granule to the centroid; granule maximum radius : the maximum Euclidean distance of all samples in the granule to the centroid; granule radius: the radius of the granule; granule label : the label of the highest proportion category in the granule; granule purity : the proportion of samples in the granule that belong to the granule label .

[0024] The real sample set is modeled using unsupervised granule clustering to obtain the first set of granules approximating the class distribution, including:

[0025] If the purity of the l-th granule is lower than a set threshold and the number of samples exceeds a minimum threshold , a division operation is triggered: for the granule , all label categories appearing in it are counted ; the granule is divided into several sub-granules according to the number of labels , and each sub-granule takes one sample in the label as a pseudo centroid; the samples in the original granule are assigned to the sub-granule represented by the nearest pseudo centroid; the granule purity and the granule size of each sub-granule are re-evaluated, and if the termination condition is met, the sub-granule is retained, otherwise the recursive division continues;

[0026] Finally, all the granules that meet the conditions are collected The known class structure on the client The granule centroid and the granule radius constitute the granule prototype of each class, which is used to represent the class distribution approximation under privacy protection.

[0027] Further, in step S13, pseudo-unknown samples are constructed based on the first granule set, and the class labels of the pseudo-unknown samples are uniformly assigned as unknown intent classes, obtaining a pseudo-unknown sample set, including:

[0028] S131: candidate sample initialization, including:

[0029] For any two granules with different labels in the first granule set , the midpoint of the centroid of the two is calculated as the initial representation of the pseudo-unknown sample:

[0030] ;

[0031] And all the above midpoints are used as candidate samples to form an initial candidate set;

[0032] S132: genetic evolution and diversity enhancement, including:

[0033] Retain the candidate sample with the highest fitness; perform cross operation to generate new samples; add Gaussian noise to the candidate samples to simulate mutation:

[0034] ;

[0035] In the formula, t represents the evolution number of the current genetic algorithm, represents a multi-dimensional Gaussian distribution with a mean vector of zero and a covariance matrix of σ 2 times the unit matrix I, is the evolution process result obtained by continuously applying cross, selection and mutation operations of adding Gaussian noise to the initial candidate and and the offspring in the t-th generation, that is, is the starting point of the 0th generation, is the representation of the offspring in the t-th generation;

[0036] Each generation of candidate samples is evaluated by the following fitness function:

[0037] ;

[0038] The farther away from all known class prototypes, the higher the score, and the more likely to represent the real unknown intent; In the formula, represents the result after evolution, is the result of the evolution process in the t-th generation, This represents the number of spheres corresponding to all known classes on client c. The fitness function is expressed as the ratio of the candidate sample to the centroid of all spheres. The minimum distance in;

[0039] S133: Filtering and label assignment, including:

[0040] Retain all fitness values ​​above the threshold Pseudo-samples that meet the following conditions:

[0041] ;

[0042] That is, the spurious sample must be outside the coverage area of ​​all spheres;

[0043] Finally, a predetermined proportion is selected from the retained pseudo-samples based on their fitness ranking. The pseudo-samples are constructed as pseudo-unknown samples; and the category labels of the pseudo-unknown samples are uniformly assigned to the class representing unknown intent. They will participate in subsequent training.

[0044] Further, in step S14, the pre-trained language model is trained locally using the real sample set and the pseudo-unknown sample set. After training, the obtained pre-trained language model parameters and the local particle sphere knowledge base are uploaded to the server, including:

[0045] S141: Establish the basic classification loss, including:

[0046] For the client The known class of sample set is the real sample set. With the set of pseudo-unknown samples Supervised learning is performed using standard cross-entropy loss:

[0047] ;

[0048] in This represents the class probability distribution output by the pre-trained language model, totaling... There are 10 categories, among which Indicates an unknown class;

[0049] S142: Establish granular structure constraint loss, including:

[0050] To enhance the intra-class compactness of known class samples, a granular structure loss is introduced. The definition is as follows:

[0051] ;

[0052] in To be consistent with the sample The closest granule centroid:

[0053] ;

[0054] S143: Establish an adaptive weighting mechanism and total loss function, including:

[0055] Introducing a dynamic weight factor , according to the uncertainty of the pre-trained language model to the unknown class discrimination adaptive adjustment:

[0056] ;

[0057] The final total loss function is defined as:

[0058] ;

[0059] S144: Use the total loss function to locally train the pre-trained language model, and after training, the obtained pre-trained language model parameters and the local granule knowledge base Upload to the server.

[0060] Further, in step S21, the pre-trained language model parameters uploaded by each client are aggregated to construct a global model, including:

[0061] After each round of local training, the client Uploads the pre-trained language model parameters , aggregated by weighted average method, to construct a global model:

[0062] ;

[0063] Where is the number of samples of the client , is the total number of samples, and C is the number of clients.

[0064] Further, in step S22, the local granule knowledge base uploaded by each client is aggregated to construct a global granule knowledge base, including:

[0065] S221: For the public class , if two granules from different clients satisfy:

[0066] ;

[0067] It is considered to be overlapped and needs to be merged into a new granule:

[0068] ;​

[0069] ;

[0070] Not satisfied, directly retain the corresponding granules; the merged granules constitute a global public knowledge base ;

[0071] S222: for private classes , retain the granules of each client as they are and directly build ;

[0072] S223: finally build a global granule knowledge base:

[0073] .

[0074] Further, in step S31, the sample to be classified is input into the global model to obtain global features ; ;

[0075] In step S32, the global features are compared with the global granule knowledge base to realize intent classification, including:

[0076] Traverse all granules , judge whether the global features satisfy:

[0077] ;

[0078] If fall into at least one granule, assign to the label corresponding to the nearest granule ; if it does not satisfy any granule coverage condition, the sample is determined as an unknown intent class.

[0079] The beneficial effects of the present application are:

[0080] In an exemplary embodiment of the present application, granule structured knowledge is constructed on the client side to realize efficient representation learning, and the model and knowledge are aggregated on the server side to support global reasoning and open recognition. This exemplary embodiment is particularly suitable for privacy protection scenarios of multi-client, multi-source heterogeneous text data. Under the premise that user data cannot be centrally aggregated, accurate recognition of known intents and effective rejection of unknown intents can still be realized. This method can be widely applied to dialog artificial intelligence systems such as intelligent customer service systems, voice assistants, financial dialogue systems, and medical question and answer platforms, to improve their intent understanding ability and security in an open environment. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1This is a flowchart of a federated text open intent classification method based on particle sphere computation provided in an exemplary embodiment of the present invention. Detailed Implementation

[0082] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0084] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0085] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0086] See Figure 1 , Figure 1 The diagram illustrates a flowchart of a federated text open intent classification method based on particle sphere computation provided in an exemplary embodiment of the present invention, including a client-side training step, a server-side aggregation step, and a client-side classification step performed sequentially. The client-side training step includes the following sub-steps:

[0087] S11: Use a pre-trained language model to extract sample features from the local training set to obtain a real sample set containing the feature vectors and class labels of the samples;

[0088] S12: Model the real sample set using unsupervised sphere clustering to obtain the first sphere set as an approximate representation of the category distribution;

[0089] S13: Construct pseudo-unknown samples based on the first granular ball set, and uniformly assign the class labels of the pseudo-unknown samples as unknown intent classes to obtain a pseudo-unknown sample set;

[0090] S14: Perform local training on the pre-trained language model using the real sample set and the pseudo-unknown sample set, and upload the obtained pre-trained language model parameters and the local granular ball knowledge base as the first granular ball set to the server after the training is completed;

[0091] The server aggregation step includes the following sub-steps:

[0092] S21: Aggregate the pre-trained language model parameters uploaded by each client to obtain a global model;

[0093] S22: Aggregate the local granular ball knowledge bases uploaded by each client to obtain a global granular ball knowledge base;

[0094] S23: Distribute the global model and the global granular ball knowledge base to each client;

[0095] The client classification step includes the following sub-steps:

[0096] S31: Input the sample to be classified into the global model to obtain global features;

[0097] S32: Compare the global features with the global granular ball knowledge base to realize intent classification.

[0098] Specifically, in the present exemplary embodiment:

[0099] (1) First, in the client training step:

[0100] (1-1) Model the known class distribution using granular balls to achieve structured local representation: On the client side, the present exemplary embodiment abstracts the representation of known classes into a number of granular balls (Granular-Ball) with centers and radii by introducing the granular ball computing idea. This structure can effectively capture the local class distribution features, achieve structured and irreversible information sharing without revealing the original samples, and provide high-quality structure units for subsequent knowledge aggregation and decision-making.

[0101] (1-2) Combine granular ball structure to generate pseudo unknown samples, and improve open space modeling capability: The generation of traditional pseudo unknown samples often relies on shallow disturbance or random reconstruction, lacks semantic or spatial guidance, and is easy to be confused with known classes. In the example embodiment, in order to simulate the feature distribution of unknown intent, a pseudo unknown sample generation mechanism based on granular ball structure is proposed, which fully utilizes the granular ball structure information of known categories, effectively simulates unknown intent, and improves the discrimination ability of the model to unknown categories and the generalization ability in an open environment.

[0102] (1-3) Then, the pre-trained language model is locally trained using the real sample set and the pseudo unknown sample set, and after training, the pre-trained language model parameters and the local granular ball knowledge base are uploaded to the server.

[0103] (2) In the server aggregation step:

[0104] A granular ball knowledge aggregation mechanism is proposed: In the federated scenario, in order to realize open classification discrimination from a global perspective, the example embodiment proposes a server module containing model parameter aggregation and granular ball structure aggregation. The module is responsible for fusing the granular ball knowledge uploaded by each client, constructing a global class boundary, and performing non-parametric inference accordingly. The granular ball structured knowledge is constructed on the client side to realize efficient representation learning, and the model and knowledge are aggregated on the server side to support global inference and open recognition.

[0105] (3) In the client classification step:

[0106] Non-parametric inference is performed using the global granular ball boundary to realize open intent classification: This mechanism defines the class boundary through multi-granularity granular ball structure, realizes the open discrimination ability at the sample level without additional classifiers, and effectively improves the unknown intent detection performance.

[0107] In summary, the example embodiment is especially suitable for privacy protection scenarios of multi-client and multi-source heterogeneous text data. Under the premise that user data cannot be centrally aggregated, accurate identification of known intent and effective rejection of unknown intent can still be realized. This method can be widely applied to dialog artificial intelligence systems such as intelligent customer service systems, voice assistants, financial dialogue systems, and medical question and answer platforms, to improve their intent understanding ability and security in an open environment.

[0108] It should be noted that the pre-training language model preferably adopts a shared global encoder (12-layer BERT-Base Transformer) + a local classification head; and the global model = only contains the same set of encoder weights, without a decoder and a classification head, and is continuously synchronized through federated averaging. There is no decoder because there is no representation transmission. Moreover, the pre-training language models of the respective clients are of the same type before training, i.e., initially, they are all pre-trained Bert models.

[0109] The following will illustrate the specific implementation of each exemplary embodiment:

[0110] Steps S11 and S12 are client granule ball modeling. In order to improve the structural expression capability of distribution modeling and take into account user privacy, the application designs a known class representation method based on granule balls, which is used in the local modeling stage of the client. The granule ball structure has the expression capability of distribution center and coverage range, and can effectively model the class distribution without revealing the original data.

[0111] More preferably, in an exemplary embodiment, in step S11, a pre-training language model is used to extract the sample features of the local training set, to obtain a real sample set containing the feature vectors of the samples and the class labels of the samples, including:

[0112] The local training set on the client is denoted as ; wherein represents the number of local intent text samples of the client , i represents the i-th sample, represents the i-th intent text sample of the client , and represents the class label of the i-th intent text sample of the client .

[0113] The features of are extracted using a pre-training language model to obtain the feature vector of the i-th intent text sample of the client ; and further to obtain a real sample set .

[0114] More preferably, in an exemplary embodiment, in step S12, the first granule ball set as an approximate expression of the class distribution has the following attributes: granule ball size : the number of samples contained in the granule ball; granule ball centroid : the mean of all sample features in the granule ball; granule ball mean radius : the average Euclidean distance of all samples in the granule ball to the centroid; and granule ball maximum radius : The maximum Euclidean distance from all samples within the sphere to the centroid; sphere radius; sphere label The label representing the category with the highest percentage of granules; granule purity. : The contents of the pellet belong to the pellet label The sample proportion;

[0115] The real sample set is modeled using unsupervised granular clustering to obtain a first set of granules that approximates the category distribution, including:

[0116] If the purity of the l-th particle Below the set threshold And sample size Exceeding the minimum threshold Then the partitioning operation is triggered: for spheres Count all the types of tags that appear in it. ; The pellets were sorted by label number Divide the original particle into several sub-spheres, with a sample within the label as the pseudo-centroid of each sub-sphere; assign the sample in the original particle to the sub-sphere represented by the nearest pseudo-centroid; re-evaluate the particle purity and particle size of each sub-sphere, and retain it if the termination condition is met, otherwise continue the recursive division.

[0117] Ultimately, the set of all spheres that satisfy the conditions. Will be used to represent the client Based on the known class structure, the centroid and radius of the sphere constitute the prototype of the sphere for each class, serving as an approximate expression of the class distribution under privacy protection.

[0118] Specifically, in this exemplary embodiment, a granular-ball is an adaptive clustering unit capable of characterizing data distribution at multiple granular levels. Let the real sample set be... ,in For the feature vector of the sample, This represents the category label of the sample. By clustering this sample set, a set of spheres can be obtained. Each of the granules Depend on It consists of 10 samples, with attributes as described above.

[0119] To enhance the structural expressiveness of distribution modeling while protecting user privacy, this exemplary embodiment designs a known class representation method based on granular spheres for the local modeling stage on the client side. The granular sphere structure simultaneously possesses the expressive power of both the distribution center and coverage area, enabling effective modeling of class distributions without revealing the original data.

[0120] After obtaining the real sample set as the feature set in step S11, step S12 models the feature set based on an unsupervised granular ball clustering method. The clustering process generates multiple granular ball units using a recursive division method with the entire feature set as the initial granular ball, and the specific strategy is as described above.

[0121] More preferably, in an exemplary embodiment, in order to simulate the feature distribution of unknown intentions, a pseudo-unknown sample generation mechanism based on the granular ball structure is proposed to enhance the open space modeling capability. Specifically, in step S13, pseudo-unknown samples are constructed based on the first granular ball set, and the class labels of the pseudo-unknown samples are uniformly assigned as unknown intention classes to obtain a pseudo-unknown sample set, including:

[0122] S131: candidate sample initialization, including:

[0123] For any two granular balls with different labels in the first granular ball set , the midpoint of the centroid of the two is calculated as the initial representation of the pseudo-unknown sample:

[0124] ;

[0125] and all the above midpoints are used as candidate samples to form an initial candidate set;

[0126] S132: genetic evolution and diversity enhancement, including:

[0127] To improve the diversity of pseudo-unknown samples and avoid the concentration of samples in a fixed area, a genetic algorithm is introduced to evolve the candidate set. The specific process includes: retaining the candidate sample with the highest fitness; performing a crossover operation to generate new samples; adding Gaussian noise to the candidate samples to simulate mutation:

[0128] ;

[0129] where t represents the evolution number of the current genetic algorithm, represents a multi-dimensional Gaussian distribution with a mean vector of zero and a covariance matrix of σ 2 multiplied by the identity matrix I, is the evolution process result obtained by continuously applying crossover, selection, and mutation operations of adding Gaussian noise to the initial candidate and and the offspring in the t-th generation, that is, is the starting point of the 0-th generation, is the representation of the offspring in the t-th generation;

[0130] Each generation of candidate samples is evaluated for fitness by the following fitness function:

[0131] ;

[0132] The further a sample is from all known class prototypes, the higher its score, and the more likely it is to represent the true unknown intent; in the formula, represents the result after evolution, It is during the process of evolution, This represents the number of spheres corresponding to all known classes on client c. The fitness function is expressed as the ratio of the candidate sample to the centroid of all spheres. The minimum distance in;

[0133] S133: Filtering and label assignment, including:

[0134] Retain all fitness values ​​above the threshold Pseudo-samples that meet the following conditions:

[0135] ;

[0136] That is, the spurious sample must be outside the coverage area of ​​all spheres;

[0137] Finally, a predetermined proportion is selected from the retained pseudo-samples based on their fitness ranking. The pseudo-sample (e.g., 0.5) is constructed as a pseudo-unknown sample; and the category label of the pseudo-unknown sample is uniformly assigned to the class representing unknown intent. They will participate in subsequent training.

[0138] Specifically, in the pseudo-unknown sample generation in this exemplary embodiment, the geometric relationship between particles is fully utilized to generate initial candidates based on the midpoint between the centroids of particles of different categories. Combined with the genetic algorithm for evolutionary search and Gaussian perturbation, and with the distance-based fitness function and boundary exclusion strategy, the position and diversity of pseudo samples are effectively controlled, ensuring that they are located in the open space and are representative. This achieves effective simulation of unknown intentions, improves the model's ability to distinguish unknown categories and its generalization ability in open environments.

[0139] More preferably, in an exemplary embodiment, to uniformly utilize real samples and pseudo-unknown samples while simultaneously improving discriminative and structural representation capabilities, this exemplary embodiment proposes a particle-guided training objective, including: a basic classification loss, a particle structure constraint loss, and an adaptive weighting mechanism. Specifically, in step S14, the pre-trained language model is locally trained using the real sample set and the pseudo-unknown sample set. After training, the obtained pre-trained language model parameters and the local particle knowledge base are uploaded to the server, including:

[0140] S141: Establish the basic classification loss, including:

[0141] For the client The known class of sample set is the real sample set. With pseudo unknown sample set , supervised learning is performed using a standard cross-entropy loss:

[0142] ;

[0143] wherein represents the category probability distribution output by the pre-trained language model, and there are categories, wherein represents an unknown class;

[0144] S142: Establish a granular ball structure constraint loss, including:

[0145] To enhance the intra-class compactness of known class samples, a granular ball structure loss is introduced to encourage samples to move closer to the nearest granular ball centroid, which is defined as follows:

[0146] ;

[0147] wherein is the nearest granular ball centroid to the sample :

[0148] ;

[0149] This loss is only applicable to real known class samples, and pseudo unknown samples do not participate in this structure constraint;

[0150] S143: Establish an adaptive weighting mechanism and a total loss function, including:

[0151] To achieve a dynamic balance between known class compression and unknown class separation, a dynamic weight factor is introduced to adaptively adjust according to the uncertainty of the pre-trained language model in identifying unknown classes:

[0152] ;

[0153] The final total loss function is defined as:

[0154] ;

[0155] S144: Use the total loss function to locally train the pre-trained language model, and after the training is completed, the obtained pre-trained language model parameters and the local granular ball knowledge base are uploaded to the server.

[0156] Specifically, in the present exemplary embodiment, through the training mechanism, each client not only obtains the pre-trained language model, but also constructs a structured knowledge base containing intra-class granules, which can be used for server aggregation and inference in the future. And design structure-aware granule supervision loss to strengthen intra-class compactness and inter-class separation: the present exemplary embodiment constructs a joint loss function in the training stage, on the one hand, based on the existing label for standard classification training, on the other hand, introduce granule supervision term, encourage the same class samples to move to their nearest granule centroid, so as to improve the intra-class compactness and discriminability. At the same time, in order to balance the trade-off between known classification and unknown detection tasks, a dynamic adjustment coefficient is introduced to adaptively adjust the strength of structure supervision according to the confidence level of the model on unknown samples.

[0157] In the federated scenario, in order to realize the open classification discrimination of global perspective, the following two exemplary embodiments propose a server module containing model parameter aggregation and granule structure aggregation (step S21 and step S22). This module is responsible for fusing the granule knowledge uploaded by each client, constructing the global class boundary, and performing non-parametric inference accordingly.

[0158] More preferably, in an exemplary embodiment, in step S21, the pre-trained language model parameters uploaded by each client are aggregated to construct a global model, including:

[0159] After each round of local training, the client uploads the pre-trained language model parameters to the server , and aggregates them by weighted average method to construct a global model:

[0160] ;

[0161] Where is the number of samples of the client , is the total number of samples, and C is the number of clients.

[0162] In step S22, in order to construct the global class distribution representation, the local granule knowledge base uploaded by each client is aggregated to construct a global granule knowledge base, including:

[0163] S221: For the public class , if two granules from different clients satisfy:

[0164] ;

[0165] , they are considered to overlap and need to be merged into a new granule:

[0166] ; ​

[0167] ;

[0168] If not, directly retain the corresponding granule; the merged granules constitute a global public knowledge base ;

[0169] S222: For private classes , retain the granules of each client as they are and directly build ;

[0170] S223: Finally, build a global granule knowledge base:

[0171] .

[0172] Specifically, in the present exemplary embodiment, the structure provides a cross-client class structure view while ensuring data privacy, which can be directly used for open intent judgment in the reasoning phase.

[0173] More specifically, the present exemplary embodiment not only adopts the classic federated average algorithm (FedAvg) to fuse the model parameters uploaded by the clients, but also designs a granule aggregation strategy based on geometric overlap determination. The public class granules are spatially merged to improve coverage and consistency, and the private class granules are directly retained for their local structural features. Finally, a global granule knowledge base is constructed, which can cover the class distribution of all clients and improve the generalization ability of the model.

[0174] After obtaining the global model and granule knowledge (steps S21 and S22), the following exemplary embodiment proposes a non-parametric reasoning mechanism for identifying whether a test sample belongs to a known class:

[0175] More preferably, in an exemplary embodiment, in step S31, the sample to be classified is input into the global model to obtain global features ;

[0176] In step S32, the global features are compared with the global granule knowledge base to realize intent classification, including:

[0177] Traverse all granules , and determine whether the global features satisfy:

[0178] ;

[0179] If falls into at least one granule, it is assigned to the label corresponding to the nearest granule If none of the granulocyte coverage conditions is satisfied, the sample is determined as unknown intent class:

[0180] .

[0181] This mechanism defines the class boundary through the multi-granularity granulocyte structure, realizes the open discrimination ability at the sample level without additional classifiers, and effectively improves the unknown intent detection performance.

[0182] The method proposed based on the above preferred exemplary embodiments has innovation in structure modeling, knowledge aggregation and reasoning strategy, significantly enhances the practicability and stability of the model in real scenarios such as multi-client, privacy protection and inconsistent distribution between classes, has wide application value, and is especially suitable for natural language understanding systems with high privacy requirements and heterogeneous data distribution, such as finance, medical treatment and intelligent customer service.

[0183] Obviously, the above embodiments are only examples for clear illustration, and are not limitations to the embodiments. For ordinary skilled in the art, other different forms of changes or variations can be made on the basis of the above description. Here, all the embodiments are not enumerated, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A federated open intent classification method based on granular computing, characterized in that: The method comprises sequentially completed client training steps, server aggregation steps and client classification steps, and the client training steps comprise the following sub-steps: S11: extracting sample features of a local training set by using a pre-trained language model to obtain a real sample set containing feature vectors of samples and category labels of the samples; S12: modeling the real sample set in an unsupervised granular ball clustering manner to obtain a first granular ball set as an approximate expression of a category distribution; S13: constructing pseudo-unknown samples based on the first granular ball set, and uniformly assigning category labels of the pseudo-unknown samples to an unknown intent category to obtain a pseudo-unknown sample set; S14: locally training the pre-trained language model by using the real sample set and the pseudo-unknown sample set, and uploading pre-trained language model parameters obtained after the training and the first granular ball set as a local granular ball knowledge base to a server; The server aggregation steps comprise the following sub-steps: S21: aggregating pre-trained language model parameters uploaded by each client to construct a global model; S22: aggregating local granular ball knowledge bases uploaded by each client to construct a global granular ball knowledge base; S23: distributing the global model and the global granular ball knowledge base to each client; The client classification steps comprise the following sub-steps: S31: inputting a sample to be classified into the global model to obtain global features; S32: comparing the global features with the global granular ball knowledge base to realize intent classification; In step S13, pseudo-unknown samples are constructed based on the first granular ball set, and category labels of the pseudo-unknown samples are uniformly assigned to an unknown intent category to obtain a pseudo-unknown sample set, comprising: S131: candidate sample initialization, comprising: For any two pairs of particles in the first set of particles with different labels , compute the midpoint of their centroids as the initial representation of the pseudo-unknown sample: ； and all the above midpoints are taken as candidate samples to form an initial candidate set; S132: genetic evolution and diversity enhancement, comprising: keeping the current highest fitness candidate sample; performing a crossover operation to generate a new sample; adding Gaussian noise to the candidate sample with simulated mutation: ； where t denotes the evolution generation of the current genetic algorithm, denotes a zero-mean vector and a covariance matrix of 2 a multi-dimensional Gaussian distribution multiplied by the identity matrix I, is the initial candidate at the tth generation, and the offspring after continuously applying the crossover, selection and mutation operations with the addition of Gaussian noise, i.e. is the starting point of the 0th generation, is the offspring representation of the tth generation; Each generation of candidate samples is evaluated in fitness by using the following fitness function: ； The farther away from all known class prototypes, the higher the score, more likely to represent the real unknown intent; In the formula, Represent the results after evolution, Is in the process of evolution, The number of granules corresponding to all known classes on the client c, The fitness function of is expressed as the minimum distance between the candidate sample and the centroid of all granules In the formula, S133: screening and label assignment, comprising: retaining all individuals with fitness above a threshold and the following condition is met: ； That is, the pseudo-sample must be outside the coverage range of all granular balls; Finally, the preset proportion of the reserved pseudo samples is selected according to the fitness ranking The pseudo samples are constructed as pseudo unknown samples; and the class labels of the pseudo unknown samples are uniformly assigned as , which participates in subsequent training.

2. The granular computing based federated open-class text intent classification method according to claim 1, wherein: In step S11, sample features of a local training set are extracted by using a pre-trained language model to obtain a real sample set containing feature vectors of samples and category labels of the samples, comprising: Client , is denoted as ; where denotes the number of local intent text samples of the client , i denotes the i-th sample, denotes the i-th intent text sample of the client , and denotes the class label of the i-th intent text sample of the client . Extracting using pre-trained language models The characteristics of the client are obtained. The feature vector of the i-th intentional text sample And further obtain the real sample set. .

3. The federated text open intent classification method based on granule computation according to claim 2, characterized in that: In step S12, the first set of blobs as a class distribution approximation expression , the jth blob has the following attributes: blob size : the number of samples contained in the blob; blob centroid : the mean of all sample features within the blob; blob mean radius : the average Euclidean distance of all samples within the blob to the centroid; blob max radius : the maximum Euclidean distance of all samples within the blob to the centroid; blob radius; blob label : the label of the most prevalent class in the blob; blob purity : the proportion of samples within the blob that belong to the blob label ; The real sample set is modeled in an unsupervised granular ball clustering manner to obtain a first granular ball set as an approximate expression of a category distribution, comprising: If the purity of the l-th particle Below the set threshold And sample size Exceeding the minimum threshold Then the partitioning operation is triggered: for spheres Count all the types of tags that appear in it. ; The pellets were sorted by label number Divide the original particle into several sub-spheres, with a sample within the label as the pseudo-centroid of each sub-sphere; assign the sample in the original particle to the sub-sphere represented by the nearest pseudo-centroid; re-evaluate the particle purity and particle size of each sub-sphere, and retain it if the termination condition is met, otherwise continue the recursive division. Finally, all the sets of granules that meet the conditions The known class structure on the client The granule centroid and the granule radius constitute the granule prototype of each class, as an approximate expression of the class distribution under privacy protection.

4. The method of claim 1, wherein the method is a granular computing based federated open-class intent classification method. In step S14, a pre-trained language model is locally trained by using a real sample set and a pseudo-unknown sample set, and pre-trained language model parameters obtained after the training and a local granular ball knowledge base are uploaded to a server, comprising: S141: establishing a basic classification loss, comprising: For the client The known class of sample set is the real sample set. With the set of pseudo-unknown samples Supervised learning is performed using standard cross-entropy loss: ； wherein represents a class probability distribution output by the pre-trained language model, over classes, wherein represents an unknown class; S142: establishing a granular ball structure constraint loss, comprising: To enhance the intra-class compactness of known class samples, a granular structure loss is introduced defined as follows: ； wherein is the distance between the sample closest granulocyte centroid: ； S143: establishing an adaptive weighting mechanism and a total loss function, comprising: Introducing dynamic weight factors Adaptive adjustment according to the uncertainty of the pre-trained language model in discriminating unknown classes: ； The final total loss function is defined as: ； S144: locally train the pre-training language model using the total loss function, and obtain pre-training language model parameters after the training is completed and the local granulocyte knowledge base upload to the server.

5. The federated open-class intent classification method based on granule computation according to claim 4, characterized in that: In step S21, pre-trained language model parameters uploaded by each client are aggregated to construct a global model, comprising: After each round of local training is completed, the client uploading the pre-training language model parameters to the server , and a global model is constructed by aggregating the local models through a weighted average method: ； wherein the number of clients the number of samples, the total number of samples, C is the number of clients.

6. The federated open-class intent classification method based on granule computation according to claim 5, characterized in that: In step S22, local granular ball knowledge bases uploaded by each client are aggregated to construct a global granular ball knowledge base, comprising: S221: For public classes , if two granules from different clients and satisfy: ； If the two granular balls are considered to be overlapped, they need to be merged into a new granular ball: ； ； If not, the corresponding granule is directly reserved; the combined granules constitute a global public knowledge base ; S222: For private classes , the granules of each client are kept as they are and are built directly ; S223: Finally, build the global granulocyte knowledge base: 。 7. The federated open-class intent classification method based on granule computation according to claim 6, characterized in that: In step S31, the sample to be classified is input into the global model to obtain global features ;​ In step S32, the global features are compared with the global granulocyte knowledge base to achieve intent classification, including: traversing all granules , determining global features whether the global features satisfy: ； If falls into at least one granule, assign to the label corresponding to the nearest granule ; if none of the granule coverage conditions are satisfied, then the sample is classified as unknown intent class.

Images