A hierarchical label text classification method and system based on a MOE model

Abstract

This invention proposes a hierarchical label text classification method based on the MOE model, comprising the following steps: encoding the input text using a shared encoder to obtain a unified vector representation of the text; inputting the text vector representation into a router module to calculate the probability distribution of the first-level labels, and activating the top-k expert modules corresponding to the first-level labels; the activated expert modules performing classification predictions within their respective second-level label subspaces; and weighted fusion of the prediction results from the expert modules to obtain the final hierarchical label classification result. This invention extracts text semantic features uniformly through a shared encoder, avoiding redundant learning across branches; it allows samples to simultaneously activate multiple candidate experts through a Top-k soft routing mechanism, breaking the rigid single-path limitation of traditional top-down methods; and it combines routing weights with expert outputs, using the maximization of joint probability as the decision criterion, significantly reducing the number of model parameters and the risk of error cascading while ensuring classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a hierarchical labeled text classification method and system based on the MOE model. Background Technology

[0002] Hierarchical Text Classification (HTC) is an important branch of natural language processing. In practical applications such as enterprise content governance, public opinion management, knowledge base entry archiving, automatic distribution of work orders / emails, and policy / official document subject classification, text often needs to be classified according to a hierarchical tagging system. For example, first determine which business domain or category the text belongs to (first-level tag), and then subdivide it into specific topics or subcategories under that category (second-level tags or deeper levels).

[0003] Its formal definition is: given text x The classification results need to be output within a predefined hierarchical label system. Let the first-level label set be: The set of secondary tags is divided into several subsets based on the primary tags: ,in Indicates a first-level tag The corresponding number of secondary labels. The output of the classification task is the prediction result of the primary label. And in Prediction results in the corresponding second-level subspace That is, output label pairs .

[0004] Hierarchical labels are more in line with actual business organization methods than flat labels, but they also bring engineering problems such as error propagation in the decision chain, significant long-tail distribution, and complex model maintenance.

[0005] Currently, for hierarchical text classification tasks, the industry generally adopts a top-down classification method using the Local Strategy. This method trains one or more classifiers for each parent node in the hierarchical structure. During prediction, it first selects the first-level label at the top level, and then proceeds to the corresponding branch for subclass prediction. The two-level task can be formalized as follows: ; ; While this type of method is intuitive and easy to break down into subtasks, it exposes the following significant shortcomings in real-world engineering and long-tail scenarios: 1) Irreversible error propagation: In a top-down chain decision-making process, once the top-level (first-level label) judgment is wrong, subsequent steps will inevitably enter the wrong branch. The prediction space of the second-level and deeper levels is limited by the error, resulting in end-to-end error amplification that is difficult to correct. This problem is particularly prominent when the boundaries of the first-level labels are ambiguous or semantically overlapping. 2) Redundant Parameters and Inefficient Training and Maintenance: Local classifiers are typically trained, deployed, or maintained independently. Different branches repeatedly learn language knowledge and domain semantics, leading to an overall increase in parameter size. Furthermore, model version management is complex, training and parameter tuning are costly, and it is difficult to achieve a unified end-to-end optimization objective. 3) Rigid computational path and insufficient representation of fuzzy samples (Hard Assignment): Most local strategies use hard partitioning, and samples are forced to be assigned to a single parent class branch, which makes it difficult to express the semantic uncertainty between two parent classes, resulting in insufficient robustness to cross-domain, mixed topic or boundary samples. 4) Difficulties in engineering implementation: Local strategies still face challenges in engineering implementation, such as underfitting of small classes due to the long-tail distribution of hierarchical labels, high maintenance costs of multi-model linkage when the label system evolves, and difficulty in locating errors in the serial link.

[0006] Therefore, in order to overcome the shortcomings and limitations of existing hierarchical text classification technologies, it is urgent to propose a hierarchical labeled text classification method and system based on a hybrid expert model (MOE). Summary of the Invention

[0007] This invention proposes a hierarchical label text classification method and system based on the MOE model, which solves the problems of irreversible error propagation, parameter redundancy and inefficient training and maintenance, rigid computational path and insufficient representation of fuzzy samples in existing hierarchical text classification in real engineering and long-tail scenarios.

[0008] The technical solution of this invention is implemented as follows: The first aspect of this invention provides a hierarchical labeled text classification method based on the MOE model, comprising the following steps: S1, Text Encoding: Encode the input text using a shared encoder to obtain a unified vector representation of the text; S2, Route Allocation: Input the text vector representation into the router module, calculate the allocation probability distribution of the first-level labels, and select the expert modules corresponding to the top-k first-level labels for activation based on the probability distribution; S3, Expert Prediction: Each activated expert module performs classification prediction in its corresponding secondary label subspace and outputs the conditional probability distribution of the secondary labels; S4, Probabilistic Fusion and Output: The prediction results of the Top-k activation expert modules are weighted and fused to obtain the final hierarchical label classification result. The weights of the weighted fusion are determined by the first-level label allocation probability output by the router module.

[0009] Specifically, in step S1, before encoding the input text, the input text is preprocessed. The preprocessing includes one or more of the following: word segmentation, removal of special characters, truncation or padding to a fixed length.

[0010] Furthermore, in step S1, a pre-trained language model is used as a shared encoder to process the input text. x Convert to dense vector representation: ; Where h is text x The vector representation of ; Enc represents the shared encoder.

[0011] Specifically, in step S2, the router module calculates the probability distribution of text belonging to each first-level label through linear transformation and the Softmax activation function. The calculation formula is as follows: ; in, Representing text x Belongs to each first-level tag The probability distribution vector, Here is the weight matrix of the router. Let h be the bias vector and h be the text. x The vector representation of .

[0012] Further, in step S2, based on the probability distribution of the first-level labels, the expert modules corresponding to the k labels with the highest probabilities are selected for activation, and the probability of each selected expert module is normalized, as shown in the following formula: ; in, This indicates that the selected number after normalization i The probability of each expert module. , These represent the selected numbers before normalization. i The and the first j The probability of the expert module; This is the set of indices for the selected Top-k expert modules.

[0013] Specifically, in step S3, each expert module consists of one or more fully connected layers. The input of the expert module is the output h of the shared encoder, and the output of the expert module is the probability distribution of the second-level label subspace corresponding to the first-level label, as expressed below: ; in, Representing text x In the first-level label Subcategories of secondary tags B The probability distribution; For the first i Weight matrices for each expert module; For the first i The bias vector of each expert module is used to capture the inherent distribution tendency of each secondary label in the first-level label subspace when it does not depend on the input features.

[0014] Specifically, step S4 includes the following steps: The formula for weighted fusion of the prediction results of the Top-k activated expert modules in the joint probability calculation is as follows: ; in, Representing text x The primary tag is And the secondary label is The probability, Indicates the first i Each expert module outputs secondary tags as follows: The conditional probability; Edge probability fusion: The edge probabilities of secondary labels are calculated by weighted summation. ; in, Indicates that in the given input text And it is known that it belongs to the first-level tag. Under the premise that the text belongs to the conditional probability distribution of each tag in the corresponding second-level tag subspace; The marginal probability representing a second-level label refers to the probability after considering the top-ranked labels selected by the router. k After considering all possible first-level label paths, the global confidence of the second-level label is approximately calculated using the law of total probability. Predicted output: Candidate label pairs are determined using joint probability, and marginal probabilities are used as the basis for the final decision. Target label pairs are initially selected based on the principle of maximizing joint probability: ; in, The label pair with the highest joint probability; The edge probability of the secondary label corresponding to the path Compared with the preset confidence threshold If a comparison is made, If so, it is confirmed as a high-confidence prediction, and the prediction result is output. Otherwise, the system determines that the current text belongs to a sample with ambiguous boundaries and outputs the "awaiting manual verification" flag.

[0015] Preferably, the present invention optimizes the MOE model through a three-stage training strategy, including: Phase 1: Truth-based routing warm-up. In the early stages of training, samples are forcibly assigned to the corresponding expert modules using real first-level labels for training. ; in, The cross-entropy loss of the router module, Indicates the true first-level tag Classification loss under guidance; Phase Two: Predictive Route Annealing. During the middle of training, gradually transition from ground truth routes to predicted routes, setting an annealing factor. If the number of training rounds decreases linearly, then the routing decision is expressed as: ; in, This represents the final generated hybrid routing decision vector, which is used to actually activate the expert module and calculate the loss during the training phase; The true value routing vector represents a one-hot encoded vector constructed based on the true first-level labels of the samples, used to guide the model to force the activation of the correct expert modules; The predicted routing vector represents the probability distribution of first-level label allocation calculated in real time by the router module based on the current parameters. Phase 3: Load balancing constraints, introducing load balancing losses to suppress expert collapse, load balancing losses for: ; in, Indicates the first i The proportion of samples received by each expert module This indicates that the router module is assigned to the first... i The average probability of each expert module This is the scaling factor; Overall loss function for MOE model training for: ; ; in, For hierarchical classification, cross-entropy loss, These are the weighting coefficients. It is the natural logarithm function; Indicates that in the given input text x Under these conditions, the model predicts the first-level label as the true first-level label. The probability of; Indicates that in the given input text x And it is known that the first-level tag is Under these conditions, the model predicts the true second-level labels. The probability; by minimizing The model learns both accurate classification ability and a balanced expert load.

[0016] A second aspect of the present invention provides a hierarchical labeled text classification system based on the MOE model, comprising: Text encoding unit: Used to receive input text and generate a text vector representation through a shared encoder; Routing allocation unit: used to calculate the probability of first-level label allocation and select Top-k experts; Expert prediction unit: contains multiple expert sub-units, each of which is used to perform classification prediction within the corresponding secondary label subspace; Probabilistic fusion unit: used to weight and fuse the prediction results of multiple experts and output the final classification result.

[0017] Preferably, the classification system further includes: Training control unit: Used to execute the three-stage training strategy, control the routing preheating, annealing process and load balancing constraints.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention extracts text semantic features uniformly through a shared encoder, avoiding repeated learning in each branch; the router module allows samples to activate multiple candidate experts simultaneously through a Top-k soft routing mechanism, breaking the rigid single-path limitation of traditional top-down methods. Even if there is uncertainty in the first-level label prediction, accurate second-level classification can still be obtained through the fusion of multiple expert results; the expert module only performs specialized predictions in the corresponding second-level subspace, realizing the specialized division of labor of divide and conquer; finally, the probability fusion combines the routing weights with the expert outputs, taking the maximization of joint probability as the decision basis, which significantly reduces the number of model parameters and the risk of error cascading while ensuring classification accuracy. (2) This invention systematically improves the stability and generalization ability of model training by designing a three-stage training strategy. In the ground truth routing warm-up stage, the use of real first-level labels to allocate samples is forced, enabling each expert to quickly establish preliminary classification ability and avoid initial routing noise interference. In the prediction routing annealing stage, the real routes and prediction routes are dynamically mixed by the annealing factor, so that the model can smoothly transition to the autonomous inference state and reduce the performance gap between training and inference. In the load balancing constraint stage, the load balancing loss is introduced to prevent some experts from being over-activated or ignored, especially ensuring the training opportunities of long-tail category experts, so that the MOE model can still maintain a balanced and stable learning process under a complex hierarchical label system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a hierarchical label text classification method based on the MOE model of the present invention.

[0021] Figure 2 This is a flowchart illustrating the three-stage training strategy in an embodiment of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] Reference Figure 1 The first aspect of this invention provides a hierarchical labeled text classification method based on the MOE model, comprising the following steps: Step S1, Text Encoding: Encode the input text using a shared encoder to obtain a unified vector representation of the text; In the specific implementation process, before encoding the input text, the input text is preprocessed. The preprocessing operations include word segmentation, removal of special characters, truncation or padding to a fixed length, etc.

[0024] In this embodiment, a pre-trained language model (such as BERT, RoBERTa, etc.) is used as a shared encoder to process the input text. xConvert to dense vector representation: ; in, For text x The vector representation of , d represents the hidden layer dimension; Enc represents the shared encoder. The shared encoder shares parameters across all classification branches, enabling the reuse of common language knowledge.

[0025] Step S2, Route Allocation: Input the text vector representation into the router module, calculate the allocation probability distribution of the first-level labels, and select the expert modules corresponding to the top-k first-level labels for activation based on this probability distribution. The specific implementation method is as follows: Level 1 Tag Probability Calculation: The router module calculates the probability distribution of text belonging to each level 1 tag using linear transformation and the Softmax activation function. The calculation formula is as follows: ; in, Representing text x Belongs to each first-level tag The probability distribution vector, Here is the weight matrix of the router. For bias vectors, n This represents the number of first-level tags.

[0026] Top-k expert selection: Based on the probability distribution of the first-level labels, select the expert modules corresponding to the k labels with the highest probabilities for activation. ; The probability of selecting each expert module is normalized using the following formula: ; in, This indicates that the selected number after normalization i The probability of each expert module, i.e., the normalized route weight in the Top-k first-level label subspace; , These represent the selected numbers before normalization. i The and the first j The probability of the expert module; This is the set of indices for the selected Top-k expert modules.

[0027] Step S3, Expert Prediction: Each activated expert module performs classification prediction in its corresponding secondary label subspace and outputs the conditional probability distribution of the secondary labels. In this embodiment, each expert module It consists of one or more fully connected layers. The input of the expert module is the output h of the shared encoder, and the output of the expert module is the probability distribution of the second-level label subspace corresponding to the first-level label, as shown in the following expression: ; in, Representing text x In the first-level label Subcategories of secondary tags B The probability distribution; For the first i Expert Module The weight matrix; For the first i Expert Module The bias vector is used to capture the inherent distribution tendency (prior preference) of each secondary label in the first-level label subspace without relying on input features. By continuously optimizing this vector during training, the model's fitting accuracy to the distribution pattern of subclass labels can be improved.

[0028] Each expert is only responsible for the classification of secondary labels under their primary category, achieving a specialized division of labor through "divide and conquer". The expert network can choose different depths and widths according to specific tasks.

[0029] Step S4, Probability Fusion and Output: The prediction results of the Top-k activation expert modules are weighted and fused to obtain the final hierarchical label classification result. The weights of the weighted fusion are determined by the first-level label assignment probabilities output by the router module. Specific methods include: The formula for weighted fusion of the prediction results of the Top-k activated expert modules in the joint probability calculation is as follows: ; in, Representing text x The primary tag is And the secondary label is The probability, Indicates the first i Each expert module outputs secondary tags as follows: The conditional probability; Edge probability fusion: The edge probabilities of secondary labels are calculated by weighted summation. ; in, Indicates that in the given input text And it is known that it belongs to the first-level tag. Under the premise that the text belongs to the conditional probability distribution of each tag within the corresponding second-level tag subspace; the marginal probability of the second-level tag This refers to taking into account routers ( The selected first k After considering all possible primary label paths, the global confidence score of the secondary label is approximately calculated using the law of total probability. This means integrating the local predictions from different expert modules based on their corresponding path weights, eliminating biases caused by single-branch decisions. Its function is to provide the system with the final secondary classification decision basis, enabling the model to simultaneously consider multiple potential primary classification paths and significantly enhancing its robustness in classifying samples with ambiguous primary label boundaries.

[0030] Final prediction output: Candidate label pairs are determined using joint probability, and marginal probabilities are used as the basis for the final decision. First, target label pairs are initially selected based on the principle of maximizing joint probability: ; in, The label pair with the highest joint probability; Subsequently, the edge probability of the secondary label corresponding to this path Compared with the preset confidence threshold If a comparison is made, If so, it is confirmed as a high-confidence prediction, and the prediction result is output. Otherwise, the system determines that the current text belongs to a sample with ambiguous boundaries and outputs the "awaiting manual verification" flag.

[0031] Preferably, such as Figure 2 As shown, to ensure the stability of model training, this invention optimizes the MOE model through a three-stage training strategy, including: Phase 1: Teacher Forcing (True Value Routing Warm-up): In the early stages of training, samples are forcibly assigned to the corresponding expert modules using the true first-level labels for training. ; in, The cross-entropy loss of the router module, Indicates the true first-level tag Classification loss under guidance; this stage enables each expert to first establish discriminative ability in their subspace, reducing training instability caused by initial routing noise.

[0032] Phase Two: Scheduled Sampling. During the middle of training, the system gradually transitions from ground truth routes to predicted routes, setting an annealing factor. If the number of training rounds decreases linearly, then the routing decision is expressed as: ; in, This represents the final generated hybrid routing decision vector, which is used to actually activate the expert module and calculate the loss during the training phase; For true value routing vectors ( Typically, these are one-hot encoded vectors constructed based on the true first-level labels of the samples, used to guide the model to force the activation of the correct expert modules; To predict the route vector ( This represents the first-level label allocation probability distribution calculated and output by the router module in real time based on current parameters; this strategy uses... The dynamic adjustment of values ​​allows the model to smoothly transition from relying on "teacher guidance" to relying on "autonomous prediction" during the training process, effectively reducing the possible cascading error abrupt changes during the inference stage and improving the consistency of model predictions.

[0033] Phase 3: Load Balancing Constraints, introducing load balancing loss to suppress expert collapse, and load balancing loss. for: ; in, Indicates the first i The proportion of samples received by each expert module This indicates that the router module is assigned to the first... i The average probability of each expert module This is the scaling factor; this loss term reaches its minimum when all expert loads tend to be balanced.

[0034] Overall Loss Function: The final training loss is a weighted sum of the task loss and the load balancing loss. This is the overall loss function for training the MOE model. for: ; ; in, Cross-entropy loss for hierarchical classification; This is the weighting coefficient (which can be adjusted according to the specific data size and the number of experts), usually taken as... ; It is the natural logarithm function; Indicates that in the given input text x Under these conditions, the model predicts the first-level label as the true first-level label. The probability of; Indicates that in the given input text x And it is known that the first-level tag is Under these conditions, the model predicts the true second-level labels. The probability; by minimizing The model learns both accurate classification ability and a balanced expert load.

[0035] It should be noted that this invention supports extension to three-level, four-level, and deeper tagging systems, and supports "non-full tree / missing layer" structures. Specifically, the "router module + sub-expert set" is abstracted into node components. Each node contains the router of the current level, a list of experts, and references to child nodes pointing to the next level. During inference, a recursive drill-down strategy is used: the current node router selects the Top-k candidate branches; if the candidate is a leaf node, the path and probability are directly output; otherwise, the model recursively enters the child node to continue prediction. The probabilities of each level are multiplied together to obtain the joint confidence of the complete path. This design enables the model to flexibly adapt to real-world business tagging systems with variable depth and irregular structures.

[0036] A second aspect of the present invention provides a hierarchical labeled text classification system based on the MOE model, comprising: Text encoding unit: Used to receive input text and generate a text vector representation through a shared encoder; Routing allocation unit: used to calculate the probability of first-level label allocation and select Top-k experts; Expert prediction unit: contains multiple expert sub-units, each of which is used to perform classification prediction within the corresponding secondary label subspace; Probabilistic fusion unit: used to weight and fuse the prediction results of multiple experts and output the final classification result.

[0037] Preferably, the classification system further includes: Training control unit: Used to execute the three-stage training strategy, control the routing preheating, annealing process and load balancing constraints.

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hierarchical labeled text classification method based on the MOE model, characterized in that, Includes the following steps: S1, Text Encoding: Encode the input text using a shared encoder to obtain a unified vector representation of the text; S2, Route Allocation: Input the text vector representation into the router module, calculate the allocation probability distribution of the first-level labels, and select the expert modules corresponding to the top-k first-level labels for activation based on the probability distribution; S3, Expert Prediction: Each activated expert module performs classification prediction in its corresponding secondary label subspace and outputs the conditional probability distribution of the secondary labels; S4, Probabilistic Fusion and Output: The prediction results of the Top-k activation expert modules are weighted and fused to obtain the final hierarchical label classification result. The weights of the weighted fusion are determined by the first-level label allocation probability output by the router module.

2. The hierarchical label text classification method based on the MOE model as described in claim 1, characterized in that, In step S1, before encoding the input text, the input text is preprocessed. The preprocessing includes one or more of the following: word segmentation, removal of special characters, truncation or padding to a fixed length.

3. The hierarchical label text classification method based on the MOE model as described in claim 1, characterized in that, In step S1, a pre-trained language model is used as a shared encoder to process the input text. x Convert to dense vector representation: ； Where h is text x The vector representation of ; Enc represents the shared encoder.

4. The hierarchical label text classification method based on the MOE model as described in claim 1, characterized in that, In step S2, the router module calculates the probability distribution of text belonging to each first-level label through linear transformation and the Softmax activation function. The calculation formula is as follows: ； in, Representing text x Belongs to each first-level tag The probability distribution vector, Here is the weight matrix of the router. Let h be the bias vector and h be the text. x The vector representation of .

5. The hierarchical label text classification method based on the MOE model as described in claim 4, characterized in that, In step S2, based on the probability distribution of the first-level labels, the expert modules corresponding to the k labels with the highest probabilities are selected for activation, and the probability of each selected expert module is normalized using the following formula: ； in, This indicates that the selected number after normalization i The probability of each expert module. , These represent the selected numbers before normalization. i The and the first j The probability of the expert module; This is the set of indices for the selected Top-k expert modules.

6. The hierarchical label text classification method based on the MOE model as described in claim 5, characterized in that, In step S3, each expert module consists of one or more fully connected layers. The input of the expert module is the output h of the shared encoder, and the output of the expert module is the probability distribution of the second-level label subspace corresponding to the first-level label, as shown in the following expression: ； in, Representing text x In the first-level label Subcategories of secondary tags B The probability distribution; For the first i Weight matrices for each expert module; For the first i The bias vector of each expert module is used to capture the inherent distribution tendency of each secondary label in the first-level label subspace when it does not depend on the input features.

7. The hierarchical label text classification method based on the MOE model as described in claim 5, characterized in that, Step S4 includes the following steps: The formula for weighted fusion of the prediction results of the Top-k activated expert modules in the joint probability calculation is as follows: ； in, Representing text x The primary tag is And the secondary label is The probability, Indicates the first i Each expert module outputs secondary tags as follows: The conditional probability; Edge probability fusion: The edge probabilities of secondary labels are calculated by weighted summation. ； in, Indicates that in the given input text And it is known that it belongs to the first-level tag. Under the premise that the text belongs to the conditional probability distribution of each tag in the corresponding second-level tag subspace; The marginal probability representing a second-level label refers to the probability after considering the top-ranked labels selected by the router. k After considering all possible first-level label paths, the global confidence of the second-level label is approximately calculated using the law of total probability. Predicted output: Candidate label pairs are determined using joint probability, and marginal probabilities are used as the basis for the final decision. Target label pairs are initially selected based on the principle of maximizing joint probability: ； in, The label pair with the highest joint probability; The edge probability of the secondary label corresponding to the path Compared with the preset confidence threshold If a comparison is made, If so, it is confirmed as a high-confidence prediction, and the prediction result is output. Otherwise, the system determines that the current text belongs to a sample with ambiguous boundaries and outputs the "awaiting manual verification" flag.

8. The hierarchical label text classification method based on the MOE model as described in claim 1, characterized in that, The MOE model is optimized using a three-stage training strategy, including: Phase 1: Truth-based routing warm-up. In the early stages of training, samples are forcibly assigned to the corresponding expert modules using real first-level labels for training. ； in, The cross-entropy loss of the router module, Indicates the true first-level tag Classification loss under guidance; Phase Two: Predictive Route Annealing. During the middle of training, gradually transition from ground truth routes to predicted routes, setting an annealing factor. If the number of training rounds decreases linearly, then the routing decision is expressed as: ； in, This represents the final generated hybrid routing decision vector, which is used to actually activate the expert module and calculate the loss during the training phase; The true value routing vector represents a one-hot encoded vector constructed based on the true first-level labels of the samples, used to guide the model to force the activation of the correct expert modules; The predicted routing vector represents the probability distribution of first-level label allocation calculated in real time by the router module based on the current parameters. Phase 3: Load balancing constraints, introducing load balancing losses to suppress expert collapse, load balancing losses for: ； in, Indicates the first i The proportion of samples received by each expert module This indicates that the router module is assigned to the first... i The average probability of each expert module This is the scaling factor; Overall loss function for MOE model training for: ； ； in, For hierarchical classification, cross-entropy loss, These are the weighting coefficients. It is the natural logarithm function; Indicates that in the given input text x Under these conditions, the model predicts the first-level label as the true first-level label. The probability of; Indicates that in the given input text x And it is known that the first-level tag is Under these conditions, the model predicts the true second-level labels. The probability; by minimizing The model learns both accurate classification ability and a balanced expert load.

9. A hierarchical labeled text classification system based on the MOE model, characterized in that, The classification method described in any one of claims 1-8 includes: Text encoding unit: Used to receive input text and generate a text vector representation through a shared encoder; Routing allocation unit: used to calculate the probability of first-level label allocation and select Top-k experts; Expert prediction unit: contains multiple expert sub-units, each of which is used to perform classification prediction within the corresponding secondary label subspace; Probabilistic fusion unit: used to weight and fuse the prediction results of multiple experts and output the final classification result.

10. A hierarchical labeled text classification system based on the MOE model as described in claim 9, characterized in that, The classification system also includes: Training control unit: Used to execute the three-stage training strategy, control the routing preheating, annealing process and load balancing constraints.

Images