A method and apparatus for generating a hierarchical MOE model

Abstract

Embodiments of the present application disclose a method and device for generating a hierarchical MOE model. In the embodiments of the present application, a multi-source data set is obtained; a classification training set and a classification prediction set are determined according to the multi-source data set; an initial hierarchical hybrid expert MOE model is trained using the classification training set; the classification prediction set is predicted using the initial hierarchical MOE model to determine a development set; the development set is labeled with consistent labels using a large language model to generate a labeled development set; the large language model is distilled according to the labeled development set to generate a semantic model; the multi-source data set is labeled with consistent labels using a language model to generate a consistent auxiliary data set; and the initial hierarchical MOE model is loss-optimized according to the consistent auxiliary data set to generate a target hierarchical MOE model. Through the above method, a target hierarchical MOE model for tax code prediction can be generated, and thus relatively accurate tax code prediction can be achieved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method and apparatus for generating hierarchical MOE models. Background Technology

[0002] With the rapid development of e-commerce platforms, a massive amount of goods enter these platforms every day. In order to pay taxes legally and compliantly, it is necessary to assign tax codes to these massive amounts of goods. Due to the classification of goods and the requirements of the complex tax system, the assignment of tax codes involves as many as ten levels and more than four thousand leaf nodes. Therefore, accurately assigning tax codes to goods is a rather complex task.

[0003] In existing technologies, automatic classification models are used to assign tax codes. However, these automatic classification models often ignore hierarchical constraints and lack semantic understanding, leading to misjudgments where the codes appear to match superficially but do not actually match. Therefore, the assigned tax codes need to be manually reviewed, which is costly and prone to errors.

[0004] In conclusion, how to efficiently and accurately determine the corresponding tax code for a product is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for generating a hierarchical MOE model, which can generate a target hierarchical MOE model for tax code prediction, thereby achieving more accurate tax code prediction.

[0006] In a first aspect, embodiments of the present invention provide a method for generating a hierarchical MOE model, the method comprising: acquiring a multi-source dataset, wherein the multi-source dataset includes multiple sample data, each sample data including item information and tax code information; determining a classification training set and a classification prediction set based on the multi-source dataset; training an initial hierarchical hybrid expert MOE model using the classification training set; predicting the classification prediction set using the initial hierarchical MOE model to determine a development set, wherein the development set includes high-confidence data, low-confidence data, and erroneous data; performing consistency labeling on the development set using a large language model to generate a labeled development set; distilling the large language model based on the labeled development set to generate a semantic model; performing consistency labeling on the multi-source dataset using the language model to generate a consistency auxiliary dataset; and optimizing the loss of the initial hierarchical MOE model based on the consistency auxiliary dataset to generate a target hierarchical MOE model.

[0007] Optionally, the method further includes: inputting the information of the item to be predicted into the target level MOE model to obtain the target tax code.

[0008] Optionally, obtaining the multi-source dataset specifically includes: obtaining an initial multi-source dataset, wherein the initial multi-source dataset includes item registration records, invoice verification logs, and a knowledge base, and the item registration records, the invoice verification logs, and the knowledge base include item information and tax code information; filtering the initial multi-source dataset to determine the multi-source dataset.

[0009] Optionally, determining the classification training set and classification prediction set based on the multi-source dataset specifically includes: determining the classification training set by multiplying the multi-source dataset with a first predetermined ratio; and determining the classification prediction set by multiplying the multi-source dataset with a second predetermined ratio, wherein the sum of the first predetermined ratio and the second predetermined ratio is 1.

[0010] Optionally, the step of using the initial hierarchical MOE model to predict the classification prediction set and determine the development set specifically includes: inputting the item information in the classification prediction set into the initial hierarchical MOE model, outputting predicted tax code information and confidence data, wherein the confidence data includes high confidence data, low confidence data, and error data; and extracting a set proportion of high confidence data, low confidence data, and error data from the classification prediction set based on the confidence data to construct the development set.

[0011] Optionally, the initial hierarchical MOE model is optimized for loss based on the consistency auxiliary dataset to generate a target hierarchical MOE model. Specifically, this includes: determining the total loss based on the consistency auxiliary dataset; and optimizing the initial hierarchical MOE model for loss based on the total loss to generate a target hierarchical MOE model.

[0012] Optionally, determining the total loss based on the consistency auxiliary dataset specifically includes: determining the prediction loss and semantic loss based on the consistency auxiliary dataset; determining a first product of a first weight and the prediction loss, and a second product of a second weight and the semantic loss, wherein the sum of the first weight and the second weight is 1; and determining the sum of the first product and the second product as the total loss.

[0013] Optionally, determining the prediction loss based on the consistency auxiliary dataset specifically includes: determining gating features and expert features based on the consistency auxiliary dataset; outputting the gating features and expert features to the initial hierarchical MOE model to determine hierarchical prediction labels and leaf node prediction labels; determining hierarchical loss and leaf node loss based on the tax code classification real labels in the consistency auxiliary dataset and the hierarchical prediction labels and leaf node prediction labels respectively; and determining the prediction loss based on the hierarchical loss and the leaf node loss.

[0014] Optionally, determining the semantic loss based on the consistency auxiliary dataset specifically includes: determining title and label knowledge based on the consistency auxiliary dataset; inputting the title into the semantic model according to the prompt word engineering to generate consistent true labels; inputting the label knowledge into the text encoder and the linear layer to generate consistent predicted labels; and determining the semantic loss based on the consistent true labels and the consistent predicted labels.

[0015] Secondly, embodiments of the present invention provide an apparatus for generating a hierarchical MOE model, the apparatus comprising: an acquisition unit for acquiring a multi-source dataset, wherein the multi-source dataset includes multiple sample data, each sample data including item information and tax code information; a first determination unit for determining a classification training set and a classification prediction set based on the multi-source dataset; a training unit for training an initial hierarchical hybrid expert MOE model using the classification training set; a second determination unit for predicting the classification prediction set using the initial hierarchical MOE model to determine a development set, wherein the development set includes high-confidence data, low-confidence data, and erroneous data; a labeling unit for performing consistency labeling on the development set using a large language model to generate a labeled development set; a generation unit for distilling the large language model based on the labeled development set to generate a semantic model; the labeling unit is further configured to perform consistency labeling on the multi-source dataset using the language model to generate a consistency auxiliary dataset; and an optimization unit for performing loss optimization on the initial hierarchical MOE model based on the consistency auxiliary dataset to generate a target hierarchical MOE model.

[0016] Optionally, the device further includes a processing unit for inputting the information of the item to be predicted into the target-level MOE model to obtain the target tax code.

[0017] Optionally, the acquisition unit is specifically used to: acquire an initial multi-source dataset, wherein the initial multi-source dataset includes item registration records, invoice verification logs, and a knowledge base, and the item registration records, the invoice verification logs, and the knowledge base include item information and tax code information; filter the initial multi-source dataset to determine the multi-source dataset.

[0018] Optionally, the first determining unit is specifically used to: determine the classification training set by multiplying the multi-source dataset with a first predetermined ratio; and determine the classification prediction set by multiplying the multi-source dataset with a second predetermined ratio, wherein the sum of the first predetermined ratio and the second predetermined ratio is 1.

[0019] Optionally, the second determining unit is specifically used to: input the item information in the classification prediction set into the initial hierarchical MOE model, and output predicted tax code information and confidence data, wherein the confidence data includes high confidence data, low confidence data and error data; and extract a set proportion of high confidence data, low confidence data and error data from the classification prediction set based on the confidence data to construct the development set.

[0020] Optionally, the optimization unit is specifically used to: determine the total loss based on the consistency auxiliary dataset; and perform loss optimization on the initial hierarchical MOE model based on the total loss to generate the target hierarchical MOE model.

[0021] Optionally, the optimization unit is further configured to: determine the prediction loss and semantic loss based on the consistency auxiliary dataset; determine a first product of a first weight and the prediction loss, and a second product of a second weight and the semantic loss, wherein the sum of the first weight and the second weight is 1; and determine the sum of the first product and the second product as the total loss.

[0022] Optionally, the optimization unit is further configured to: determine gating features and expert features based on the consistency auxiliary dataset; output the gating features and expert features to the initial hierarchical MOE model to determine hierarchical prediction labels and leaf node prediction labels; determine hierarchical loss and leaf node loss based on the tax code classification real labels in the consistency auxiliary dataset and the hierarchical prediction labels and leaf node prediction labels respectively; and determine the prediction loss based on the hierarchical loss and the leaf node loss.

[0023] Optionally, the optimization unit is further configured to: determine title and label knowledge based on the consistency auxiliary dataset; input the title into the semantic model according to the prompt word engineering to generate consistent true labels; input the label knowledge into the text encoder and the linear layer to generate consistent predicted labels; and determine the semantic loss based on the consistent true labels and the consistent predicted labels.

[0024] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possibilities of the first aspect.

[0026] In this embodiment of the invention, a multi-source dataset is acquired, comprising multiple sample data points, each including item information and tax code information. A classification training set and a classification prediction set are determined based on the multi-source dataset. An initial hierarchical hybrid expert MOE model is trained using the classification training set. The initial hierarchical MOE model is used to predict the classification prediction set to determine a development set, which includes high-confidence data, low-confidence data, and erroneous data. A large language model is used to perform consistency labeling on the development set, generating a labeled development set. The large language model is distilled based on the labeled development set to generate a semantic model. The language model is used to perform consistency labeling on the multi-source dataset to generate a consistency auxiliary dataset. The initial hierarchical MOE model is optimized for loss based on the consistency auxiliary dataset to generate a target hierarchical MOE model. Through the above method, efficient and accurate tax code prediction can be achieved by training on multi-source data based on the MOE structure and generating a target hierarchical MOE model based on LLM-based semantic consistency supervision. Attached Figure Description

[0027] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of a method for generating a hierarchical MOE model according to an embodiment of the present invention; Figure 2 This is a flowchart of another method for generating a hierarchical MOE model in an embodiment of the present invention; Figure 3 This is a flowchart of a method for determining prediction loss in an embodiment of the present invention; Figure 4 This is a flowchart of a method for determining semantic loss in an embodiment of the present invention; Figure 5 This is a flowchart of a method for determining total loss in an embodiment of the present invention; Figure 6 This is a schematic diagram of an apparatus for generating a hierarchical MOE model according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0029] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0030] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0031] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0032] In existing technologies, automatic classification models and traditional hierarchical text classification are employed. Tax codes are assigned using methods such as automatic classification (e.g., hierarchical text classification) and prompt tuning. Specifically, the automatic classification model often ignores hierarchical constraints and lacks semantic understanding, leading to misjudgments of superficial matches but substantive discrepancies, causing compliance risks and financial losses. Therefore, the tax codes assigned by the automatic classification model need to be manually reviewed, which is costly and error-prone. While traditional hierarchical text classification methods can model hierarchical relationships, they rely solely on text classification loss, ignoring semantic consistency, and are prone to "correct labels but unreasonable" predictions. For example, classifying "maternity pants" into "other clothing" instead of the more specific maternal and infant category. These traditional hierarchical text classification methods can include Hierarchical Graph Contrastive Learning for Representation (HGCLR) and Hierarchical Inductive Learning for Long-Tail Distributions. Long-tailed (HILL) and other methods; the aforementioned prompting fine-tuning methods can improve classification performance with few samples by utilizing prompting engineering, but in Chinese e-commerce scenarios, they are still limited by insufficient dictionary coverage and contextual ambiguity, and cannot guarantee that the final output meets the requirements for tax code definition; among them, the prompting fine-tuning methods can be layer-wise hierarchical mixing (LH-Mix). Therefore, how to efficiently and accurately determine the tax code corresponding to the product is a problem that needs to be solved.

[0033] In this embodiment of the invention, to solve the above problems, a method for generating a hierarchical MOE model is proposed, specifically as follows: Figure 1 As shown, the method includes: Step S101: Obtain multi-source datasets.

[0034] Specifically, the multi-source dataset includes multiple sample data entries, each of which includes item information and tax code information.

[0035] In this embodiment of the invention, the item information includes item title, source, and business attributes; the tax code information includes tax code title, tax code, and tax code definition; wherein, the item title may also be referred to as item title, product title, or product description.

[0036] In one possible implementation, obtaining the multi-source dataset specifically includes: obtaining an initial multi-source dataset, wherein the initial multi-source dataset includes item registration records, invoice verification logs, and a knowledge base, and the item registration records, the invoice verification logs, and the knowledge base include item information and tax code information; filtering the initial multi-source dataset to determine the multi-source dataset.

[0037] Specifically, the multivariate dataset is a high-quality training set, and the filtering can be to remove duplicate sample data and / or remove incorrectly labeled items, etc.

[0038] Step S102: Determine the classification training set and classification prediction set based on the multi-source dataset.

[0039] Specifically, the product of the multi-source dataset and a first predetermined ratio is determined as the classification training set; the product of the multi-source dataset and a second predetermined ratio is determined as the classification prediction set, wherein the sum of the first predetermined ratio and the second predetermined ratio is 1.

[0040] In one possible implementation, it is assumed that the first set ratio is 0.8, that is, 80% of the sample data in the multi-source dataset is extracted as the classification training set, and the second set ratio is 0.2, that is, the remaining 20% ​​of the sample data after extracting 80% of the sample data in the multi-source dataset is used as the classification prediction set.

[0041] Step S103: Train the initial hierarchical Mixture-of-Experts (MOE) model using the classification training set.

[0042] Specifically, the initial hierarchical hybrid expert MOE model is an efficient and scalable neural network architecture that routes different inputs to different sub-modules for specialized inference. The input of the initial MOE model is the item information in the sample data, and the output of the initial MOE model is the tax code information. The initial MOE model is trained based on multiple sample data in the classification training set.

[0043] Step S104: Use the initial hierarchical MOE model to predict the classification prediction set and determine the development set.

[0044] Specifically, the item information in the classification prediction set is input into the initial hierarchical MOE model, and the predicted tax code information and confidence data are output. The confidence data includes high-confidence data, low-confidence data, and error data. Based on the confidence data, a set proportion of high-confidence data, low-confidence data, and error data are extracted from the classification prediction set to construct the development set. The development set includes high-confidence data, low-confidence data, and error data.

[0045] In one possible implementation, the item information in the classification prediction set is input into the initial hierarchical MOE model, and the predicted tax code information is output. The predicted tax code information is compared with the tax code information corresponding to the item information in the classification prediction set to determine the confidence level. The sample data in the classification prediction set is divided according to the value of the confidence level. For example, sample data with a confidence level greater than or equal to a set threshold is determined as high-confidence data, and sample data with a confidence level less than the set threshold is determined as low-confidence data and error data.

[0046] In this embodiment of the invention, since a large number of tokens are required when using a large language model to perform consistency labeling on the sample data, in order to reduce consumption, in step S104 above, after the classification prediction set is used for prediction, a set proportion of high-confidence data, low-confidence data and error data are extracted to construct the development set. The sum of the low-confidence data and error data in the development set is equal to the number of high-confidence data.

[0047] Step S105: Use a large language model to perform consistent labeling on the development set to generate the labeled development set.

[0048] In this embodiment of the invention, the Large Language Model (LLM) can also be called a large model or a large-scale language model, etc. The large language model is a deep learning model based on a transformer architecture that can process and generate natural language text. It is usually trained on a large amount of text data and has the ability to understand and generate language. It is widely used in dialogue systems, text generation and other natural language processing tasks. In this invention, it is used as an expert in judging the semantic consistency between tax code definition and item title.

[0049] In one possible implementation, the consistency label includes three categories: consistent, inconsistent, and uncertain. The consistent label can be represented by Y, the inconsistent label can be represented by N, and the uncertain label can be represented by —. This is only an illustrative example.

[0050] In one possible implementation, before the LLM performs consistency labeling on each sample data in the development set, a prompt word needs to be entered. For example, a role prompt such as "You are a professional tax expert", JSON structured instructions, and few-shot examples are required to ensure labeling quality and consistency. The prompt word here is only an example and should be determined according to the actual situation.

[0051] Step S106: Distill the large language model based on the annotated development set to generate a semantic model.

[0052] In this embodiment of the invention, the semantic model obtained by distillation inherits the ability of the large semantic model to perform consistent labeling of sample data. Compared with the large language model, it consumes fewer tokens and is suitable for labeling multi-source datasets with a large amount of data.

[0053] Step S107: Use the language model to perform consistent labeling on the multi-source dataset to generate a consistent auxiliary dataset.

[0054] Specifically, each sample data in the multi-source dataset is labeled, that is, Y, N or — is labeled onto the sample data.

[0055] Step S108: Optimize the loss of the initial hierarchical MOE model based on the consistency auxiliary dataset to generate the target hierarchical MOE model.

[0056] Specifically, the total loss is determined based on the consistency auxiliary dataset, and the initial hierarchical MOE model is optimized based on the total loss to generate the target hierarchical MOE model.

[0057] In this embodiment of the invention, a complete flowchart is used to describe in detail the process of generating the target-level MOE model in four parts: dataset cleaning, dataset construction, semantic annotation, and final training. Specifically, as follows... Figure 2The process includes the following: The dataset cleaning part specifically involves obtaining the original internal dataset (assuming the original internal dataset contains 8 million records), cleaning the original internal dataset to determine the cleaned dataset (Cleansed Set), assuming the cleaned dataset contains 853,000 records, and using a portion of the cleaned dataset as a classification training set (Classif. Training Set), which contains 682,000 records. A Feature-Gating Hybrid Expert Model (MOE) is trained based on the classification training set, i.e., the initial hierarchical hybrid expert MOE model. The dataset construction part includes: predicting the classification prediction set based on the initial hierarchical hybrid expert MOE model, wherein the classification prediction set consists of the remaining data after removing the classification training set from the cleaned dataset; determining the high-confidence dataset, low-confidence data, and error dataset based on the prediction results, for example, the high-confidence dataset contains 33,000 records; the low-confidence data and error dataset contains 31,000 records; and constructing a development set from the high-confidence dataset, low-confidence data, and error dataset. For example, the development set contains 64,000 entries; the semantic annotation part includes annotating the development set using a large language model, annotating consistency labels, generating an annotated development set, distilling the large language model based on the annotated development set to generate a semantic model, and using the language model to annotate the cleaned dataset with consistency labels to generate a consistency auxiliary dataset; the final training part includes performing loss optimization training on the initial hierarchical MOE model based on the consistency auxiliary dataset and the semantic model to generate a target hierarchical MOE model.

[0058] In one possible implementation, determining the total loss based on the consistency auxiliary dataset specifically includes: determining a prediction loss and a semantic loss based on the consistency auxiliary dataset; determining a first product of a first weight and the prediction loss, and a second product of a second weight and the semantic loss, wherein the sum of the first weight and the second weight is 1; and determining the sum of the first product and the second product as the total loss.

[0059] For example, suppose the first weight is 1- The second weight is The total loss = (1 - Predicted loss + Semantic loss.

[0060] The process of determining the prediction loss and the language loss will be described in detail below through two specific embodiments. Specific Implementation Example 1 The process of determining the prediction loss based on the consistency auxiliary dataset is as follows: Figure 3 As shown, the specific steps include the following: Step S301: Determine the gating features and expert features based on the consistency auxiliary dataset.

[0062] In one possible implementation, gating features are determined based on the business attributes in the item information of the sample data in the consistency auxiliary dataset. For example, business features corresponding to enumerable business attributes such as bu_code and ou_code are used. Expert features are determined based on the item title, source, business attributes, etc. in the item information.

[0063] Step S302: Output the gating features and expert features to the initial hierarchical MOE model to determine the hierarchical prediction labels and leaf node prediction labels.

[0064] In one possible implementation, the initial hierarchical MOE model consists of gates and multiple hybrid experts (MOEs), belonging to a hierarchical feature-gated MOE architecture. Each level corresponds to a tax code classifier. The business features are input into the gates, and the expert weights (also known as routing weights) of each expert among the multiple hybrid experts are learned and assigned. The expert features are input into the MOEs to predict the level prediction label and the leaf node prediction label.

[0065] Step S303: Determine the hierarchical loss and leaf node loss based on the true tax code classification labels in the consistency auxiliary dataset and the hierarchical prediction labels and leaf node prediction labels, respectively.

[0066] In one possible implementation, the true label for the digital classification is the true tax code of each sample data in the consistency auxiliary dataset. The level loss is calculated based on the level prediction label and the true tax code classification label, and the level loss is calculated based on the leaf node prediction label and the true tax code classification label. The level loss is the loss corresponding to the root node above the leaf node loss.

[0067] Step S304: Determine the prediction loss based on the hierarchical loss and the leaf node loss.

[0068] Specifically, the third weight of the hierarchical loss and the fourth weight of the leaf node loss are obtained, and the third product of the third weight and the hierarchical loss, and the fourth product of the fourth weight and the leaf node loss are determined, wherein the sum of the third weight and the fourth weight is 1; the sum of the third product and the fourth product is determined as the prediction loss.

[0069] For example, suppose the third weight is... The fourth weight is 1- The predicted loss Hierarchical loss + (1- Leaf node loss. Specific Implementation Example 2 The process of determining semantic loss based on the consistency auxiliary dataset is as follows: Figure 4 As shown, the specific steps include the following: Step S401: Determine title and label knowledge based on the consistency auxiliary dataset.

[0071] Specifically, the title is the item title in the item information of the sample data in the consistency auxiliary dataset, and the label knowledge is the name, definition, and tax code of the tax code.

[0072] Step S402: Input the title into the semantic model according to the prompt word engineering to generate consistent real labels.

[0073] Step S403: Input the label knowledge into the text encoder and the linear layer to generate consistent prediction labels.

[0074] Step S404: Determine the semantic loss based on the consistent true label and the consistent predicted label.

[0075] In this embodiment of the invention, a complete structural flowchart is used to illustrate the process of determining the total loss in detail, as follows: Figure 5 As mentioned above, in the Figure 5In this process, gated features are input into gates, and expert features are input into multiple hybrid expert models (MOEs). An initial hierarchical hybrid expert MOE model constructed based on the gates and the multiple hybrid expert models outputs hierarchical predicted labels and leaf node predicted labels. The hierarchical loss is determined based on the hierarchical predicted labels and the classification ground truth labels. The leaf node loss is determined based on the leaf node predicted labels and the classification ground truth labels. The prediction loss is calculated based on the hierarchical loss and the leaf node loss. The title is input into the prompt word project (Prompting), and a consistency ground truth label is generated based on the semantic model (Semantic LLM). Label knowledge is input into the text encoder and the linear layer (Linear), generating a consistency prediction label. The semantic loss is determined based on the consistency ground truth label and the consistency prediction label. The total loss is determined based on the semantic loss and the prediction loss.

[0076] In one possible implementation, after training and generating the target-level MOE model, the method further includes: inputting the item information to be predicted into the target-level MOE model to obtain the target tax code.

[0077] In this embodiment of the invention, the generated target-level MOE model is embedded into the enterprise tax system in the form of microservices, supporting real-time query and closed-loop feedback of manual review.

[0078] In one possible implementation, since leaf nodes and root nodes are weakly constrained, a path reconstruction (RePath) technique is used to rebuild a valid path from the leaf nodes upwards for redundant or invalid nodes in the prediction path. This greatly improves the F1 score at the path level, where the F1 score represents the harmonic mean of precision and recall, a core indicator for measuring the overall performance of the classification task. The RePath technique automatically recovers the complete root-to-leaf path based on the leaf node prediction results, solving the problem of misjudgment in the intermediate layer. Specifically, a RePath post-processing step is introduced so that even if the intermediate layer prediction is inaccurate, a reasonable path is reconstructed based on high-confidence leaf nodes to avoid cascading errors.

[0079] Through the above embodiments, LLM is regarded as a domain expert, generating high-quality semantic tags. Through LLM-driven semantic loss, the target-level MOE model not only focuses on keyword matching but also deeply understands the essence of tax code definition, reducing misjudgments caused by similar expressions. By integrating multiple data sources such as product publishing, invoice verification, and business rules, the target-level MOE model forms dual supervision of structure and semantics, improving the accuracy of the target-level MOE model in predicting tax codes and enhancing the robustness and practicality of the target-level MOE model.

[0080] In this embodiment of the invention, an apparatus for generating a hierarchical MOE model is provided, such as... Figure 6 As shown, it specifically includes: an acquisition unit 601, a first determination unit 602, a training unit 603, a second determination unit 604, a labeling unit 605, a generation unit 606, and an optimization unit 607; The acquisition unit 601 is used to acquire a multi-source dataset, wherein the multi-source dataset includes multiple sample data, each of which includes item information and tax code information; the first determination unit 602 is used to determine a classification training set and a classification prediction set based on the multi-source dataset; the training unit 603 is used to train an initial hierarchical hybrid expert MOE model using the classification training set; the second determination unit 604 is used to predict the classification prediction set using the initial hierarchical MOE model to determine a development set, wherein the development set includes high-confidence data, low-confidence data, and erroneous data; the annotation unit 605 is used to perform consistency labeling on the development set using a large language model to generate an annotated development set; the generation unit 606 is used to distill the large language model based on the annotated development set to generate a semantic model; the annotation unit 605 is also used to perform consistency labeling on the multi-source dataset using the language model to generate a consistency auxiliary dataset; the optimization unit 607 is used to perform loss optimization on the initial hierarchical MOE model based on the consistency auxiliary dataset to generate a target hierarchical MOE model.

[0081] Furthermore, the device also includes a processing unit for inputting the information of the item to be predicted into the target-level MOE model to obtain the target tax code.

[0082] Further, the acquisition unit is specifically used to: acquire an initial multi-source dataset, wherein the initial multi-source dataset includes item registration records, invoice verification logs, and a knowledge base, and the item registration records, the invoice verification logs, and the knowledge base include item information and tax code information; filter the initial multi-source dataset to determine the multi-source dataset.

[0083] Further, the first determining unit is specifically used to: determine the classification training set by multiplying the multi-source dataset with a first predetermined ratio; and determine the classification prediction set by multiplying the multi-source dataset with a second predetermined ratio, wherein the sum of the first predetermined ratio and the second predetermined ratio is 1.

[0084] Further, the second determining unit is specifically used to: input the item information in the classification prediction set into the initial hierarchical MOE model, and output predicted tax code information and confidence data, wherein the confidence data includes high confidence data, low confidence data and error data; and extract a set proportion of high confidence data, low confidence data and error data from the classification prediction set based on the confidence data to construct the development set.

[0085] Furthermore, the optimization unit is specifically used to: determine the total loss based on the consistency auxiliary dataset; and perform loss optimization on the initial hierarchical MOE model based on the total loss to generate the target hierarchical MOE model.

[0086] Furthermore, the optimization unit is specifically configured to: determine the prediction loss and semantic loss based on the consistency auxiliary dataset; determine a first product of a first weight and the prediction loss, and a second product of a second weight and the semantic loss, wherein the sum of the first weight and the second weight is 1; and determine the sum of the first product and the second product as the total loss.

[0087] Furthermore, the optimization unit is specifically used for: determining gating features and expert features based on the consistency auxiliary dataset; outputting the gating features and expert features to the initial hierarchical MOE model to determine hierarchical prediction labels and leaf node prediction labels; determining hierarchical loss and leaf node loss based on the tax code classification real labels in the consistency auxiliary dataset and the hierarchical prediction labels and leaf node prediction labels respectively; and determining the prediction loss based on the hierarchical loss and the leaf node loss.

[0088] Furthermore, the optimization unit is specifically used to: determine title and label knowledge based on the consistency auxiliary dataset; input the title into the semantic model according to the prompt word engineering to generate consistent true labels; input the label knowledge into the text encoder and the linear layer to generate consistent predicted labels; and determine the semantic loss based on the consistent true labels and the consistent predicted labels.

[0089] Figure 7 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. Figure 7As shown, it includes a general computer hardware architecture, which includes at least a processor 701 and a memory 702. The processor 701 and the memory 702 are connected via a bus 703. The memory 702 is adapted to store instructions or programs executable by the processor 701. The processor 701 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 701 executes the instructions stored in the memory 702 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 703 connects the above-mentioned components together, and also connects the above-mentioned components to a display controller 704, a display device, and an input / output (I / O) device 705. The input / output (I / O) device 705 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 705 is connected to the system via an input / output (I / O) controller 706.

[0090] The instructions stored in memory 702 are executed by at least one processor 701 to achieve the following: acquiring a multi-source dataset; determining a classification training set and a classification prediction set based on the multi-source dataset; training an initial hierarchical hybrid expert MOE model using the classification training set; predicting the classification prediction set using the initial hierarchical MOE model to determine a development set; applying consistent labels to the development set using a large language model to generate a labeled development set; distilling the large language model based on the labeled development set to generate a semantic model; applying consistent labels to the multi-source dataset using the language model to generate a consistent auxiliary dataset; and optimizing the loss of the initial hierarchical MOE model based on the consistent auxiliary dataset to generate a target hierarchical MOE model.

[0091] Specifically, the electronic device includes: one or more processors 701 and a memory 702. Figure 7 Take a processor 701 as an example. The processor 701 and the memory 702 can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 701 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 702, thereby implementing the aforementioned method for determining the generation hierarchy MOE model.

[0092] Memory 702 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 702 may optionally include memory remotely located relative to processor 701, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0093] One or more modules are stored in memory 702, and when executed by one or more processors 701, they execute the method for generating a hierarchical MOE model in any of the above method embodiments.

[0094] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0095] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0096] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0097] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0098] Computer program code for performing operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0099] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0100] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0101] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0102] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

Claims

1. A method for generating a hierarchical MOE model, characterized in that, The method includes: Obtain a multi-source dataset, wherein the multi-source dataset includes multiple sample data, and each sample data includes item information and tax code information; Based on the multi-source dataset, determine the classification training set and the classification prediction set; The initial hierarchical hybrid expert MOE model was trained using the aforementioned classification training set; The initial hierarchical MOE model is used to predict the classification prediction set to determine the development set, wherein the development set includes high-confidence data, low-confidence data, and erroneous data; The development set is labeled with consistent tags using a large language model to generate a labeled development set. Distill the large language model based on the annotated development set to generate a semantic model; The language model is used to perform consistent labeling on the multi-source dataset to generate a consistent auxiliary dataset; The initial hierarchical MOE model is optimized for loss based on the consistency auxiliary dataset to generate the target hierarchical MOE model.

2. The method according to claim 1, characterized in that, The method further includes: Input the information of the item to be predicted into the target level MOE model to obtain the target tax code.

3. The method according to claim 1, characterized in that, The acquisition of multi-source datasets specifically includes: Obtain an initial multi-source dataset, wherein the initial multi-source dataset includes item registration records, invoice verification logs, and a knowledge base, and the item registration records, the invoice verification logs, and the knowledge base include item information and tax code information; The initial multi-source dataset is filtered to determine the multi-source dataset.

4. The method according to claim 1, characterized in that, The step of determining the classification training set and classification prediction set based on the multi-source dataset specifically includes: The product of the multi-source dataset and the first predetermined ratio is used to determine the classification training set; The product of the multi-source dataset and the second predetermined ratio is determined as the classification prediction set, wherein the sum of the first predetermined ratio and the second predetermined ratio is 1.

5. The method according to claim 1, characterized in that, The step of using the initial hierarchical MOE model to predict the classification prediction set and determine the development set specifically includes: The item information in the classification prediction set is input into the initial hierarchical MOE model, and the predicted tax code information and confidence data are output, wherein the confidence data includes high confidence data, low confidence data and error data; Based on the confidence data, a set proportion of high-confidence data, low-confidence data, and erroneous data are extracted from the classification prediction set to construct the development set.

6. The method according to claim 1, characterized in that, Based on the consistency auxiliary dataset, the initial hierarchical MOE model is optimized for loss to generate the target hierarchical MOE model, specifically including: The total loss is determined based on the consistency auxiliary dataset; Based on the total loss, the initial hierarchical MOE model is optimized to generate the target hierarchical MOE model.

7. The method according to claim 6, characterized in that, The total loss is determined based on the consistency auxiliary dataset, specifically including: The prediction loss and semantic loss are determined based on the consistency auxiliary dataset. Determine a first product of a first weight and the prediction loss, and a second product of a second weight and the semantic loss, wherein the sum of the first weight and the second weight is 1; The sum of the first product and the second product is determined as the total loss.

8. The method according to claim 7, characterized in that, The step of determining the prediction loss based on the consistency auxiliary dataset specifically includes: Gated features and expert features are determined based on the consistency auxiliary dataset; The gating features and expert features are output to the initial hierarchical MOE model to determine the hierarchical prediction labels and leaf node prediction labels; Based on the true tax code classification labels in the consistency auxiliary dataset, the hierarchical loss and leaf node loss are determined by comparing them with the hierarchical prediction labels and leaf node prediction labels, respectively. The prediction loss is determined based on the hierarchical loss and the leaf node loss.

9. The method according to claim 7, characterized in that, The step of determining the semantic loss based on the consistency auxiliary dataset specifically includes: Determine title and label knowledge based on the aforementioned consistency auxiliary dataset; The title is input into the semantic model based on the prompt word engineering to generate consistent real labels; The label knowledge is input into a text encoder and a linear layer to generate consistent prediction labels; The semantic loss is determined based on the consistent true label and the consistent predicted label.

10. An apparatus for generating hierarchical MOE models, characterized in that, The device includes: The acquisition unit is used to acquire a multi-source dataset, wherein the multi-source dataset includes multiple sample data, and each sample data includes item information and tax code information; The first determining unit is used to determine the classification training set and the classification prediction set based on the multi-source dataset; Training unit, used to train an initial hierarchical hybrid expert MOE model using the classification training set; The second determining unit is used to predict the classification prediction set using the initial hierarchical MOE model and determine the development set, wherein the development set includes high-confidence data, low-confidence data and erroneous data; The annotation unit is used to perform consistent labeling on the development set using a large language model, and generate an annotated development set. A generation unit is used to distill the large language model based on the labeled development set to generate a semantic model; The annotation unit is also used to perform consistency labeling on the multi-source dataset using the language model to generate a consistency auxiliary dataset; An optimization unit is used to perform loss optimization on the initial hierarchical MOE model based on the consistency auxiliary dataset to generate a target hierarchical MOE model.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images