Data processing method based on large language model of improved hybrid expert architecture

Abstract

The application relates to the technical field of artificial intelligence, and particularly provides a data processing method of a large language model based on an improved mixed expert architecture. The method comprises the following steps: processing input query information through a semantic understanding module to obtain an understanding result; based on the understanding result and a pre-stored knowledge graph, sending the query information to a plurality of corresponding expert models, obtaining corresponding output results from the expert models, and the plurality of expert models being provided with multiple layers based on the improved mixed expert architecture, and the expert models in different layers being obtained according to different expert dimensions; and performing fusion processing on the output results of the plurality of expert models to obtain a corresponding answer result of the query information. The method can solve the problems that the large language model based on the mixed expert architecture has low accuracy in professional subdivision fields and poor problem solving capability in the related art.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data processing method based on a large language model with an improved hybrid expert architecture. Background Technology

[0002] In related technologies, large language models (LMMs) employ a Mixture-of-Experts (MoE) architecture. This architecture distributes model parameters across multiple expert models and dynamically selects the expert's neural network through a gating routing mechanism for query processing. This approach can handle more specialized query information and output more accurate answers.

[0003] However, existing hybrid expert architectures are all single-layer, flat designs with coarse-grained expert model partitioning. This often results in poor performance when dealing with more specialized problems, leading to lower accuracy and professionalism in the answers. Furthermore, the routing module, primarily based on statistical learning methods, exhibits poor routing accuracy for more specialized queries.

[0004] In summary, no effective solution has yet been proposed for the problem that large language models with hybrid expert architectures in related technologies have low accuracy in specialized sub-domains and poor problem-solving capabilities. Summary of the Invention

[0005] The present invention provides a data processing method based on an improved hybrid expert architecture large language model, which at least solves the problem that the hybrid expert architecture large language model has low accuracy and poor problem-solving ability in professional sub-fields in related technologies.

[0006] According to one aspect of the present invention, a data processing method based on an improved hybrid expert architecture for a large language model is provided, comprising: processing input query information through a semantic understanding module to obtain an understanding result; based on the understanding result and a pre-stored knowledge graph, sending the query information to multiple corresponding expert models, wherein the expert models are configured with multiple layers based on the improved hybrid expert architecture, the expert models at different layers are divided according to different expert dimensions, at least one expert model in each layer processes the query information and obtains an output result, the professional domain of the knowledge graph matches the professional domain of the query information; and fusing the output results of the multiple expert models to obtain an answer result corresponding to the query information.

[0007] As an optional approach, based on the understanding results and a pre-stored knowledge graph, the query information is sent to multiple corresponding expert models, which then obtain corresponding output results. This includes: mapping the query information to the knowledge graph based on the understanding results to determine the relevant knowledge nodes of the query information; calculating the fit between the query information and each of the expert models based on the relevant knowledge nodes; incorporating expert models with the required fit into an expert model set and filtering them to obtain an expert combination; and sending the query information to multiple expert models in the expert combination, which then output corresponding results.

[0008] As an optional approach, calculating the fit between the query information and each of the expert models based on the relevant knowledge nodes includes: calculating the domain relevance between the query information and the expert models based on the relevant knowledge nodes; determining the similarity between the semantic features in the understanding results of the query information and the semantic profile of the expert models; obtaining the historical scores of the expert models for similar queries to which the query information belongs; and determining the fit based on the domain relevance, the similarity, and the historical scores, combined with the corresponding dynamic weights.

[0009] As an optional approach, expert models that meet the suitability requirements are included in an expert model set and screened to obtain an expert combination. This includes: determining multiple candidate subsets from the expert model set using a screening algorithm; calculating the knowledge coverage of the query information by the expert models in each candidate subset, as well as the knowledge redundancy between different expert models; and calculating the optimal candidate subset as the expert combination based on the knowledge coverage and the knowledge redundancy using an optimal solution algorithm.

[0010] As an optional approach, the query information is sent to multiple expert models in the expert portfolio, and the multiple expert models in the expert portfolio output corresponding results. This includes: inputting the query information into the expert model at the first layer of the expert portfolio and outputting a first result, wherein the first layer is the first layer of the improved hybrid expert architecture; sending the first result to the expert model at the next layer and outputting a second result; and inputting the second result layer by layer into the expert model at the next layer of the expert portfolio to obtain the output results of multiple expert models in the expert portfolio at each layer. The improved hybrid expert architecture includes at least: an upper professional dimension layer and a lower functional dimension layer. The expert models in the professional dimension layer include at least one of the following: customs expert, tax expert, logistics expert, and financial expert. The expert models in the functional dimension layer include at least one of the following: knowledge interpretation expert, rule reasoning expert, risk assessment expert, and decision support expert.

[0011] As an optional approach, the query information is input into the expert model at the first layer of the expert portfolio, and a first result is output. This includes: embedding the query information into professional terms through a domain terminology embedding layer to obtain a professional terminology embedding result; processing the professional terminology embedding result using an adaptive multi-head attention layer to obtain a context-aware representation; mapping the structured professional knowledge in the context-aware representation to the model parameter space using a knowledge-parameter mapping engine to obtain a knowledge-enhanced representation; and obtaining the corresponding first result based on the knowledge-enhanced representation using an interpretable output layer. The next layer of the expert model receives the result from the previous layer and obtains the corresponding output result using the domain terminology embedding layer, the adaptive multi-head attention layer, the knowledge-parameter mapping engine, and the interpretable output layer.

[0012] As an optional approach, the outputs of multiple expert models are fused to obtain the answer corresponding to the query information. This includes: assessing the credibility of the expert models and their corresponding outputs based on the expert models' professional coverage of the knowledge nodes of the query information using their knowledge graphs, the accuracy of the expert models' historical outputs, and the certainty of their outputs for the query information; calculating the weights of each output based on the credibility, the query relevance between the expert models' expertise vectors and the query vectors of the query information, and the complementarity parameters between different expert models; reconciling conflicting outputs based on the credibility to obtain adjusted outputs; and determining the answer based on the adjusted outputs and their corresponding weights.

[0013] As an optional approach, based on the understanding results and the pre-stored knowledge graph, the query information is sent to multiple corresponding expert models. After the expert models obtain the corresponding output results, the process includes: calculating a self-evaluation score based on the expert models' knowledge coverage of the query information, as well as the reasoning reliability and quantification uncertainty of the output results; determining whether there are errors in the output results of the expert models through expert peer review; if errors are found, the corresponding peer review experts provide error correction suggestions for error correction and feedback learning; and obtaining the output results of multiple expert models that reach a consensus through multiple rounds of peer review, which are then used for subsequent fusion to obtain the answer result.

[0014] As an optional approach, before sending the query information to multiple corresponding expert models based on the understanding results and pre-stored knowledge graphs, and before the expert models obtain the corresponding output results, the method further includes: independently pre-training the expert models using training data matched to professional domains; training the routing module of the improved hybrid expert architecture based on the pre-trained expert models, wherein the routing module is used to send the query information to multiple corresponding expert models based on the understanding results and pre-stored knowledge graphs; and jointly optimizing the trained routing module and the pre-trained expert models using an objective optimization function, wherein the objective optimization function includes: an answer result loss function, a routing quality loss function, an expert diversity loss function, and a knowledge consistency loss function.

[0015] According to another aspect of the invention, an electronic device is provided, comprising: a processor, and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0016] According to one aspect of the present invention, an electronic device is provided, comprising: a processor, and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0017] The data processing method based on an improved hybrid expert architecture using a large language model provided by this invention employs a multi-layered expert model divided according to different expert dimensions. It generates output results based on query information and then fuses these outputs to obtain the corresponding answer, resulting in higher accuracy and professionalism. Furthermore, by combining semantic understanding results of the query information with a knowledge graph matching the professional domain, the query information is routed and sent to the expert model within the improved hybrid expert architecture, achieving even higher accuracy. This further improves the professionalism and accuracy of answers to queries with deeper knowledge depth. Consequently, it addresses the problem of low accuracy and poor problem-solving ability of large language models using hybrid expert architectures in related technologies within specialized professional domains. Attached Figure Description

[0018] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating a data processing method for a large language model based on an improved hybrid expert architecture, as described in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of an improved hybrid expert architecture according to an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of data processing in an embodiment of the improved hybrid expert architecture created in this invention.

[0022] Figure 4 This is a schematic diagram of the expert model architecture of an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the routing module architecture of an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the fusion architecture of the output results of an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the self-evaluation and mutual evaluation architecture of an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the training architecture of an embodiment of the present invention.

[0027] Figure 9 This is a schematic diagram of the knowledge graph architecture of an embodiment of the present invention.

[0028] Figure 10 This is a schematic diagram of the implementation architecture of the large language model of the improved hybrid expert architecture according to an embodiment of the present invention.

[0029] Figure 11 This is a schematic diagram of the structure of an electronic device created by the present invention. Detailed Implementation

[0030] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0031] In related technologies, with the rapid development of artificial intelligence, large language models have become the mainstream technology in the field of natural language processing. However, with the continuous expansion of application scenarios, general-purpose large language models face many challenges in specialized fields, especially in terms of knowledge depth, specialization, and accuracy.

[0032] Hybrid expert model (MoE) architectures have been widely applied in large language models in recent years. Currently, mainstream MoE architectures include Google's Hybrid Expert Model Sharding Architecture (GShard) and the Switch Transformer architecture. GShard is a sparse conditional computation system based on transformers, which enhances the model's representational capabilities by using MoE in the feedforward network layer. The Switch Transformer further simplifies routing design based on GShard, employing a routing strategy where each query is routed to only one optimal expert model.

[0033] However, the aforementioned MoE architecture has the following main drawbacks.

[0034] (1) Flat structure design of single-layer expert architecture: The traditional MoE architecture adopts a flat expert model organization structure, with all expert models at the same level. It cannot effectively express the hierarchical knowledge system of the professional field and is difficult to capture the multi-dimensional professional needs in complex problems.

[0035] (2) Coarse-grained expert model partitioning: The partitioning of expert models in existing MoE models is usually based on computational efficiency rather than knowledge boundaries, resulting in knowledge overlap or gaps between expert models, which reduces the accuracy of professional domain applications.

[0036] (3) The routing mechanism lacks semantic awareness: The routing mechanism in the traditional MoE architecture is mainly based on statistical learning methods, lacking a deep understanding of professional concepts and terms, making it difficult to accurately distribute queries to the most suitable combination of expert models.

[0037] (4) Simple expert model collaboration mechanism: The existing MoE architecture usually adopts a simple weighted average or selects the highest-scoring expert model for expert model collaboration, which lacks the ability of expert models to complement each other, correct errors and conduct collaborative reasoning in complex scenarios.

[0038] The aforementioned shortcomings limit the application of existing MoE architectures in complex professional fields (such as trade, law, and healthcare), making it difficult to meet the high requirements of these fields for knowledge accuracy, reasoning reliability, and semantic understanding depth.

[0039] To address the aforementioned problems, according to one aspect of the present invention, a data processing method based on an improved hybrid expert architecture for large language models is provided. Figure 1This is a flowchart illustrating a data processing method for a large language model based on an improved hybrid expert architecture, as described in an embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps.

[0040] Step S101: The input query information is processed by the semantic understanding module to obtain the understanding result.

[0041] Step S102: Based on the understanding results and the pre-stored knowledge graph, the query information is sent to the corresponding multiple expert models, and the expert models obtain the corresponding output results. The multiple expert models are set up with multiple layers based on the improved hybrid expert architecture. The expert models in different layers are divided according to different expert dimensions. At least one expert model in each layer processes the query information and obtains the output results. The professional domain of the knowledge graph matches the professional domain of the query information.

[0042] Step S103: The output results of multiple expert models are fused to obtain the answer result corresponding to the query information.

[0043] The data processing method based on the improved hybrid expert architecture and large language model described in this embodiment employs a multi-layered expert model divided according to different expert dimensions. It obtains output results based on query information and then fuses these output results to obtain the corresponding answer, resulting in higher accuracy and professionalism. Furthermore, by combining the semantic understanding results of the query information with a knowledge graph matching the professional domain, the query information is routed and sent to the expert model in the improved hybrid expert architecture, further enhancing accuracy.

[0044] The entity executing the above steps can be a large language model based on an improved hybrid expert architecture, which can run on a computer, server, or cloud. The semantic understanding module can be a semantic understanding-based processing module that ultimately outputs a structured understanding result of the query information, which may include the key concepts and requirement types of the query information.

[0045] The semantic understanding module described above typically includes text preprocessing to remove meaningless words such as prepositions like "and" and "of". The preprocessed text is then used to identify the intent of the query, i.e., the demand type, through a classification model. Next, key concepts, i.e., entities, are extracted from the query. Based on these extracted key concepts, entity labeling and relation extraction are performed. The final understanding results include the entities, entity relationships, and demand types.

[0046] Entities and relationships are mapped onto a knowledge graph for later use. The knowledge graph described above is matched to the professional domain of the queried information. For example, professional domains could be trade, law, medicine, or a sub-domain of law such as commercial law.

[0047] The aforementioned knowledge graph is pre-created and stored in a fixed location before use. The creation of the knowledge graph may involve extracting entities and relationships from authoritative sources such as professional materials and books in the corresponding patent field. Then, knowledge from different sources is integrated, eliminating redundancy and contradictions. Next, the knowledge graph is built and expanded through rule-based reasoning. Finally, the correctness of the knowledge is verified through expert review and automatic consistency checks; once verification is successful, it is stored.

[0048] Since the understanding results include the type of query information and key concepts, these key concepts can be mapped onto a knowledge graph. Through the nodes and relationships of the knowledge graph, we can find the knowledge nodes closest to the query information, or the expert model with the closest coverage of those knowledge nodes. In other words, not all expert models in the improved hybrid expert architecture will process the query information.

[0049] Based on the understanding of the query information, it is necessary to search and match to find a subset of expert models that meet the requirements to process the query information. Expert models typically have their own knowledge base, representing the expert's expertise. The fit between the knowledge base and key concepts represents the fit between the expert model and the query information. Based on this fit, the expert model most relevant to the knowledge of the query information can be identified to solve the problem, thereby improving the accuracy of the expert model's output.

[0050] Based on the understanding results and the pre-stored knowledge graph, the query information is sent to multiple corresponding expert models, which then obtain the corresponding output results. The specific details will be explained later.

[0051] The aforementioned expert models are based on an improved hybrid expert architecture with multiple layers. The expert models at different layers are derived from different expert dimensions. For example, ... Figure 2 The improved hybrid expert architecture shown features a two-layer expert model: a professional dimension layer and a functional dimension layer. This multi-layered expert model allows for vertical collaboration between different layers, enhancing the depth of knowledge in query information processing and understanding.

[0052] For example, after processing the query information, the first-layer expert model uses its output as input to the next-layer expert model to obtain its output. This enables multi-dimensional collaboration among experts, allowing for a multi-dimensional understanding and processing of the query information, and solving problems related to understanding and resolving queries with higher knowledge depth.

[0053] At least one expert model in each layer processes the query information and produces the output. This is because different layers are divided from different professional dimensions. In order to improve the depth of query information processing and understanding, it is necessary for each professional dimension to cooperate and deepen the processing of query information, thereby meeting the needs of query information with higher knowledge depth.

[0054] Because different expert models have different levels and different dimensions of division, the results output by different expert models may not reach a consensus. Therefore, this embodiment fuses the output results of multiple expert models to obtain the answer result corresponding to the query information.

[0055] It should be noted that the fusion of expert model outputs can begin by evaluating the credibility of the expert models using multiple parameters. Weights can then be assigned to the expert models based on this credibility, and fusion can proceed accordingly. Furthermore, conflict detection can be performed before direct fusion to avoid significant conflicts and logical flaws between the outputs of different expert models, which could lead to a decrease in the accuracy and professionalism of the final fused answer.

[0056] The details of the credibility assessment, weighting, and conflict detection described above will be explained later.

[0057] In summary, by combining semantic understanding results of query information with a knowledge graph matching the professional domain, the query information is routed and sent to the expert model in the improved hybrid expert architecture. A multi-layered expert model, segmented according to different expert dimensions, is used to obtain output results based on the query information. Finally, the output results are fused to obtain the corresponding answer. This improves the accuracy and professionalism of query information processing and can meet the response requirements of queries with higher knowledge depth. Furthermore, it addresses the problem of low accuracy and poor problem-solving ability of large language models in hybrid expert architectures in related technologies at the level of specialized domains.

[0058] As an optional implementation, based on the understanding results and a pre-stored knowledge graph, the query information is sent to multiple corresponding expert models, which then obtain corresponding output results. This includes mapping the query information to the knowledge graph based on the understanding results to determine the relevant knowledge nodes of the query information; calculating the fit between the query information and each expert model in the improved hybrid expert architecture based on the relevant knowledge nodes; incorporating expert models that meet the fit requirements into the expert model set and filtering them to obtain an expert combination; and sending the query information to multiple expert models in the expert combination, which then output the corresponding results.

[0059] Since the understanding results only contain the key entity concepts inherent in the query information itself, they cannot be directly used to represent the knowledge nodes involved in the query information. Therefore, based on the understanding results, the query information can be mapped to a knowledge graph to determine the relevant knowledge nodes of the query information. These relevant knowledge nodes can represent the knowledge involved in the query information.

[0060] Then, based on relevant knowledge nodes, the fit between the query information and each expert model in the improved hybrid expert architecture is calculated to characterize the knowledge matching degree between each expert model and the query information. Clearly, the higher the knowledge matching degree, the more accurate the processing and understanding of the query information, and the higher its professionalism. Therefore, we can select which expert models can be used to process the query information based on the fit degree. Specifically, expert models that meet the fit degree requirements are first included in the expert model set. This expert model set can cover the expert models in the improved expert architecture that can be used for query information processing.

[0061] However, there's a situation where, if the depth of knowledge queried isn't deep enough or the level of expertise isn't high enough, the number of expert models in the expert model set will be large, resulting in numerous outputs. This can lead to new problems during fusion, increasing the probability of knowledge conflicts. Furthermore, it will consume more processing resources, take longer, and reduce efficiency.

[0062] To address the aforementioned issues, the expert model set can be filtered to obtain an optimized expert combination, which serves as the set of expert models for processing query information. During the filtering process, considerations can be given to the knowledge redundancy of different expert models, the knowledge coverage of each expert model regarding the query information, and the difficulty of collaboration between different expert models. These will be explained in detail later.

[0063] After obtaining the expert portfolio, the query information can be sent to multiple expert models within the portfolio, which will then output corresponding results. This serves as the data foundation for the final fusion. Furthermore, the expert models within the portfolio exist in a hierarchical structure. The query information can be sent directly to the portfolio or layer by layer. As mentioned above, this embodiment employs a layer-by-layer sending method to achieve collaboration among experts from multiple dimensions; details will be explained in more detail later.

[0064] As an optional implementation, the fitness of query information with each expert model in the improved hybrid expert architecture is calculated based on relevant knowledge nodes, including: calculating the domain relevance between the query information and the expert models based on relevant knowledge nodes; determining the similarity between the semantic features in the understanding results of the query information and the semantic profiles of the expert models; obtaining the historical scores of the expert models for similar queries to which the query information belongs; and determining the fitness based on the domain relevance, similarity, and historical scores, combined with the corresponding dynamic weights.

[0065] The above-mentioned calculation of the domain relevance between query information and the expert model based on relevant knowledge nodes can be performed by extracting the knowledge nodes of the query information and determining the knowledge nodes of the expert model's knowledge base, and then calculating the domain relevance using a relevance algorithm. Specifically, the Jaccard similarity function can be used to calculate the domain relevance. Alternatively, the node set can be encoded into vectors using a vector space model (such as One-Hot or graph embedding) and cosine similarity can be calculated. A personalized PageRank algorithm can also be run on the knowledge graph to calculate the relevance score of the expert nodes starting from the query nodes.

[0066] It should be noted that the expert model itself also has a semantic profile, which is a metadata file used to structurally describe the expert model's knowledge domain, capability boundaries, and semantic characteristics. Its core function is to achieve accurate matching, collaborative optimization, and system interpretability. Considering the semantic characteristics of the query information, this embodiment also refers to the similarity between the semantic features of the query information and the semantic profile of the expert model when calculating the expert model's fit. The semantic profile is a structured semantic representation, typically containing information such as the semantic category, attributes, and contextual dependencies of entities. Semantic features are the smallest units describing the semantics of entities and are the basic components of the semantic profile. Specific similarity calculation methods can include the overlap coefficient method based on feature matching, the AND similarity algorithm based on vector space, and the Euclidean distance algorithm, etc.

[0067] The historical scores of expert models can be obtained from historical data, which can reflect the historical performance of the expert model in improving the hybrid expert architecture and language model, and can also characterize the accuracy and professionalism of the expert model in processing query information to a certain extent.

[0068] This embodiment determines the fit based on domain relevance, similarity, historical scores, and corresponding dynamic weights. This can be expressed by the following formula.

[0069] S_i = α·sim(Q_emb, E_i_profile) + β·KG_relevance(Q, E_i) + γ·E_i_performance. Where sim(Q_emb, E_i_profile) represents the similarity between the semantic features of the query information and the expert's semantic profile. KG_relevance(Q, E_i) represents the domain relevance between the knowledge graph-based query information and the expert model. E_i_performance represents the expert's historical ratings on similar queries. α, β, and γ are dynamically adjusted weight coefficients.

[0070] The above method can accurately calculate the fit between the expert model and the query information, and can more accurately capture the relevance and hierarchy of professional domain knowledge, thus better meeting the needs of the professional domain.

[0071] As an optional implementation, expert models that meet the fitness requirements are included in the expert model set and screened to obtain an expert combination. This includes: determining multiple candidate subsets from the expert model set using a screening algorithm; calculating the knowledge coverage of the expert models in each candidate subset for the query information, as well as the knowledge redundancy between different expert models; and calculating the optimal candidate subset as the expert combination based on the knowledge coverage and knowledge redundancy using an optimal solution algorithm.

[0072] When selecting expert combinations from an expert model set, a screening algorithm is first used to identify multiple candidate subsets. The expert model set involves multiple expert models at different levels, and the collaborative effects of different expert models vary, especially considering the large number of possible combinations. Screening algorithms, such as the Top-K algorithm, can be used to select multiple candidate subsets that meet the requirements. This initial screening of potential expert models quickly narrows down the candidate pool, improving the efficiency of selecting expert combinations. After the initial screening, an exploration combination algorithm is used to generate more candidate subsets.

[0073] Then, for each candidate subset, the knowledge coverage of the expert model for the query information in each subset, as well as the knowledge redundancy between different expert models, are calculated. Based on the knowledge coverage and knowledge redundancy, the optimal candidate subset is calculated using an optimal solution algorithm to form the expert combination.

[0074] The knowledge coverage of the query information by multiple expert models in the aforementioned candidate subset can be measured by comparing the relevant knowledge nodes of the query information with the knowledge coverage of the combined knowledge base of the multiple expert models in the candidate subset. It should be noted that when determining the coverage of query postures based on relevant knowledge nodes, more relevant knowledge nodes can be extracted from the knowledge graph to ensure that the candidate subset provides sufficient knowledge coverage of the query information.

[0075] When determining the fit between the query information and the expert model, a higher threshold can be used to filter knowledge nodes in the knowledge graph, thereby reducing the number of relevant knowledge nodes. This ensures that the number of relevant knowledge nodes in the query information is not too different from the size of the expert model's knowledge base, and also guarantees a certain degree of accuracy in determining the fit. Based on knowledge coverage and knowledge redundancy, an optimal candidate subset is calculated using an optimal solution algorithm as the expert combination. This optimal solution algorithm can be an integer programming algorithm, a Monte Carlo tree search algorithm, or a greedy algorithm, etc. The optimal subset is then selected as the expert combination from multiple candidate subsets.

[0076] As described above, the method of inputting query information layer by layer in this embodiment is as follows.

[0077] As an optional embodiment, query information is sent to multiple expert models in an expert portfolio, and the multiple expert models in the expert portfolio output corresponding results, including: inputting query information into the first-layer expert model in the expert portfolio and outputting a first result, wherein the first layer is the first layer of the improved hybrid expert architecture; sending the first result to the next-layer expert model and outputting a second result, and then inputting the second result layer by layer to the next-layer expert model in the expert portfolio to obtain the output results of multiple expert models in each layer of the expert portfolio.

[0078] In this embodiment, as Figure 2 As shown, the improved hybrid expert architecture for the trade field includes at least: an upper professional dimension layer and a lower functional dimension layer. The expert models of the professional dimension layer include at least one of the following: customs expert, tax expert, logistics expert, and financial expert. The expert models of the functional dimension layer include at least one of the following: knowledge interpretation expert, rule reasoning expert, risk assessment expert, and decision support expert.

[0079] The aforementioned two-layer expert model allows for finer-grained expert segmentation, improving the accuracy of expert model application. Furthermore, it enables expert collaboration, achieving expert complementarity and joint reasoning, thereby enhancing the accuracy and professionalism of the improved hybrid expert architecture's large language model in processing query information.

[0080] like Figure 3As shown, as an optional embodiment, the query information is input into the expert model at the first layer of the expert portfolio, and a first result is output, including: embedding the query information into professional terms through a domain terminology embedding layer to obtain a professional terminology embedding result; processing the professional terminology embedding result using an adaptive multi-head attention layer to obtain a context-aware representation; mapping the structured professional knowledge in the context-aware representation to the model parameter space using a knowledge-parameter mapping engine to obtain a knowledge-enhanced representation; and obtaining the corresponding first result based on the knowledge-enhanced representation using an interpretable output layer. The next layer of the expert model receives the result from the upper layer and obtains the corresponding output result using the domain terminology embedding layer, the adaptive multi-head attention layer, the knowledge-parameter mapping engine, and the interpretable output layer.

[0081] The execution entity of the aforementioned domain-specific terminology embedding layer can be a domain-customized embedding matrix. Processing steps: Input: Original text (e.g., "FOB Shanghai Incoterms® 2020"). Processing: Using a pre-trained trade terminology embedding matrix (e.g., trained based on Incoterms®, UCP600, etc. corpora), the terminology is mapped into a high-dimensional vector. Semantic combination encoding is performed on compound terms (e.g., "FOB Shanghai"), preserving terminology integrity. Output change: Original word segmentation result ["FOB", "Shanghai"] → Vector fused with professional semantics [0.7, -0.2, ..., 1.4], which is the aforementioned terminology embedding result.

[0082] The aforementioned adaptive multi-head attention layer takes the above vectors as input and calculates the weight allocation for each attention head through domain attention gating. For example, trade term-related tokens (such as "CIF") are automatically assigned higher weights to the "Transportation Liability" attention head. Legal clause-related tokens (such as "Force Majeure") activate the "Legal Clause Resolution" attention head. Finally, a context-aware representation is generated, in which the relevance of technical terms is reinforced (such as the implicit association between "FOB" and "risk transfer").

[0083] The aforementioned expertise enhancement layer processing can include the knowledge-parameter mapping engine described above, which injects knowledge into the attention-treated context representation: querying a structured knowledge base (such as the Incoterms® rule tree) to convert clause logic into parameter constraints. Dynamic parameter adjustment: reconstructing network parameters based on activated knowledge units (such as trade terms modules and tariff calculation modules). Knowledge fusion: fusing the rule logic output (such as "FOB seller's liability cut-off port of shipment") with the neural network representation through residual connections. Enhancing the output of ordinary Transformer hidden layers into a representation infused with rule constraints.

[0084] The aforementioned interpretable output layer can leverage a traceable decoder to enhance the knowledge representation, enabling multi-granularity output and logical verification: a lightweight inference engine checks the consistency between the output and the knowledge base. For example, the input might be: "Payment by LC (Irrevocable Letter of Credit) must comply with UCP600 (Uniform Customs and Practice for Documentary Credits)." The expert module's processing includes: A word embedding layer labels "LC" as a letter of credit-specific vector (not a common abbreviation). An attention mechanism associates "UCP600" with the "Letter of Credit Operation Specification" attention head. A knowledge enhancement layer injects the parameter constraints of Article 9 of UCP600 regarding irrevocable letters of credit. The output is annotated with specific clauses of UCP600 as a basis.

[0085] This allows expert models to maintain the data-driven advantages of deep learning while strictly adhering to domain knowledge norms when processing specialized texts. It also ensures that data flow conforms to the forward propagation mechanism of deep learning while possessing the rule transparency of an expert system.

[0086] As an optional implementation, the outputs of multiple expert models are fused to obtain the answer to the query information, including: Assessing the credibility of the expert models and their outputs based on the expert models' professional coverage of the knowledge nodes of the query information, the accuracy of the expert models' historical outputs, and the certainty of the outputs for the query information; calculating the credibility of each output based on the credibility, the query relevance between the expert models' expertise vectors and the query vectors of the query information, and the complementarity parameters between different expert models; reconciling conflicting outputs based on the credibility to obtain an adjusted output; and determining the answer based on the adjusted output and its corresponding weights.

[0087] like Figure 6As shown, when fusing the outputs of multiple expert models in an expert ensemble, it is necessary to determine the weights of the experts to ensure the effectiveness and accuracy of the fusion and to reconcile conflicts to avoid deterioration of the fusion results. The weights can be calculated based on the credibility of the expert models. Specifically, based on the expert model's knowledge graph's professional coverage of the knowledge nodes of the query information, the accuracy of the expert model's historical outputs, and the certainty of the output results for the query information, the credibility of the expert models and their corresponding output results is evaluated to obtain credibility. Then, based on the credibility, the query relevance between the expert model's expertise vector and the query vector of the query information, and the complementarity parameters between different expert models, the weights of each output result are calculated.

[0088] Specifically, the calculation can be performed using the following formula.

[0089] C_i=ω1·Domain_Coverage_i+ω2·Historical_Accuracy_i+ω3·Certainty_ Score_i, in, C_i The credibility of expert model i. Domain_Coverage_i For professional coverage, Historical Accuracy_i For historical accuracy, Certainty_Score_i For the determinism of the current output, ω1, ω2, ω3 Each represents its corresponding weight.

[0090] Dynamic weight allocation: W_i= softmax(λ1·C_i+λ2·Relevance_i+λ3· Complementarity_i), Where λ1, λ2, and λ3 are learnable weights. softmax() Let be the probability distribution function. W_i For the dynamic weights of expert model i, Relevance_i To query relevance, Complementarity_i This refers to the collaborative characteristics of experts, also known as complementary parameters.

[0091] In the event of a conflict, reconciliation can be performed based on the level of confidence. Specifically, the conflict reconciliation function is used to coordinate when knowledge conflicts are detected among experts. These steps effectively address knowledge conflicts and complementarity issues in multi-expert collaboration, significantly improving the system's accuracy and reliability in handling complex queries.

[0092] As an optional implementation, based on the understanding results and a pre-stored knowledge graph, the query information is sent to multiple corresponding expert models. After the expert models obtain their corresponding output results, the process includes: calculating a self-evaluation score based on the expert models' knowledge coverage of the query information, as well as the reasoning reliability and quantification uncertainty of the output results; determining whether there are errors in the expert models' output results through expert peer review; if errors are found, the corresponding peer review experts provide error correction suggestions for error correction and feedback learning; and obtaining a consensus from the output results of multiple expert models through multiple rounds of peer review, which is then used for subsequent fusion to obtain the answer result.

[0093] like Figure 7 As shown, the experts' self-assessment is evaluated using the following indicators: Knowledge Coverage: Assessing the extent to which the expert covers the knowledge required for the current query. Reasoning Reliability: Assessing the logical rigor of the reasoning process. Uncertainty Quantification: Quantifying the uncertainty of the output.

[0094] The expert peer review and error correction process is as follows: Output Peer Review: The output of expert model A is reviewed by expert model B. Error Correction Suggestion Generation: When an error is found, specific error correction suggestions are generated. Error correction is initiated when there is a clear conflict between the judgment of expert model B and the output result A. Consensus Formation: Expert consensus is formed through multiple rounds of peer review and discussion. The maximum number of peer review rounds is 3. Feedback Learning is also possible: Feedback learning is conducted based on the error correction results to continuously improve the experts' capabilities.

[0095] This self-evaluation and error correction mechanism greatly enhances the model's self-testing ability and improves the system's reliability and credibility, making it particularly suitable for highly specialized and low-tolerance application scenarios such as trade.

[0096] As an optional embodiment, before sending query information to multiple corresponding expert models based on the understanding results and pre-stored knowledge graphs, and before the expert models obtain the corresponding output results, the method further includes: independently pre-training the expert models using training data matched to the professional domain; training the improved hybrid expert architecture routing module based on the pre-trained expert models, wherein the routing module is used to send query information to multiple corresponding expert models based on the understanding results and pre-stored knowledge graphs; and jointly optimizing the trained routing module and the pre-trained expert models using objective optimization functions, wherein the objective optimization functions include: an answer result loss function, a routing quality loss function, an expert diversity loss function, and a knowledge consistency loss function.

[0097] The above-described approach of pre-training the expert model and routing module before performing multi-objective optimization training allows for rapid and accurate model training, avoiding training difficulties. Furthermore, multi-objective optimization ensures balanced development of the model across various aspects, including task performance, routing quality, expert diversity, and knowledge consistency.

[0098] It can also optimize computational efficiency, improve model response rate, reduce latency, and reduce content usage, etc., which will be explained in detail in the implementation method.

[0099] It should be noted that this embodiment also provides an optional implementation method, specifically involving an example of the data processing method for large language models based on the improved hybrid expert architecture in practical application. This implementation method will be described in detail below.

[0100] This implementation provides a large language model based on an improved hybrid expert architecture, such as... Figure 2 As shown, this large language model mainly includes the following modules.

[0101] User query / input module: Receives queries or questions input by users.

[0102] Query understanding module: Analyzes the semantic intent of user queries and extracts key concepts and demand information.

[0103] Knowledge Graph Module: Stores concepts, relationships, and knowledge systems within a specific field, providing support for routing decisions.

[0104] Expert Router: Based on query understanding results and knowledge graph information, it distributes queries to suitable expert models.

[0105] Domain Layer: A collection of expert models categorized by professional field, such as customs experts, tax experts, logistics experts, and financial experts.

[0106] Function Layer: A collection of expert models categorized according to functional requirements, such as knowledge interpretation, rule reasoning, risk assessment, and decision support.

[0107] Expert Fusion and Output Module: Integrates the outputs of multiple expert models to generate the final answer.

[0108] This implementation adopts a two-tiered expert organizational structure, dividing the expert model into a professional dimension layer and a functional dimension layer, which can simultaneously meet the needs of both depth of professional knowledge and diversity of functions. This hierarchical hybrid expert architecture better adapts to the layered knowledge system of the professional domain, improving the model's knowledge coverage and accuracy within that domain. Furthermore, each layer of expert models can be based on a refined division of experts within the professional domain, giving the expert models clear knowledge boundaries and professional characteristics, reducing knowledge overlap and gaps.

[0109] By employing an expert routing mechanism that understands the semantics of query results, the system accurately interprets the intent of professional queries and distributes them to the most suitable expert group. Finally, an efficient expert collaboration framework is constructed to support complementarity, error correction, and collaborative reasoning among experts, enhancing the ability to solve complex problems. This approach is more suitable for specialized fields and significantly improves the application effectiveness of large models in those fields.

[0110] like Figure 3 As shown, the overall data processing flow of the large language model based on the improved hybrid expert architecture is as follows.

[0111] User query input: Receives query information input by the user.

[0112] Query understanding: The system analyzes the semantic intent of the query information and extracts key concepts and requirements.

[0113] Expert routing: Selecting a suitable expert model based on query understanding results and knowledge graph.

[0114] Professional dimension processing: Selected experts in the professional field process the query and provide professional knowledge output.

[0115] Functional dimension processing: The selected functional expert processes the query and executes specific functional tasks.

[0116] Expert self-assessment and peer review: Experts assess the quality of their own output, review and correct each other's work, and reach a consensus.

[0117] Expert fusion and final output: Integrate the outputs of multiple experts to generate the final answer.

[0118] Throughout the process, the knowledge graph is pre-created for specific professional fields, providing necessary knowledge support for query understanding, expert routing, and professional knowledge processing.

[0119] Specifically, the improved hybrid expert architecture described above can be called Hierarchical Mixture-of-Experts (HMoE). It innovatively divides the complex knowledge system of trade into two levels according to professional and functional dimensions, constructing a more refined expert collaboration mechanism. It exhibits better professional performance for trade-related queries.

[0120] 1. Two-tiered division of professional and functional dimensions: The first tier divides expert models according to professional dimensions such as customs, taxation, logistics, and finance; the second tier divides expert models according to functional dimensions such as knowledge interpretation, rule reasoning, and risk assessment. This two-tiered design can more accurately address the complexity and interdisciplinary nature of the trade knowledge system.

[0121] 2. Knowledge Graph-Based Expert Routing Mechanism: This innovative approach introduces domain-specific knowledge graphs to assist expert routing decisions. Compared to traditional statistics-based routing mechanisms, it can more accurately identify query intent and select appropriate expert combinations. Experiments show that this mechanism can improve expert selection accuracy by 15-20%.

[0122] 3. Adaptive expert fusion algorithm: An adaptive fusion algorithm that considers expert credibility, knowledge coverage and reasoning ability is designed. It can dynamically adjust the weight contribution of different experts according to the complexity of the query and the domain involved, which solves the limitations of fixed weight fusion in traditional MoE.

[0123] 4. Expert self-assessment and error correction mechanism: An innovative self-assessment mechanism for expert capabilities is introduced, allowing experts to rate the credibility of their own answers and achieve mutual error correction and supplementation in multi-expert collaboration, which significantly improves the accuracy and reliability of the overall system.

[0124] Current mainstream MoE architectures, such as Google's GShard and Switch Transformer, primarily focus on improving computational efficiency and general language capabilities, while paying insufficient attention to the refined expression of specialized knowledge in vertical domains. The HMoE architecture proposed in this implementation is a significant innovation in the application of existing MoE technologies in specialized fields, and is particularly suitable for complex professional scenarios such as trade, which have a vast knowledge system and numerous subdomains.

[0125] The above-mentioned two-level division of professional and functional dimensions mainly includes the professional dimension layer (Domain Layer) and the functional dimension layer (Function Layer).

[0126] The professional dimension layer categorizes expert models according to different sub-fields of trade, including customs experts, tax experts, logistics experts, and financial experts; the functional dimension layer categorizes expert models according to functional requirements such as knowledge interpretation, rule reasoning, and risk assessment. This two-tiered division allows the models to possess both professional depth and functional flexibility.

[0127] Each expert model internally employs a customized Transformer architecture, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the expert model architecture according to an embodiment of the present invention. The expert model includes the following core components.

[0128] Domain-specific terminology embedding layer: This layer uses word embeddings designed for trade terminology to enhance understanding of the terminology.

[0129] Adaptive Multi-Head Attention Layer: A multi-head attention layer that can adaptively adjust the distribution of attention according to the characteristics of a professional field.

[0130] Professional knowledge enhancement layer: A special layer structure that integrates pre-trained knowledge and professional knowledge.

[0131] Interpretable output layer: The output layer that supports the tracking of the reasoning process and the annotation of the basis for decision-making.

[0132] The key technical point is that the professional knowledge enhancement layer adopts the "knowledge-parameter mapping engine" technology, which explicitly maps structured professional knowledge (such as trade term definitions, rule clauses, etc.) to the model parameter space, so that the model can maintain the flexibility of the neural network and have the ability to accurately express professional knowledge.

[0133] The aforementioned expert router innovatively combines knowledge graph technology with expert routing mechanisms, designing a dual-drive routing algorithm of "semantic perception + graph reasoning". Figure 5 This is a schematic diagram of the routing module architecture of an embodiment of the present invention, as shown below. Figure 5 As shown, the routing mechanism consists of four core components.

[0134] Semantic understanding module: Analyzes the semantic intent of user queries and extracts key concepts and demand types.

[0135] Knowledge Graph Query Module: Maps user queries to the trade knowledge graph and identifies relevant knowledge nodes.

[0136] Expert Mapping Module: Based on knowledge nodes and query types, determine the set of expert models that need to be activated.

[0137] Decision integration module: Determines the final expert combination strategy based on query complexity and professional coverage requirements.

[0138] The core routing algorithm consists of three key steps.

[0139] Semantic space mapping: Mapping query content to a high-dimensional semantic space through a semantic encoder.

[0140] Expert fit calculation: Calculate the fit score between the query and each expert model.

[0141] S_i = α·sim(Q_emb, E_i_profile) + β·KG_relevance(Q, E_i) + γ·E_i_performance. Where S_i is the fit between expert model i and the query information. sim(Q_emb, E_i_profile) is the similarity between the semantic features of the query information and the expert's semantic profile. KG_relevance(Q, E_i) is the domain relevance between the knowledge graph-based query information and the expert model. E_i_performance is the expert's historical rating on similar queries. α, β, and γ are dynamically adjusted weight coefficients.

[0142] Optimal expert combination selection: The final expert combination is determined based on Top-K screening and collaborative scoring mechanism.

[0143] E_selected = argmax_{E_subset} [Coverage(E_subset) - Redundancy(E_subset)]. Here, Coverage(E_subset) evaluates the knowledge coverage of the query by the expert subsets. Redundancy(E_subset) evaluates the knowledge redundancy between the expert subsets.

[0144] Compared to traditional statistical learning-based routing methods, this knowledge graph-based expert routing mechanism can more accurately capture the correlation and hierarchy of knowledge in a professional field, and better meet the needs of that field.

[0145] The aforementioned expert fusion and output module employs an adaptive expert fusion algorithm, which features "dynamic weights, credibility awareness, and conflict coordination." Figure 6 This is a schematic diagram of the fusion architecture of the output results of an embodiment of the present invention, as shown below. Figure 6 As shown, the adaptive expert fusion algorithm consists of four key modules.

[0146] Expert credibility assessment: Expert credibility is assessed based on the expert's professional coverage in the field, historical accuracy, and certainty of current output.

[0147] Dynamic weight calculation: The weight of each expert is dynamically calculated based on the credibility score, query relevance, and expert collaboration characteristics.

[0148] Conflict detection and coordination: Automatically detect and coordinate knowledge conflicts between different expert outputs.

[0149] Fusion strategy optimization: Continuously optimize the fusion strategy based on feedback from the fusion results.

[0150] The core mathematical model for expert fusion is shown below.

[0151] Expert credibility calculation: The following formula is used for calculation.

[0152] C_i=ω1·Domain_Coverage_i+ω2·Historical_Accuracy_i+ω3·Certainty_ Score_i, in, C_i The credibility of expert model i. Domain_Coverage_i For professional coverage, Historical Accuracy_i For historical accuracy, Certainty_Score_i For the determinism of the current output, ω1, ω2, ω3 Each represents its corresponding weight.

[0153] Dynamic weight allocation: W_i= softmax(λ1·C_i+λ2·Relevance_i+λ3· Complementarity_i), Where λ1, λ2, and λ3 are learnable weights. softmax() Let be the probability distribution function. W_i For the dynamic weights of expert model i, Relevance_i To query relevance, Complementarity_i This refers to the collaborative characteristics of experts, also known as complementary parameters.

[0154] Conflict reconciliation function: When a knowledge conflict is detected among experts, a reconciliation function is used to coordinate the conflict.

[0155] O_harmonized = Harmonize({O_i}, {W_i}, Conflict_Matrix). Here, O_harmonized is the harmonic output, Conflict_Matrix captures the conflict relationships between expert outputs, O_i is the output of expert model i, and Harmonize() is the harmonic function.

[0156] The final output is calculated as: O_final = ∑(W_i·O_i) + Adjustment(O_harmonized). Here, O_final is the fused answer, and Adjustment() is the adjustment function.

[0157] This adaptive fusion algorithm can effectively handle knowledge conflicts and complementarity issues in multi-expert collaboration processes, significantly improving the accuracy and reliability of the system in processing complex queries.

[0158] The above data processing flow also includes expert self-assessment and mutual review steps, which are based on the expert capability self-assessment and error correction mechanism provided by this implementation method for large language models. Figure 7 This is a schematic diagram of the self-evaluation and mutual evaluation architecture of an embodiment of the present invention, as shown below. Figure 7 As shown, the self-evaluation mechanism is as follows.

[0159] Each expert model has self-evaluation capabilities, assessing the quality of its output using the following metrics:

[0160] Knowledge coverage: Assess the extent to which experts have the knowledge required for the current query.

[0161] Coverage_score = Overlap(Query_knowledge_req, Expert_knowledge). Here, Coverage_score is the knowledge coverage, Query_knowledge_req is the knowledge requirement for the query, Expert_knowledge is the knowledge base of the expert model, and Overlap() is the overlap function.

[0162] Reasoning reliability: assessing the logical rigor of the reasoning process.

[0163] Reliability_score = Consistency(Reasoning_steps) * Completeness(Evidence_chain). Where Reliability_score represents the reliability of the reasoning, Consistency(Reasoning_steps) represents the consistency of the reasoning steps, and Completeness(Evidence_chain) represents the completeness of the chain of evidence.

[0164] Uncertainty quantification: Quantifying the uncertainty of the output.

[0165] Uncertainty = 1 - confidence(Output) + entropy(Distribution). Where Uncertainty is the uncertainty score, confidence(Output) is the output confidence score, and entropy(Distribution) is the distribution entropy.

[0166] The expert peer review and error correction process is as follows.

[0167] Output peer review: The output of expert model A is reviewed by expert model B.

[0168] The `Review(Expert_B, Output_A)` function returns a peer review result. Here, `Review()` is the peer review feedback function, `Expert_B` represents expert B's judgment, and `Output_A` represents the output result A. `Correct` indicates that expert model B's judgment is completely consistent with expert model A's output result A. `Incorrect` indicates that expert model B's judgment and output result A have a clear conflict. `Uncertain` indicates that expert model B's judgment is partially consistent with output result A, but there is ambiguity or insufficient evidence.

[0169] Error correction suggestion generation: When an error is found, specific error correction suggestions are generated. `if Review == Incorrect: Correction_suggestion = Generate_correction(Expert_B, Output_A)`. Here, `Review == Incorrect` means that the error correction is triggered when the peer review feedback result is `Incorrect`, that is, when there is a clear conflict between the judgment of expert model B and the output result A. `Generate_correction()` is the function to generate error correction suggestions. `Correction_suggestion` is the error correction suggestion.

[0170] Consensus Formation: Expert consensus was formed through multiple rounds of mutual review and discussion.

[0171] Consensus = Deliberation({Experts}, {Outputs}, {Reviews}, max_rounds=3). Here, Consensus is the overall consistency score, {Experts} is the set of experts, {Outputs} is the set of expert outputs, {Reviews} is the set of peer review feedback, and max_rounds=3 indicates a maximum of 3 rounds of peer review.

[0172] Feedback learning: Based on the results of error correction, feedback learning is conducted to continuously improve the capabilities of experts.

[0173] `Update_expert(Expert_A, Correction_feedback)`. Here, `Expert_A` is the expert model A, `Correction_feedback` is the correction feedback, and `Update_expert()` updates the knowledge of the expert model.

[0174] This self-evaluation and error correction mechanism greatly enhances the model's self-testing ability and improves the system's reliability and credibility, making it particularly suitable for highly specialized and low-tolerance application scenarios such as trade.

[0175] The large language model based on the HMoE architecture described above also requires phased training. Considering the characteristics of the HMoE architecture, this implementation devises a three-stage training strategy, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the training architecture of an embodiment of the present invention.

[0176] Specifically, this includes: Expert pre-training stage: Each expert model is independently pre-trained based on domain-specific data.

[0177] Router training phase: Fix expert parameters and train an expert router.

[0178] Overall fine-tuning phase: Jointly optimize the expert model and router, where α is a small learning rate coefficient to prevent overfitting.

[0179] Multi-objective joint optimization: A multi-objective joint optimization method is adopted during training, and the objective function is designed as follows.

[0180] L=ζ1·L_task+ζ2·L_routing+ζ3·L_diversity+ζ4·L_consistency .in: L_ task The main task loss function; L_routing This is the routing quality loss function; L_diversity For expert diversity loss function; L_consistency The knowledge consistency loss function; ζ1, ζ2, ζ3, ζ4 To balance the weighting coefficients of each loss term.

[0181] This multi-objective optimization ensures the model's balanced development in terms of task performance, routing quality, expert diversity, and knowledge consistency.

[0182] Furthermore, computational efficiency can be optimized. To improve the computational efficiency of HMoE, the following techniques were adopted in this project.

[0183] Sparse activation: Each inference only activates a portion of the experts (usually 2-4), significantly reducing computational overhead.

[0184] Expert Parallel Computing: Utilizing the parallel capabilities of GPUs to simultaneously compute the outputs of multiple experts.

[0185] Dynamic batch processing: Adaptively adjusts the batch size based on query complexity.

[0186] Model quantization: Int8 quantization is performed on non-critical experts to reduce memory usage.

[0187] Through these optimization techniques, the HMoE architecture maintains high accuracy while keeping inference latency within the range of 100-500ms, meeting the needs of real-time applications.

[0188] The aforementioned knowledge graph is a crucial supporting technology for the HMoE architecture. This implementation also constructs a dedicated trade knowledge graph. The construction of the trade knowledge graph includes the following steps.

[0189] Knowledge extraction: Extracting entities and relationships from authoritative sources such as trade regulations, agreement texts, and professional books.

[0190] Knowledge integration: Integrating knowledge from different sources to eliminate redundancy and contradictions.

[0191] Knowledge reasoning: Expanding the knowledge graph through rule-based reasoning.

[0192] Knowledge verification: Verify the correctness of knowledge through expert review and automatic consistency checks.

[0193] Figure 9 This is a schematic diagram of the knowledge graph architecture of an embodiment of the present invention. The core structure of the trade knowledge graph is as follows: Figure 9 As shown, it includes the following main entity types: Term, Rule, Condition, Organization, Country / Region, Product, and Process. It also supports a rich set of relationship types, such as: define, contain, depend on, apply to, exempt, amend, and replace.

[0194] Finally, in the engineering implementation and deployment of the large language model of the HMoE architecture described above, this embodiment follows the following... Figure 10 The architecture shown includes the following core services: Expert Service Cluster: A service cluster where each expert model is deployed independently. Routing Service: Responsible for request distribution and expert selection. Fusion Service: Handles the integration of multiple expert outputs. Knowledge Graph Service: Provides knowledge query and reasoning support. Quality Assessment Service: Implements self-assessment and error correction functions. API Gateway: A unified service entry point and interface management system.

[0195] Specifically, this implementation also provides the following key parameters for the HMoE system implementation in the trade field, which has already been specifically implemented.

[0196] The system's hardware environment utilizes a distributed computing cluster, comprising 16 GPU nodes (NVIDIA A10080GB), each equipped with 128GB of memory and 1TB of SSD storage. The software environment is based on the PyTorch framework, using Python 3.8 for model training and inference.

[0197] Expert Model Classification: Specialized Dimensions: Customs Regulations Expert: Focuses on national trade policies and tariff calculations. Trade Agreement Expert: Handles trade rules and agreements. Logistics and Transportation Expert: Optimizes international transportation routes and costs.

[0198] Functional Dimension Layer: Risk Assessment Expert: Analyzes trade risks. Contract Review Expert: Analyzes trade contract terms. Decision Support Expert: Generates trade strategy recommendations.

[0199] Knowledge Graph Construction: Data Sources: Trade databases, publicly available trade documents, and historical transaction records of enterprises. Construction Tool: Neo4j graph database, containing 100,000+ nodes (concepts) and 500,000+ edges (relationships). This facilitates the convenient and rapid construction of a high-coverage knowledge graph.

[0200] Expert Routing Mechanism: Routing Algorithm: Two-layer routing based on attention mechanism (Top-K professional layer experts + Top-M functional layer experts). Example: When a user inputs "Tariff and risk analysis for exporting electronic products from country A to country B", the routing is directed to customs regulation experts (professional layer) and risk assessment experts (functional layer).

[0201] Training and Optimization: Training data: 1 million trade-related question-and-answer pairs and documents. Loss function: Cross-entropy loss + expert load balancing loss (λ=0.1). Training period: 50 epochs (training rounds), batch size 256.

[0202] The hierarchical hybrid expert architecture large model system and method proposed in this embodiment bring the following significant benefits through innovative technical solutions.

[0203] Significantly improved coverage and accuracy of professional knowledge: Through a two-tiered expert organizational structure and refined expert division, this invention can more comprehensively and accurately cover professional field knowledge.

[0204] The reasoning ability for complex problems is greatly enhanced: Through the knowledge graph-based expert routing mechanism and adaptive expert fusion algorithm, this implementation method can handle complex problems more accurately, especially those involving the intersection of knowledge from multiple fields.

[0205] Significantly improved reliability and stability: Through expert capability self-assessment and error correction mechanisms, this implementation method significantly reduces the system error rate.

[0206] Optimization of computational resource utilization efficiency: Through techniques such as sparse activation and parallel computing, this implementation method significantly reduces computational overhead while maintaining model performance.

[0207] Enhanced interpretability: This implementation improves the transparency of system decision-making through an interpretability output layer and an evidence chain tracing mechanism.

[0208] Enhanced adaptability and scalability: The modular design and domain knowledge enhancement mechanism of this implementation make it easier for the system to adapt to new professional fields and tasks.

[0209] Based on the above-described data processing method for large language models based on an improved hybrid expert architecture provided by the embodiments of the present invention, the embodiments of the present invention also provide a data processing apparatus for large language models based on an improved hybrid expert architecture, the apparatus comprising:

[0210] The input module is used to process the input query information through the semantic understanding module to obtain the understanding result.

[0211] The routing module is used to send query information to multiple expert models based on the understanding results and the pre-stored knowledge graph. The expert models then obtain the corresponding output results. The multiple expert models are set up in multiple layers based on an improved hybrid expert architecture. The expert models in different layers are divided according to different expert dimensions. At least one expert model in each layer processes the query information and obtains the output results. The professional domain of the knowledge graph matches the professional domain of the query information.

[0212] The fusion module is used to merge the outputs of multiple expert models to obtain the answer corresponding to the query information.

[0213] The data processing device based on the improved hybrid expert architecture and large language model described in this embodiment employs a multi-layered expert model divided according to different expert dimensions. It obtains output results based on query information and then fuses these output results to obtain the corresponding answer, resulting in higher accuracy and professionalism. Furthermore, by combining the semantic understanding results of the query information with a knowledge graph matching the professional domain, the query information is routed and sent to the expert model in the improved hybrid expert architecture, further enhancing accuracy.

[0214] Embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0215] Embodiments of the present invention also provide a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of an embodiment of the present invention.

[0216] An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of the embodiment of the present invention.

[0217] refer to Figure 11 The present invention will now describe a structural block diagram of an electronic device that can serve as an embodiment of the present invention, serving as an example of a hardware device applicable to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0218] like Figure 11 As shown, the electronic device includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the electronic device. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0219] Multiple components in the electronic device are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information into the electronic device. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, disks and optical discs. Communication unit 1109 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0220] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as computer programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1102 and / or communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0221] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0222] In the context of embodiments of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0223] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A data processing method based on an improved hybrid expert architecture for large language models, characterized in that, include: The input query information is processed by the semantic understanding module to obtain the understanding result; Based on the understanding results, the query information is mapped to a knowledge graph to determine the relevant knowledge nodes of the query information, wherein the professional field of the knowledge graph matches the professional field of the query information; Based on the relevant knowledge nodes, the domain relevance between the query information and the expert model is calculated; Determine the similarity between the semantic features in the understanding results of the query information and the semantic profile of the expert model; Obtain the historical scores of the expert model for similar queries to which the query information belongs; The fit is determined based on the domain relevance, the similarity, and the historical scores, combined with the corresponding dynamic weights. Expert models that meet the suitability requirements are included in the expert model set, and then filtered based on the knowledge coverage and knowledge redundancy of the expert models to obtain the expert combination. The query information is sent to multiple expert models in the expert combination, and the multiple expert models in the expert combination output the corresponding output results. The multiple expert models are set up with multiple layers based on an improved hybrid expert architecture. The expert models in different layers are divided according to different expert dimensions. At least one expert model in each layer processes the query information and obtains the output results. A self-evaluation score is calculated based on the expert model's knowledge coverage of the query information, as well as the reasoning reliability and quantification uncertainty of the output results. Through peer review by experts, it is determined whether there are any errors in the output results of the expert model. If errors are found, the corresponding peer review experts will provide error correction suggestions for error correction and feedback learning. Through multiple rounds of mutual review, the outputs of several expert models that reach a consensus are obtained and used for subsequent fusion to obtain the answer. The outputs of multiple expert models are fused to obtain the answer corresponding to the query information; Specifically, the query information is sent to multiple expert models in the expert group, and the multiple expert models in the expert group output corresponding results, including: The query information is input into the expert model at the first layer of the expert portfolio, and a first result is output, wherein the first layer is the first layer of the improved hybrid expert architecture; The first result is sent to the expert model at the next level, and the second result is output. The second result is then input into the expert model at the next level in the expert combination layer by layer to obtain the output results of multiple expert models in the expert combination at each level. The improved hybrid expert architecture includes at least an upper professional dimension layer and a lower functional dimension layer. The expert models of the professional dimension layer include at least one of the following: customs expert, tax expert, logistics expert, and financial expert. The expert models of the functional dimension layer include at least one of the following: knowledge interpretation expert, rule reasoning expert, risk assessment expert, and decision support expert.

2. The method according to claim 1, characterized in that, Expert models that meet the suitability requirements are included in the expert model set, and then filtered based on the knowledge coverage and knowledge redundancy of the expert models to obtain an expert portfolio, including: Multiple candidate subsets are determined from the set of expert models using a screening algorithm; Calculate the knowledge coverage of the expert model in each candidate subset for the query information, and the knowledge redundancy between different expert models; Based on the knowledge coverage and the knowledge redundancy, the optimal candidate subset is calculated using the optimal solution algorithm as the expert combination.

3. The method according to claim 1, characterized in that, The query information is input into the expert model at the first level of the expert portfolio, and a first result is output, including: The query information is embedded with professional terms through a domain-specific terminology embedding layer to obtain the professional terminology embedding result. An adaptive multi-head attention layer is used to process the terminology embedding results to obtain a context-aware representation; For the context-aware representation, a knowledge-parameter mapping engine is used to map the structured professional knowledge in the context-aware representation to the model parameter space to obtain a knowledge-enhanced representation; Based on the knowledge-enhanced representation, an interpretable output layer is used to obtain the corresponding first result; The next-layer expert model receives the results from the upper layer and obtains the corresponding output results using the domain terminology embedding layer, the adaptive multi-head attention layer, the knowledge-parameter mapping engine, and the interpretable output layer.

4. The method according to claim 1, characterized in that, The outputs of multiple expert models are fused to obtain the answer result corresponding to the query information, including: Based on the expert model's knowledge graph's professional coverage of the knowledge nodes of the query information, the accuracy of the expert model's historical outputs, and the certainty of the output results for the query information, the credibility of the expert model and its corresponding output results is evaluated to obtain the credibility. Based on the credibility, the query relevance between the expert model's expertise vector and the query vector of the query information, and the complementarity parameters between different expert models, the weights of each output result are calculated. Based on the aforementioned credibility, conflicting output results are reconciled to obtain adjusted output results. The answer result is determined based on the adjusted output and the corresponding weights.

5. The method according to claim 1, characterized in that, Based on the understanding results and the pre-stored knowledge graph, the method further includes sending the query information to multiple corresponding expert models before the expert models obtain the corresponding output results: The expert model is independently pre-trained using training data matched to the professional field; Based on the pre-trained expert model, the routing module of the improved hybrid expert architecture is trained, wherein the routing module is used to send the query information to the corresponding multiple expert models based on the understanding results and the pre-stored knowledge graph. Based on the trained routing module and the pre-trained expert model, joint optimization is performed through objective optimization functions, which include: response result loss function, routing quality loss function, expert diversity loss function, and knowledge consistency loss function.

6. A computer program product comprising a computer program, wherein, A computer program, when executed by a computer's processor, is used to cause the computer to perform the method according to any one of claims 1 to 5.

7. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 5.

Images