Expert clustering method and system for cooperation among multiple models

Abstract

The invention relates to the technical field of natural language processing, and relates to an expert clustering method and system for cooperation among multiple models, and the method comprises the steps: obtaining an input instruction which is transmitted by a user side and is configured with a user identifier, and obtaining a historical interaction context sequence; generating a task query vector, and calculating a similarity value of the feature vector of each expert node; screening out a target expert node, and transmitting the input instruction and the historical interaction context sequence to the target expert node; obtaining a task execution track feature vector of the target expert node; performing track feature vector aggregation calculation on all tasks to obtain global fusion features; and calculating conditional probability distribution of each candidate word pair in a preset output word list of the decoder, and generating a target text sequence according to the conditional probability distribution. According to the method, the current task demand of the user is taken as a reference, and the reasoning logic quality of each expert node is dynamically evaluated and subjected to weighted fusion, so that the accuracy of finally generating the text is improved.

Description

Technical Field

[0001] This invention generally relates to the field of natural language processing technology. More specifically, this invention relates to a method and system for expert cluster collaboration among multiple models. Background Technology

[0002] Existing large language models have demonstrated strong general language understanding and generation capabilities. However, with the exponential growth in the number of model parameters, single ultra-large-scale models are gradually encountering bottlenecks in practical industrial deployments. On the one hand, a single model may lead to parameter redundancy, and in vertical fields such as medicine, law, and precision manufacturing, general-purpose models often suffer from poor accuracy due to a lack of deep professional knowledge. On the other hand, single large models have high inference costs and large response latency, making it difficult to meet real-time requirements.

[0003] Therefore, leveraging collaborative expert models from multiple domains has become a mainstream approach for solving complex tasks. This architecture attempts to compensate for the shortcomings of a single model by integrating the knowledge from multiple models.

[0004] However, existing multi-model collaboration techniques typically employ shallow fusion strategies at the result level, such as directly voting on the output texts of multiple models or simply averaging the final probability distribution. Because they cannot perceive the intermediate reasoning logic that leads the models to their conclusions, when faced with complex logical tasks, even if some models guess the correct answer but their reasoning process is flawed, or if different models present conflicting reasoning paths, existing systems struggle to distinguish between superior and inferior models, resorting only to mechanical fusion. This can result in logical inconsistencies, contradictions, or even incorrect answers due to the adoption of flawed reasoning paths. Summary of the Invention

[0005] To address the technical problem of potentially low accuracy and consistency in the output of multi-model collaboration, the present invention provides solutions in the following aspects.

[0006] In a first aspect, a method for multi-model collaboration among expert clusters includes: obtaining an input instruction sent by a user terminal, configured with a user identifier, and obtaining a historical interaction context sequence indexed from a database based on the user identifier; vectorizing the input instruction to generate a task query vector, and calculating the similarity value between the task query vector and the feature vectors of each expert node in a preset expert model library; selecting the expert node whose similarity value is greater than a preset threshold as the target expert node, and transmitting the input instruction and the historical interaction context sequence to the target expert node; inputting the input instruction and the historical interaction context sequence to each of the target expert nodes, causing each target expert node to generate an intermediate inference text sequence according to an autoregressive generation strategy, and mapping the intermediate inference text sequence into a high-dimensional feature vector, denoted as the task execution trajectory feature vector of the target expert node; aggregating and calculating the task execution trajectory feature vectors output by all target expert nodes through a multi-head attention mechanism to obtain a global fusion feature; mapping the global fusion feature to the input layer of a pre-trained decoder, calculating the conditional probability distribution of each candidate word pair in a preset output vocabulary of the decoder, and generating a target text sequence based on the conditional probability distribution.

[0007] Preferably, the database is used to store historical session records and their corresponding timestamps in key-value pairs, with the user identifier as the index key. Indexing and extracting the corresponding historical interaction context sequence from the database includes: obtaining all historical session records associated with the user identifier in the database, and constructing a candidate context list in order of timestamp from most recent to oldest; calculating the semantic relevance score between the input command and each historical session record in the candidate context list; sorting each historical session record according to the product of the semantic relevance score and the time decay factor, and selecting the first preset number of historical session records as the historical interaction context sequence.

[0008] Preferably, calculating the semantic relevance score between the input command and historical session records includes: extracting features from the input command to obtain a task feature vector and a first keyword set corresponding to the input command; extracting features from the historical session records to obtain a historical feature vector and a second keyword set corresponding to the historical session records; calculating the vector space similarity between the task feature vector and the historical feature vector; calculating the feature word overlap between the first keyword set and the second keyword set; and performing a weighted sum based on the vector space similarity and the feature word overlap to obtain the semantic relevance score; wherein the semantic relevance score is related to the vector space similarity and the feature word overlap.

[0009] Preferably, the formula for calculating the semantic relevance score is: .

[0010] in S The semantic relevance score is given. The feature vector of the task. The historical feature vector, For the first set of keywords, This is the second set of keywords. The intersection operator represents the set intersection operator. This represents the number of keywords shared by the first keyword set and the second keyword set. The union operator represents the set union operator. This represents the total number of unique keywords contained in both the first and second keyword sets. μ This is a preset retrieval balance coefficient, and 0 < μ <1.

[0011] Preferably, the process of vectorizing the input instruction to generate a task query vector includes: segmenting the input instruction using a preset language model word segmenter to obtain an instruction word embedding sequence; inputting the instruction word embedding sequence into a bidirectional encoder model for semantic feature extraction to obtain the hidden state vector output by the last layer of the bidirectional encoder model, wherein the bidirectional encoder model includes multiple vertically stacked encoder network layers; performing global average pooling on the hidden state vector sequence to obtain a global pooling vector; and normalizing the global pooling vector output by the last layer of the bidirectional encoder model to obtain the task query vector.

[0012] Preferably, mapping the intermediate inference text sequence to a high-dimensional feature vector to obtain the task execution trajectory feature vector includes: obtaining the hidden layer state for each generation step during the generation of the intermediate inference text sequence by the target expert node; and performing weighted aggregation calculation on the hidden layer states according to the temporal position of the generation step to obtain the task execution trajectory feature vector, wherein the formula for weighted aggregation calculation of the hidden layer states is: .

[0013] V traj This is the feature vector of the task execution trajectory. L This represents the total generation step size of the intermediate inference text sequence. t Indicates the current generation step number; λ This represents the time decay factor used to control the degree to which historical information is retained. H t Indicates the target expert node at the th t Each hidden layer state vector generates a step-size output; E finalThis represents the embedding vector corresponding to the end identifier in the intermediate inference text sequence; δ This indicates the preset reinforcement weight coefficient of the end identifier.

[0014] Preferably, the global fusion feature is obtained by aggregating and calculating the task execution trajectory feature vectors output by all target expert nodes through a multi-head attention mechanism, including: using the task query vector as the input source of the query matrix, and using the task execution trajectory feature vectors output by each target expert node as the input sources of the key matrix and the value matrix, respectively; calculating the dot product attention score of the query matrix and the key matrix in multiple attention heads, and performing a weighted summation of the value matrix using the dot product attention score to obtain the subspace features of each attention head; concatenating the subspace features of each attention head and then performing dimensional projection through a linear transformation layer to obtain the global fusion feature.

[0015] Preferably, calculating the conditional probability distribution of each candidate word pair in the decoder output vocabulary includes: inputting the global fusion features into the output mapping layer of the decoder, so that the decoder maps the feature data to a candidate word score space with the same dimension as the preset output vocabulary, to obtain the original score vector of each candidate word; adjusting the smoothness of the original score vector to obtain the adjusted score vector; and performing exponential normalization on the adjusted score vector to obtain the conditional probability distribution corresponding to each candidate word in the preset output vocabulary, wherein the conditional probability distribution includes the probability value of each candidate word.

[0016] Preferably, generating the target text sequence based on the conditional probability distribution includes: constructing a candidate word sorting set in descending order of probability values ​​of each candidate word in the conditional probability distribution; sequentially accumulating the probability values ​​in the candidate word sorting set until the accumulated value meets a preset truncation threshold, retaining candidate words within the truncation threshold range to form an effective sampling set; normalizing the probability values ​​of candidate words in the effective sampling set to obtain the candidate sampling weights; performing probabilistic sampling from the effective sampling set according to the sampling weights to obtain a preset number of target output words, and generating a target text sequence based on all target output words.

[0017] In a second aspect, a multi-model collaborative expert cluster system includes a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement a multi-model collaborative expert cluster method as described in any of the foregoing inventions. The beneficial effects of this invention are as follows: This invention constructs a task execution trajectory feature vector, which can capture the hidden layer state and temporal logic of each expert node during the generation process. Combined with a multi-head attention mechanism, it dynamically evaluates and weights the reasoning logic quality of each expert based on the current task requirements, thereby improving the logical rigor and accuracy of the final generated text.

[0018] This invention employs a probabilistic sampling strategy based on truncation thresholds and sampling weights during the decoding stage. Compared to traditional greedy search, this mechanism eliminates low-probability outrageous results while preserving a reasonable range of diverse expressions, effectively avoiding rigidity and repetition in model-generated content, and resulting in a more natural output. Attached Figure Description

[0019] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart illustrating the steps of an expert cluster method for multi-model collaboration according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the structure of an expert cluster system for multi-model collaboration according to this embodiment. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic flowchart illustrating the steps of an expert cluster method for multi-model collaboration according to an embodiment of the present invention.

[0023] like Figure 1 As shown, an expert cluster method for multi-model collaboration includes steps S1 to S6.

[0024] Step S1: Obtain the input instruction sent by the user terminal, which is configured with a user identifier, and retrieve the historical interaction context sequence from the database based on the user identifier.

[0025] In one embodiment, the database is used to store historical session records and their corresponding timestamps in key-value pairs, with the user identifier as the index key. Indexing and extracting the corresponding historical interaction context sequence from the database includes: obtaining all historical session records associated with the user identifier in the database, and constructing a candidate context list in order of timestamp from most recent to oldest; calculating the semantic relevance score between the input command and each historical session record in the candidate context list; sorting each historical session record according to the product of the semantic relevance score and the time decay factor, and selecting the first preset number of historical session records as the historical interaction context sequence.

[0026] It should be noted that historical conversation records are a collection of question-and-answer pairs generated during the user's past interactions with the system, including the text of the commands sent by the user and the text of the responses from the system. The context list is an ordered collection of one or more historical conversation records retrieved from the database that are related to the current input command.

[0027] The database is a relational database used to store the structured mapping relationship between user IDs and session content. The time decay factor is a weighting coefficient with a value between 0 and 1, used to reflect the time value of information. The closer the timestamp is to the current moment, the larger the value of this factor, indicating that recent interaction information has a greater weight in influencing the current task than older interaction information.

[0028] In one embodiment, calculating the semantic relevance score between an input instruction and historical session records includes: extracting features from the input instruction to obtain a task feature vector and a first keyword set corresponding to the input instruction; extracting features from the historical session records to obtain a historical feature vector and a second keyword set corresponding to the historical session records; calculating the vector space similarity between the task feature vector and the historical feature vector; calculating the feature word overlap between the first keyword set and the second keyword set; and calculating the semantic relevance score based on the vector space similarity and the feature word overlap; wherein the semantic relevance score is related to the vector space similarity and the feature word overlap.

[0029] It should be noted that the first keyword set is the vocabulary set obtained after segmenting, filtering stop words, and extracting key phrases (e.g., TF-IDF and TextRank algorithms) the user's current input command; the second keyword set is the vocabulary set obtained after performing the same keyword extraction process on historical conversation records; the task feature vector is a high-dimensional dense vector representing the semantic information of the input command after encoding it through a pre-trained language model; the historical feature vector is a semantic vector obtained after performing the same encoding process on historical conversation records; vector space similarity usually refers to cosine similarity, which is used to measure the directional consistency of two vectors in multidimensional space, and the closer the value is to 1, the more similar the semantics; feature word overlap refers to the proportion of identical words between two keyword sets, which is used to measure the degree of matching at the literal level (e.g., Jaccard similarity coefficient).

[0030] In one embodiment, the formula for calculating the semantic relevance score is: .

[0031] in S The semantic relevance score is given. The feature vector of the task. The historical feature vector, For the first set of keywords, This is the second set of keywords. The intersection operator represents the set intersection operator. This represents the number of keywords shared by the first keyword set and the second keyword set. The union operator represents the set union operator. This represents the total number of unique keywords contained in both the first and second keyword sets. μ This is a preset retrieval balance coefficient, and 0 < μ <1.

[0032] Step S2: The input instruction is vectorized and encoded to generate a task query vector, and the similarity value between the task query vector and the feature vectors of each expert node in the preset expert model library is calculated.

[0033] In one embodiment, vectorizing the input instruction to generate a task query vector includes: segmenting the input instruction using a preset language model word segmenter to obtain an instruction word embedding sequence; inputting the instruction word embedding sequence into a bidirectional encoder model for semantic feature extraction to obtain the hidden state vector output by the last layer of the bidirectional encoder model, wherein the bidirectional encoder model includes multiple vertically stacked encoder network layers; performing global average pooling on the hidden state vector sequence to obtain a global pooling vector; and normalizing the global pooling vector output by the last layer of the bidirectional encoder model to obtain the task query vector.

[0034] It should be noted that the language model segmenter is used to serialize natural language text into a sequence of token IDs that the model can recognize. In one embodiment, the language model segmenter is a WordPiece or Byte-Pair Encoding segmenter. The bidirectional encoder model is a deep neural network model that can simultaneously utilize contextual information, such as BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa, used to extract deep semantic features. The hidden state vector is the output vector corresponding to each token in the model's network layer, containing the semantic representation of that token in a specific context. Global average pooling calculation, i.e., averaging the hidden state vectors at all positions in the sequence, results in a global pooling vector, which is used to represent the semantics of the entire sentence using a fixed-length vector. The task query vector is a normalized global pooling vector, serving as the query credential for subsequent retrieval of expert nodes.

[0035] In this embodiment, an expert node is an integrated computing unit capable of independently performing a specific reasoning task, such as a large language model or intelligent agent fine-tuned on domain-specific data. The expert model library is a database or parameter set used for centralized storage, management, and indexing of the aforementioned multiple expert nodes.

[0036] Step S3: Select the expert node whose similarity value is greater than a preset threshold as the target expert node, and transmit the input instruction and the historical interaction context sequence to the target expert node.

[0037] In one embodiment, mapping the intermediate inference text sequence to a high-dimensional feature vector to obtain the task execution trajectory feature vector includes: obtaining the hidden layer state for each generation step during the generation of the intermediate inference text sequence by the target expert node; and performing weighted aggregation calculation on the hidden layer states according to the temporal position of the generation step to obtain the task execution trajectory feature vector, wherein the formula for weighted aggregation calculation of the hidden layer states is: .

[0038] V traj This is the feature vector of the task execution trajectory. L This represents the total generation step size of the intermediate inference text sequence. t Indicates the current generation step number; λ This represents the time decay factor used to control the degree to which historical information is retained. H t Indicates the target expert node at the th t Each hidden layer state vector generates a step-size output; E final This represents the embedding vector corresponding to the end identifier in the intermediate inference text sequence; δ This indicates the preset reinforcement weight coefficient of the end identifier.

[0039] It should be noted that the intermediate inference text sequence is a text stream generated by the target expert node based on the thought chain mechanism when processing input instructions, containing logical deduction steps, intermediate conclusions, and the final answer. The task execution trajectory feature vector is a vector obtained by mathematically representing the above intermediate inference process; it not only contains the semantics of the final result but also incorporates the temporal features of the inference process. The hidden state vector is the instantaneous state vector output by the decoding layer when the expert model generates each inference word. The global average pooling calculation and the global pooling vector are compressed representations of the features of the entire inference sequence.

[0040] Step S4: Input the input instructions and the historical interaction context sequence to each of the target expert nodes, so that each of the target expert nodes generates an intermediate inference text sequence according to the autoregressive generation strategy, and maps the intermediate inference text sequence into a high-dimensional feature vector, which is denoted as the task execution trajectory feature vector of the target expert node.

[0041] It should be noted that after receiving the input instructions and historical context, the target expert node performs reasoning using a word-by-word prediction method. Based on the input information and the generated text content, the target expert node calculates the conditional probability distribution of the next token and samples it to generate the token for the current step. This process is continuously iterated until an end identifier is generated, thus forming a complete intermediate reasoning text sequence containing logical deduction steps.

[0042] Step S5: Aggregate and calculate the task execution trajectory feature vectors output by all target expert nodes through a multi-head attention mechanism to obtain global fusion features.

[0043] In one embodiment, the global fusion feature is obtained by aggregating and calculating the task execution trajectory feature vectors output by all target expert nodes through a multi-head attention mechanism. This includes: using the task query vector as the input source of the query matrix, and using the task execution trajectory feature vectors output by each target expert node as the input sources of the key matrix and the value matrix, respectively; calculating the dot product attention score of the query matrix and the key matrix in each of the multiple attention heads, and performing a weighted summation of the value matrix using the dot product attention score to obtain the subspace features of each attention head; and concatenating the subspace features of each attention head and then performing dimensional projection through a linear transformation layer to obtain the global fusion feature.

[0044] It should be noted that multi-head attention is a mechanism that allows the model to simultaneously focus on information from different representation subspaces. The query matrix, key matrix, and value matrix are the three core input components of the attention mechanism. The attention head is an independent computational unit in the multi-head attention mechanism. The dot product attention score is a relevance score measured by calculating the dot product of the query and key, determining the extent to which the model should focus on the corresponding value. The linear transformation layer is used to map the feature dimension back to the original or target dimension.

[0045] In this invention, a multi-head attention mechanism is used to dynamically calculate the contribution weight of each expert node based on the semantic matching degree between the task query vector (i.e., the mathematical representation of the user's intent) and the task execution trajectory feature vector of each expert node (i.e., the mathematical representation of the expert's reasoning logic). Based on this, the features of all experts are weighted and summed and their dimensions are transformed to achieve weighted fusion of the features of multiple expert nodes.

[0046] In this invention, the current task requirements are used as a benchmark to scan the reasoning trajectories of each expert and extract the substantive content information in the reasoning trajectories of each expert according to the degree of matching, thereby realizing task-oriented expert information extraction. This allows the task execution trajectory feature vectors output by each target expert node to be used as the input sources of the key matrix and value matrix, respectively.

[0047] This invention stitches together features from different aspects captured by different attention heads (e.g., some heads focus on logical coherence, while others focus on the accuracy of domain knowledge), and fuses these information through a fully connected layer to generate a final unified global fusion feature vector. This allows for the stitching together of subspace features from each attention head and then projecting the dimensions through a linear transformation layer.

[0048] Step S6: Map the global fusion features to the input layer of the pre-trained decoder, calculate the conditional probability distribution of each candidate word pair in the preset output vocabulary of the decoder, and generate the target text sequence according to the conditional probability distribution.

[0049] In one embodiment, calculating the conditional probability distribution of each candidate word pair in the decoder output vocabulary includes: inputting the global fusion features into the output mapping layer of the decoder, so that the decoder maps the feature data to a candidate word score space with the same dimension as the preset output vocabulary, to obtain the original score vector of each candidate word; adjusting the smoothness of the original score vector to obtain the adjusted score vector; and performing exponential normalization on the adjusted score vector to obtain the conditional probability distribution corresponding to each candidate word in the preset output vocabulary, wherein the conditional probability distribution includes the probability value of each candidate word.

[0050] It's important to note that the decoder is the network structure in a generative model responsible for progressively predicting the output text based on input features. The output mapping layer is typically a fully connected linear layer with an output dimension equal to the vocabulary size; the candidate word score space is an unnormalized raw numerical space, where the numerical values ​​represent potential trends but not probabilities; the original score vector is the vector output by the mapping layer, containing the score for each word in the vocabulary. Smoothness adjustment refers to adjusting the steepness of the score distribution by introducing a temperature parameter or other strategies; the score vector is the adjusted Logits vector.

[0051] This invention utilizes the Softmax function to convert a score vector containing negative numbers and arbitrary real values ​​into a probability distribution with a sum of 1 and each value between 0 and 1. This enables exponential normalization of the adjusted score vector, yielding the mathematical probability of each candidate word being the next output word.

[0052] In one embodiment, generating a target text sequence based on the conditional probability distribution includes: constructing a candidate word sorting set in descending order of probability values ​​of each candidate word in the conditional probability distribution; sequentially accumulating the probability values ​​in the candidate word sorting set until the accumulated value meets a preset truncation threshold, retaining candidate words within the truncation threshold range to form an effective sampling set; normalizing the probability values ​​of candidate words in the effective sampling set to obtain the candidate sampling weights; performing probabilistic sampling from the effective sampling set according to the sampling weights to obtain a preset number of target output words, and generating a target text sequence based on all target output words.

[0053] It should be noted that constructing the candidate word ranking set refers to the process by which the system sorts all the tokens (lexical units) in the preset vocabulary according to the conditional probability distribution output by the decoder.

[0054] Probabilistic sampling is a nondeterministic text generation strategy. When determining the next output word, probabilistic sampling does not directly lock the word with the highest probability, but introduces randomness based on the probability distribution, so that words with lower probabilities but within a reasonable range also have a chance to be selected.

[0055] The candidate word score space here refers to the data set containing all word elements in the vocabulary and their original probability values ​​after sorting and before truncation.

[0056] The sampling weight refers to the value obtained after renormalizing the probabilities of candidate words in the remaining "effective sampling set" after the truncation operation. Since a portion of the words were truncated, the sum of the probabilities of the remaining words is less than 1. Therefore, their probabilities need to be divided by the total probability of the retained set to make the sum equal to 1 again. This new probability value is the sampling weight, which directly determines the probability that the random number generator will fall within the corresponding interval of that word.

[0057] The truncation threshold range refers to a preset upper limit for cumulative probability in the kernel sampling algorithm. The system's filtering logic is dynamic: starting with the highest probability words after sorting, the probability is accumulated. Once the total accumulated probability reaches the threshold, all subsequent low probability words are truncated and eliminated. The system also accumulates the probability values ​​of each word starting with the highest probability word after sorting. Once the total accumulated probability exceeds the preset threshold, the selection of subsequent low probability words stops, and only these leading high probability words are retained as candidates to form an effective sampling set. This ensures both output quality (excluding absurdly low-probability words at the end) and a certain degree of diversity (not just selecting the single highest probability word).

[0058] Figure 2 This is a schematic diagram illustrating the structure of an expert cluster system for multi-model collaboration according to this embodiment.

[0059] This invention also provides an expert cluster system for collaboration among multiple models. For example... Figure 2 As shown, the system includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement an expert cluster method for multi-model collaboration according to the first aspect of the present invention.

[0060] The system also includes other components well known to those skilled in the art, such as communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0061] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0062] In the description of this specification, "multiple" or "several" means at least two, such as two, three or more, unless otherwise explicitly specified.

[0063] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for expert cluster collaboration among multiple models, characterized in that, include: Obtain input instructions sent by the user terminal that are configured with a user identifier, and retrieve the historical interaction context sequence from the database based on the user identifier; The input instruction is vectorized and encoded to generate a task query vector, and the similarity value between the task query vector and the feature vectors of each expert node in the preset expert model library is calculated. The input instruction and the historical interaction context sequence are transmitted to the target expert node, with the expert node whose similarity value is greater than a preset threshold as the target expert node; The input instructions and the historical interaction context sequence are input to each of the target expert nodes, so that each of the target expert nodes generates an intermediate inference text sequence according to the autoregressive generation strategy, and the intermediate inference text sequence is mapped to a high-dimensional feature vector, which is denoted as the task execution trajectory feature vector of the target expert node. The task execution trajectory feature vectors output by all target expert nodes are aggregated and calculated using a multi-head attention mechanism to obtain global fusion features. The global fusion features are mapped to the input layer of the pre-trained decoder, the conditional probability distribution of each candidate word pair in the preset output vocabulary of the decoder is calculated, and the target text sequence is generated according to the conditional probability distribution.

2. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, The database is used to store historical session records and their corresponding timestamps in key-value pairs, with the user identifier as the index key. Indexing and extracting the corresponding historical interaction context sequence from the database includes: Obtain all historical session records associated with the user identifier from the database, and construct a candidate context list in order of timestamp from most recent to oldest; Calculate the semantic relevance score between the input command and each historical session record in the candidate context list; sort each historical session record according to the product of the semantic relevance score and the time decay factor, and select the first preset number of historical session records as the historical interaction context sequence.

3. The expert cluster method for multi-model collaboration according to claim 2, characterized in that, The semantic relevance score between input commands and historical session records is calculated as follows: Feature extraction is performed on the input command to obtain the task feature vector and the first keyword set corresponding to the input command; feature extraction is performed on the historical session record to obtain the historical feature vector and the second keyword set corresponding to the historical session record. Calculate the vector space similarity between the task feature vector and the historical feature vector; Calculate the feature word overlap between the first keyword set and the second keyword set; perform a weighted sum based on the vector space similarity and the feature word overlap to obtain the semantic relevance score; wherein, the semantic relevance score is related to the vector space similarity and the feature word overlap.

4. The expert cluster method for multi-model collaboration according to claim 3, characterized in that, The formula for calculating the semantic relevance score is as follows: ； in S The semantic relevance score is given. The feature vector of the task. The historical feature vector, For the first set of keywords, This is the second set of keywords. The intersection operator represents the set intersection operator. This represents the number of keywords shared by the first keyword set and the second keyword set. The union operator represents the set union operator. This represents the total number of unique keywords contained in both the first and second keyword sets. μ This is a preset retrieval balance coefficient, and 0 < μ <1.

5. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, Vectorizing the input instructions to generate a task query vector includes: The input command is segmented by a pre-defined language model word segmenter to obtain the command word embedding sequence; The instruction word embedding sequence is input into the bidirectional encoder model to extract semantic features, and the hidden state vector output by the last layer of the bidirectional encoder model is obtained. The bidirectional encoder model includes multiple vertically stacked encoder network layers. The hidden state vector sequence is subjected to global average pooling to obtain a global pooling vector; the global pooling vector output from the last layer of the bidirectional encoder model is normalized to obtain the task query vector.

6. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, Mapping the intermediate inference text sequence into a high-dimensional feature vector to obtain the task execution trajectory feature vector includes: During the process of generating intermediate inference text sequences at the target expert node, the hidden layer state of each generation step is obtained; Based on the temporal position of the generated step size, the hidden layer states are weighted and aggregated to obtain the task execution trajectory feature vector. The formula for weighted aggregation of the hidden layer states is as follows: ； V traj This is the feature vector of the task execution trajectory. L This represents the total generation step size of the intermediate inference text sequence. t Indicates the current generation step number; λ This represents the time decay factor used to control the degree to which historical information is retained. H t Indicates the target expert node at the th t Each hidden layer state vector generates a step-size output; E final This represents the embedding vector corresponding to the end identifier in the intermediate inference text sequence; δ This indicates the preset reinforcement weight coefficient of the end identifier.

7. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, By aggregating and calculating the task execution trajectory feature vectors output by all target expert nodes through a multi-head attention mechanism, the global fusion features are obtained, including: The task query vector is used as the input source of the query matrix, and the task execution trajectory feature vectors output by each target expert node are used as the input sources of the key matrix and value matrix, respectively. The dot product attention score of the query matrix and the key matrix is ​​calculated in multiple attention heads respectively, and the value matrix is ​​weighted and summed using the dot product attention score to obtain the subspace features of each attention head; The subspace features of each attention point are concatenated and then dimensionally projected through a linear transformation layer to obtain the global fusion feature.

8. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, Calculating the conditional probability distribution of each candidate word pair in the decoder output vocabulary includes: The global fusion features are input to the output mapping layer of the decoder so that the decoder maps the feature data to a candidate word score space that is consistent with the preset output vocabulary dimension, thereby obtaining the original score vector of each candidate word. The original score vector is smoothed to obtain the adjusted score vector. The adjusted score vector is subjected to exponential normalization to obtain the conditional probability distribution of each candidate word in the preset output vocabulary. The conditional probability distribution includes the probability value of each candidate word.

9. The expert cluster method for multi-model collaboration according to claim 1, characterized in that, Generating the target text sequence based on the conditional probability distribution includes: Based on the probability values ​​of each candidate word in the conditional probability distribution, construct a candidate word sorting set in descending order; The probability values ​​in the candidate word sorting set are accumulated sequentially until the accumulated value meets the preset truncation threshold. Candidate words within the truncation threshold range are retained to form an effective sampling set. The probability values ​​of candidate words in the effective sampling set are normalized to obtain the candidate sampling weights; probabilistic sampling is performed from the effective sampling set according to the sampling weights to obtain a preset number of target output words, and a target text sequence is generated based on all target output words.

10. A multi-model collaborative expert cluster system, comprising a processor and a memory, characterized in that, The memory stores a computer program, and the processor executes the computer program to implement an expert cluster method for multi-model collaboration as described in any one of claims 1-9.

Images