Hybrid expert model word group scheduling method and electronic device

Abstract

The application discloses a mixed expert model word grouping scheduling method and electronic equipment, and relates to the technical field of artificial intelligence, which comprises the following steps: in response to the arrival of word units in the form of a stream to a preset buffer, obtaining the embedding vector features, routing history and arrival time corresponding to each word unit to splice into the target feature vector corresponding to each word unit, and the preset buffer is configured with a preset waiting time threshold and a capacity upper limit value; clustering and grouping each word unit according to the target feature vector to generate at least one task group; determining the priority of each task group based on the average waiting time of each word unit in each task group and the historical hot index; under the condition of meeting the limitation of the preset buffer, scheduling each task group according to the priority to enable the mixed expert model to perform operation on each word unit. The method of the application improves the utilization rate of experts and the system throughput while ensuring the upper bound of delay through short window buffering and dynamic clustering.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a word grouping scheduling method and electronic device based on a hybrid expert model. Background Technology

[0002] Hybrid expert (MoE) models, leveraging sparse activation mechanisms, maintain computational efficiency while scaling parameter size and have been widely applied in online inference scenarios such as real-time dialogue, code generation, and multilingual translation. In these scenarios, user input arrives dynamically as streaming tokens, placing stringent requirements on inference response latency. Currently, two main approaches are used to optimize MoE inference performance: first, dynamic token redistribution and weight prefetching across devices to alleviate uneven expert load; second, accumulating tokens to form large batches to increase computational density, with some solutions employing fixed-time-window timeout batch processing mechanisms to aggregate tokens for unified scheduling.

[0003] However, in online inference of the MoE model, the expert activation mode of the token depends on contextual features and changes dynamically. Traditional static batch processing strategies are difficult to adapt to this characteristic, making it difficult to balance latency constraints and expert utilization. Summary of the Invention

[0004] This application provides a word grouping scheduling method and electronic device based on a hybrid expert model, to at least solve the problem that related technologies cannot meet the fine-grained balance requirements of latency and throughput in online inference scenarios.

[0005] Firstly, this application provides a word grouping scheduling method based on a hybrid expert model, including:

[0006] In response to the arrival of words in the preset buffer in the form of a stream, the embedding vector features, routing history and arrival time of each word are obtained and concatenated into the target feature vector corresponding to each word. The preset buffer is configured with a preset waiting time threshold and a capacity limit.

[0007] Cluster and group each word according to the target feature vector to generate at least one task group;

[0008] The priority of each task group is determined based on the average waiting time of each word in each task group and the historical hot spot index.

[0009] Under the condition of satisfying the preset buffer constraints, each task group is scheduled according to priority so that the hybrid expert model can perform operations on each word.

[0010] Secondly, this application also provides an electronic device, comprising:

[0011] Memory, used to store computer programs;

[0012] A processor is used to implement the steps of any of the methods in the first aspect when executing a computer program.

[0013] This application provides a word grouping scheduling method and electronic device based on a hybrid expert model. This method acquires the embedding vector features, routing history, and arrival time of words, concatenates them into a target feature vector, and integrates multi-dimensional information to provide a comprehensive and accurate basis for subsequent clustering and priority determination. A preset buffer is set with a waiting time threshold and a capacity limit to initially manage words and ensure system stability and reliability. Clustering based on the target feature vector groups similar feature words into task groups, making subsequent processing more targeted and improving efficiency. When determining priorities, the average waiting time of words in the task group is considered, emphasizing timeliness and ensuring the system prioritizes those with longer waiting times, enhancing real-time response capabilities. Historical hotspot indices reflect the historical activity of experts, and are combined with average waiting times to determine priorities, balancing long waiting times for words with expert load and improving overall system performance. Scheduling is performed within buffer limits to ensure reasonable resource allocation. The hybrid expert model processes words in each task group sequentially according to priority, avoiding task conflicts and improving processing efficiency and accuracy. Furthermore, because the task groups are divided according to word features, the model can adopt appropriate strategies for different groups, further improving the execution effect. Attached Figure Description

[0014] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram illustrating an application scenario for a word grouping and scheduling method based on a hybrid expert model provided in an embodiment of this application;

[0016] Figure 2 A flowchart illustrating a word grouping and scheduling method using a hybrid expert model, provided in an embodiment of this application;

[0017] Figure 3 A flowchart illustrating a word grouping and scheduling method based on a hybrid expert model, provided in another embodiment of this application;

[0018] Figure 4 A schematic diagram of the structure of a word grouping scheduling device for a hybrid expert model provided in an embodiment of this application;

[0019] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those of ordinary skill in the art without creative effort are all within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Hybrid Expert (MoE) models, leveraging the core advantage of sparse activation, effectively maintain high computational performance while significantly expanding the model parameter scale, and have become a key technical support for online inference scenarios such as real-time dialogue, code generation, and multilingual translation. In these scenarios, user input arrives dynamically and continuously in the form of streaming tokens, placing extremely high demands on the response latency of inference services. To optimize the inference performance of MoE models, the industry has mainly developed two technical paths. One is to improve the problem of uneven expert load through dynamic token redistribution and weight prefetching schemes across devices; the other is to improve the utilization density of computing resources by accumulating tokens to build large batch tasks. Some schemes also adopt a timeout batch processing mechanism with a fixed time window to aggregate the arriving tokens and schedule them for unified execution. However, in the actual online inference process of MoE models, the expert activation mode corresponding to the token is highly dependent on contextual features and changes dynamically with the input content. Traditional static batch processing strategies lack the ability to adapt to this dynamic characteristic and cannot flexibly adjust the grouping and scheduling logic according to the real-time characteristics of the tokens, ultimately making it difficult to achieve an effective balance between meeting strict latency constraints and improving the utilization of expert resources.

[0024] Therefore, when facing the technical challenges of the aforementioned technologies, and addressing the issue that traditional static batch processing cannot perceive the correlation of token features, this approach predicts expert activation patterns by mining the token's own features. Embedded vector features and routing history are extracted as core features, and arrival time is added to represent the waiting state. These are then concatenated into a target feature vector, providing a basis for accurate grouping. Simultaneously, buffers, waiting time thresholds, and capacity limits are set to prevent infinite waiting and excessive latency. To reduce expert switching overhead and improve utilization, clustering is performed based on the target feature vector. Tokens with similar features are likely to activate overlapping expert sets. Clustering groups these experts into task groups, enabling centralized utilization of expert resources and solving the problem of dispersed expert distribution in traditional grouping. To address the resource allocation imbalance caused by the lack of priority in scheduling, priority is calculated by combining waiting time and expert hotspots, balancing latency control and efficient resource allocation. Finally, to ensure scheduling meets constraints, scheduling is executed according to priority. Under the buffer threshold limit, high-priority task groups are processed first, satisfying latency requirements while improving expert utilization through group aggregation, ultimately forming a complete scheduling scheme.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario corresponding to a word grouping and scheduling method based on a hybrid expert model provided in an embodiment of this application. For example... Figure 1 As shown, the application scenario includes: terminal device 101 and server 102. The word grouping and scheduling device of the hybrid expert model can be integrated into server 102, and the hybrid expert model is also deployed on server 102.

[0026] Specifically, the user initiates an inference request to the server 102 through the terminal device 101. The inference request includes multiple lexical units, including the first lexical unit to the i-th lexical unit. The lexical units arrive at the preset buffer in the form of a stream. After receiving the lexical units, the server 102 begins to acquire the embedding vector features, routing history, and arrival time corresponding to each lexical unit, and concatenates them to form the target feature vector corresponding to each lexical unit, including the first target feature vector to the i-th target feature vector. Then, the server 102 clusters and groups the lexical units according to the target feature vectors to generate at least one task group, including the first task group to the K-th task group. Expert set prediction is then performed on each task group, such as the first task group corresponding to the first expert set, the second task group corresponding to the second expert set, and the K-th task group corresponding to the K-th expert set. The server 102 determines the priority of each task group based on the average waiting time of each lexical unit in each task group and the historical hotspot index. Finally, under the condition of satisfying the constraints of the preset buffer, the server schedules each task group according to priority. A mini-batch is created based on the task group, and the task is submitted to the execution engine of the hybrid expert model so that the hybrid expert model can perform operations on each lexical unit.

[0027] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0028] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0029] Figure 2 This is a flowchart illustrating a word grouping and scheduling method based on a hybrid expert model, as provided in an embodiment of this application. Figure 2 As shown, the execution entity in this embodiment is a word grouping and scheduling device based on a hybrid expert model. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc; alternatively, it can be implemented through a physical device integrating or installing the relevant computer program, such as a chip or electronic device. The electronic device can be a computer or a server, etc. The word grouping and scheduling method based on a hybrid expert model provided in this embodiment includes the following steps:

[0030] S201. In response to the arrival of words in the preset buffer in the form of a stream, the embedded vector features, routing history and arrival time of each word are obtained and concatenated into the target feature vector corresponding to each word. The preset buffer is configured with a preset waiting time threshold and a capacity limit value.

[0031] Among them, "flow arrival" means that word units are not input all at once, but are entered continuously and in real time segment by segment.

[0032] The preset buffer refers to a temporary storage area used to temporarily store the words arriving in the stream.

[0033] Among them, routing history refers to the record of the expert number and corresponding matching weight assigned to each term as it passes through the expert modules of each layer of the MoE model.

[0034] Among them, the target feature vector refers to the comprehensive feature carrier that integrates the core information of word elements, which is composed of embedded vector features, routing history, and arrival time.

[0035] Among them, the preset waiting time threshold (T) max () refers to the longest allowed waiting time for a word in the buffer.

[0036] Among them, the upper limit of capacity (N) max () refers to the maximum number of tokens that the buffer can hold.

[0037] Specifically, as tokens arrive continuously in a streaming manner, each arriving token is immediately sent to a pre-defined buffer for temporary storage. The buffer assigns a unique identifier to each token and records its precise arrival time. For each token temporarily stored in the buffer, its contextual information is fully utilized to transform it into a fixed-dimensional embedding vector feature. The routing history of this token is extracted, including the expert ID it was assigned in previous model layers and the matching weights given by the gating network at each assignment, and this information is transformed into standardized numerical features. The arrival time of the tokens recorded in the buffer is directly read and transformed into temporal features.

[0038] Furthermore, the extracted embedding vector features, routing history features, and time features are concatenated in a preset order to form a target feature vector with a unified dimension. Each word corresponds to a unique target feature vector, fully carrying the semantics, adaptation preferences, and waiting state information of that word.

[0039] Understandably, the buffer monitors the waiting time of each word in real time. If the waiting time of any word exceeds the preset waiting time threshold, it will be marked as a timed-out word and a rollback strategy will be triggered. Secondly, the current word storage capacity of the buffer will be monitored. If the capacity limit is reached, high-priority words will be prioritized for scheduling to make room for newly arriving words.

[0040] S202. Cluster and group each word according to the target feature vector to generate at least one task group.

[0041] In this context, a task group refers to a set of lexical units formed after clustering. Each set serves as an independent scheduling unit and contains several lexical units with similar features.

[0042] Specifically, online clustering strategies, such as mini-batch K-means clustering or streaming clustering algorithms, are employed to dynamically generate task groups based on the actual distribution of word features. Initially, the first word is used as the initial core of the first task group. Each subsequent word is then compared for similarity with the core features of existing task groups. If the similarity between a word and a core feature of a task group is higher than a preset threshold, the feature fit is considered satisfactory, and the word is assigned to that task group. If the similarity with all existing task group cores is lower than the threshold, a new task group is created with that word as the core. When all words in the buffer have completed clustering and assignment, or when preset grouping trigger conditions are met (e.g., the number of task groups reaches a preset range or the clustering duration meets requirements), the clustering process stops. The resulting set of several compact and reasonably sized word sets constitutes the task group.

[0043] Optionally, the size and feature compactness of each task group are monitored in real time during the clustering process. If the number of lexical units in a task group is too large, it may lead to an increase in the feature differences within the group. In this case, a secondary split will be performed based on the feature distribution of lexical units within the group. If the core features of multiple adjacent task groups are highly similar, it indicates that the expert activation tendencies of lexical units in each group are highly overlapping, and these task groups will be merged.

[0044] S203. Determine the priority of each task group based on the average waiting time of each word in each task group and the historical hot spot index.

[0045] The average waiting time refers to the average waiting time of all words in a single task group, which is the time difference between the moment a word arrives in the buffer and the moment the current priority is calculated.

[0046] Among them, the historical hotspot index refers to a comprehensive index obtained by statistically analyzing the historical call frequency or cumulative load value of all experts in the set of prediction experts corresponding to the task group.

[0047] Specifically, the process iterates through the tokens within the target task group, extracts the arrival time of each token from the buffer records, and subtracts this arrival time from the current system time to obtain the individual waiting time for each token. Then, the sum of the waiting times for all tokens within the task group is calculated, and divided by the total number of tokens in the task group to obtain the average waiting time for the task group. Next, the set of prediction experts obtained through feature clustering and expert prediction for the task group is acquired. The stored historical operation data of these experts is retrieved, and the historical call frequency of each expert in the set is extracted, such as the number of times they were activated in the past hour; or the load index, such as the total number of tokens processed cumulatively. The call frequencies or load indices of the experts are summed to obtain the historical hotspot index for the task group.

[0048] Optionally, the weighting coefficients of average waiting time and historical hotspot index can be preset. The weights can be dynamically adjusted according to the business scenario. For example, the weight of average waiting time can be increased in latency-sensitive scenarios, and the weight of historical hotspot index can be increased in throughput-priority scenarios.

[0049] Furthermore, the average waiting time of the task group is multiplied by its corresponding weight, and the historical hotspot index is multiplied by its corresponding weight. The two are then added together to obtain the priority score for the task group. Finally, the task groups are classified into three levels based on their priority scores, such as high, medium, and low, or the scores are used directly as the sorting criteria. Task groups with higher scores have higher priority and will be scheduled for execution first in subsequent scheduling.

[0050] S204. Under the condition of satisfying the preset buffer constraints, schedule each task group according to priority so that the hybrid expert model can perform operations on each word.

[0051] Among them, the restrictions of the preset buffer refer to the core constraint rules that the preset buffer must follow when running, including the preset waiting time threshold and the capacity limit.

[0052] Scheduling refers to the process of assigning task groups to the corresponding expert modules of the hybrid expert model according to preset rules.

[0053] The execution operation refers to the entire process by which the hybrid expert model, for each word in a task group, activates the corresponding expert module based on its set of prediction experts, completes the inference calculation of the word, and outputs the result.

[0054] Specifically, first, the total number of unscheduled terms in the current buffer is counted to confirm that it does not exceed the capacity limit. Then, it is determined whether there are any terms in each task group whose waiting time exceeds the preset waiting time threshold, and the timed-out task groups that need to be processed first are marked. Based on the calculated priority scores or levels of each task group, the task groups to be scheduled are globally sorted. The task groups are scheduled in order according to the sorting results. The task group with the highest ranking is selected, and it is confirmed that its corresponding prediction expert set is currently in an executable state, and that the remaining capacity of the buffer still meets the constraints after scheduling the task group. The term information of each term in the task group is then sent to the MoE model. After receiving the task group, the model only activates the expert modules in its prediction expert set, performs inference calculations on all terms in the group in parallel, outputs the term processing results after completion, and removes the terms of the task group from the buffer.

[0055] Optionally, if the remaining buffer capacity is insufficient or a new timeout term appears after scheduling a task group, the scheduling of the current sorting queue is immediately suspended, and the timeout task group is processed first or some buffer space is released; if the prediction expert set corresponding to a task group is currently under high load, the task group is temporarily skipped, and the task group with the second highest priority and whose corresponding expert is idle is scheduled first, and the scheduling is backtracked after the target expert load decreases.

[0056] The word grouping and scheduling method of the hybrid expert model provided in this application obtains the embedding vector features, routing history, and arrival time of words, concatenates them into a target feature vector, and integrates multi-dimensional information to provide a comprehensive and accurate basis for subsequent clustering and priority determination. A preset buffer is set with a waiting time threshold and a capacity limit to initially manage words and ensure system stability and reliability. Based on the target feature vector, clustering and grouping similar feature words into task groups makes subsequent processing more targeted and improves processing efficiency. When determining priorities, the average waiting time of words in the task group is considered, emphasizing timeliness and ensuring that the system prioritizes processing those with longer waiting times, enhancing real-time response capabilities. The historical hotspot index reflects the historical activity of experts, and is combined with the average waiting time to determine priorities, balancing the avoidance of long word waiting times and expert load, thus improving overall system performance. Scheduling is performed under buffer constraints to ensure reasonable resource allocation. The hybrid expert model processes words in each task group sequentially according to priority, avoiding task conflicts and improving processing efficiency and accuracy. Furthermore, because task groups are divided according to word features, the model can adopt appropriate strategies for different groups, further improving execution performance.

[0057] As an optional implementation, based on any of the above embodiments, each word is clustered and grouped according to the target feature vector to generate at least one task group, including:

[0058] A pre-defined clustering algorithm is used to calculate the feature similarity between each word based on the target feature vector.

[0059] Based on feature similarity, terms whose feature distance is less than a preset similarity threshold are grouped together to generate at least one task group.

[0060] A pre-defined expert predictor is used to predict the probability value of each word in each task group activating each expert based on the target feature vector.

[0061] The probability values ​​of each word activating each expert in each task group are aggregated to obtain the average probability value of each expert activating each task group.

[0062] The set of prediction experts corresponding to each task group is determined based on the average probability value of activating each expert and the preset prediction rules.

[0063] Among them, the pre-selected clustering algorithm refers to the clustering method pre-selected for the characteristics of streaming tokens, such as online K-means, mini-batch K-means, density clustering, etc.

[0064] Feature similarity refers to an indicator that quantifies the degree of fit between the target feature vectors of two words.

[0065] Feature distance refers to an index that is inversely related to feature similarity, such as Euclidean distance and cosine distance. The smaller the distance, the higher the similarity.

[0066] The preset similarity threshold refers to the pre-set critical value of feature distance.

[0067] Among them, the pre-configured expert predictor refers to a lightweight prediction model that is pre-configured, such as a small MLP (Multi-Layer Perceptron) or a sparse perceptron. After offline training, it can output the probability of each expert being activated by a word based on the word target feature vector.

[0068] Among them, the preset prediction rules refer to the judgment rules for determining the set of prediction experts, such as threshold rules and Top-K rules.

[0069] The prediction expert set refers to the subset of experts that are most likely to be suitable for the task group.

[0070] Specifically, the target feature vectors of all words are preprocessed, and normalization is used to eliminate the differences in numerical ranges of features across different dimensions. A preset clustering algorithm is invoked, and for each pair of target feature vectors, a distance calculation method suitable for high-dimensional features is selected to calculate the feature distance, which is then converted into feature similarity. An empty task group set is initialized, with the first word to be processed as the initial member of the first task group. The remaining words are processed sequentially, and the feature distance between the word and the core words of existing task groups is calculated. If the distance is less than a preset similarity threshold, the word is assigned to the corresponding task group. If the feature distance with all existing task groups exceeds the threshold, a new task group is created with that word as the core.

[0071] Furthermore, for each generated task group, the target feature vectors of all words within the group are extracted and input one by one into a preset expert predictor. For each word, the probability value of activating all experts in its model is output. The words within the task group are traversed, and the activation probability values ​​of each expert corresponding to each word are collected to form a probability matrix. If an arithmetic mean is used, the values ​​in each column of the matrix are summed directly and then divided by the total number of words in the group to obtain the average probability value of activating each expert in that task group. If a weighted average is used, a confidence weight is assigned to each word, and the weighted sum of each column is calculated using the weight as a coefficient. Finally, the average probability value is obtained by dividing by the sum of the weights.

[0072] Furthermore, preset prediction rules are selected based on the business scenario. If a threshold rule is used, an activation probability threshold is set, such as 0.5. The average probability values ​​of each expert are iterated, and experts with an average probability value greater than or equal to the threshold are included in the prediction expert set. If a Top-K rule is used, a K value is set, such as K=4. The average probability values ​​of each expert are sorted in descending order, and the top K experts are selected to form the prediction expert set.

[0073] The word grouping and scheduling method of the hybrid expert model provided in this application adopts a preset clustering algorithm to calculate the feature similarity between words based on the target feature vector. This accurately measures the similarity between words in multiple dimensions such as semantics, routing history, and arrival time, accurately identifying and aggregating truly similar words, laying the foundation for efficient subsequent processing. A preset expert predictor predicts the probability value of word activation experts in each task group based on the target feature vector, fully considering the relationship between individual word features and expert activation, accurately reflecting the matching degree. The probability values ​​of word activation experts in each task group are aggregated to avoid bias and provide a comprehensive and objective reference for determining the prediction expert set. Then, the prediction expert set is determined based on the average probability value of the activated experts and preset rules, accurately allocating the most suitable expert resources to meet processing needs and make reasonable use of resources. Grouping similar words and assigning appropriate expert sets allows the hybrid expert model to adopt more targeted strategies for the characteristics of different groups. Centralized processing of similar words reduces repetitive operations and resource switching, improving processing efficiency.

[0074] As an optional implementation, based on any of the above embodiments, the priority of each task group is determined according to the average waiting time of each word in each task group and the historical hotspot index, including:

[0075] Obtain the historical call frequency of each expert in the prediction expert set corresponding to each task group.

[0076] Calculate the historical hotspot index for each task group based on historical call frequency.

[0077] A preset weighted algorithm is used to calculate the priority index of each task group based on the average waiting time of each word in each task group and the historical hot spot index.

[0078] The priority of each task group is determined based on the priority index and the preset priority threshold.

[0079] Among them, historical call frequency refers to the number of times a certain expert has been successfully activated and processed word units within a statistical period, such as the past 5 minutes.

[0080] Among them, the preset weighting algorithm refers to a predefined mathematical algorithm used to integrate the average waiting time and the historical hot spot index, such as linear weighting, non-linear weighting, etc.

[0081] The priority index is a numerical indicator obtained by weighting the average waiting time and the historical hot spot index using an algorithm.

[0082] Among them, the preset priority threshold refers to a number of pre-set critical values ​​used to divide continuous priority indices into discrete priority levels.

[0083] Specifically, the system locates the task group to be calculated and its corresponding set of prediction experts, retrieves the stored expert operation logs, and obtains the historical call frequency of each expert within the prediction expert set. If a basic summation method is used, all frequency values ​​in the list are directly added together to obtain the historical hotspot index of the task group. For more precise calculations, a load weight is assigned to the historical call frequency of each expert, and then a weighted sum is calculated as the historical hotspot index. The system iterates through the keywords within the task group, extracts the arrival time of each keyword, subtracts the arrival time from the current system time to obtain the waiting time for a single keyword, sums these values, and divides the sum by the total number of keywords in the group to obtain the average waiting time for the task group.

[0084] Understandably, basic weights, such as α and β, are pre-set for average waiting time and historical hotspot index, with α+β=1. The weights can be dynamically adjusted according to the business scenario, such as increasing α for latency-sensitive scenarios and increasing β for throughput-priority scenarios.

[0085] Furthermore, the average waiting time is standardized to a value in the 0-1 range and multiplied by the corresponding weight α; the historical hotspot index is also standardized to a value in the 0-1 range and multiplied by the corresponding weight β; the two are added together to obtain the priority index of the task group. A preset priority threshold system is retrieved, such as a high-priority threshold T1 and a medium-priority threshold T2, where T1>T2>0. The calculated task group priority index is compared with each threshold. If the priority index ≥ T1, the task group is determined to be high-priority and requires priority scheduling; if T2 ≤ priority index < T1, it is determined to be medium-priority and scheduled according to the regular queue; if the priority index < T2, it is determined to be low-priority and can be delayed in scheduling.

[0086] Optionally, if the priority indices of multiple task groups are in the same threshold range, further reference can be made to auxiliary indicators, such as the number of terms in the task group and the current real-time load of experts, to refine the priority ranking.

[0087] The word grouping scheduling method of the hybrid expert model provided in this application calculates the historical hotspot index by obtaining the historical call frequency of each expert in the prediction expert set, and combines it with the average waiting time of words to comprehensively consider multiple key factors affecting the priority of task groups. A preset weighted algorithm is used to calculate the priority index, which can assign different weights to the average waiting time and historical hotspot index according to actual needs, accurately quantifying the priority of each task group, making the priority differences between different task groups clearer, and facilitating accurate scheduling and resource allocation by the system.

[0088] As an optional implementation, based on any of the above embodiments, task groups are scheduled according to priority so that the hybrid expert model performs operations on each lexical unit, including:

[0089] The task groups are sorted in descending order of priority to generate a scheduling sequence.

[0090] Based on the prediction expert set corresponding to each task group, tasks that are adjacent in the scheduling sequence and whose prediction expert set overlap is greater than a preset overlap threshold are grouped into aggregated task groups.

[0091] The aggregated task group and the unmerged task group are respectively generated into mini-batch execution groups, which are then sent to the hybrid expert model. The model is instructed to activate only the corresponding set of predictive experts for forward propagation.

[0092] The scheduling sequence refers to the ordered queue formed by sorting the task groups to be scheduled from high to low priority.

[0093] Among them, the overlap of prediction expert sets refers to the index that measures the degree of overlap between the prediction expert sets of two task groups, which is usually calculated by the number of experts in the intersection / the number of experts in the union.

[0094] Among them, the preset overlap threshold refers to the pre-set critical value of overlap, which serves as the standard for determining whether adjacent task groups should be merged.

[0095] Among them, an aggregated task group refers to a new task group formed by merging multiple adjacent task groups whose expert sets have sufficient overlap.

[0096] Among them, mini-batch refers to the smallest execution unit that integrates the lexical units of a single task group, which is adapted to the batch inference characteristics of the MoE model.

[0097] Specifically, the determined priorities of each task group are extracted. High-priority task groups are placed at the top, followed by medium-priority and low-priority task groups. Task groups within the same priority level are further sorted from highest to lowest priority index. The generated scheduling sequence is traversed, starting with the first task group, comparing the prediction expert sets of the current task group with the next adjacent task group. The intersection and union of the two expert sets are calculated, and the overlap is obtained by dividing the number of intersections by the number of unions. If the overlap is greater than a preset overlap threshold, and the number of terms in the merged aggregated task group does not exceed a preset batch limit, then the two task groups are merged into one aggregated task group; if the number of terms exceeds the limit after merging, only some terms are merged or the merging is skipped. After completing the current comparison, the above operation is repeated with the next adjacent task group using the aggregated task group until the entire scheduling sequence has been traversed. Task groups that do not meet the merging conditions remain independent, ultimately forming a mixed queue of aggregated task groups and independent task groups.

[0098] Furthermore, the merged aggregated task groups and the unmerged independent task groups are processed separately to generate unique execution identifiers. Core information such as the target feature vectors of the words within each group and the prediction expert set are integrated and encapsulated into structurally unified mini-batch execution groups. Following the scheduling sequence, the mini-batch execution groups are sent sequentially to the hybrid expert model, along with an expert activation instruction for each group. This instruction explicitly marks the index information of the prediction expert set corresponding to that group. Upon receiving the mini-batch execution group, the model quickly locates and activates only the expert modules within the prediction expert set based on the index information. The activated expert modules perform parallel forward propagation calculations on all words within the group, completing the word inference processing.

[0099] The lexical grouping scheduling method for the hybrid expert model provided in this application sorts task groups according to priority from high to low to generate a scheduling sequence, ensuring that important and urgent task groups are processed first. Adjacent tasks in the scheduling sequence with an overlap greater than a preset overlap threshold are grouped into aggregated task groups, reducing the number of times the model switches experts between different task groups, effectively reducing switching costs and further improving execution efficiency. Aggregated task groups and unmerged task groups are respectively generated into small-batch execution groups and sent to the hybrid expert model, enabling the model to rationally allocate computing resources according to the characteristics and needs of the task groups. The model is instructed to activate only the corresponding set of predictive experts for forward propagation, avoiding unnecessary expert activation, reducing resource waste, and achieving precise utilization of expert resources.

[0100] As an optional implementation, based on any of the above embodiments, it further includes:

[0101] In response to any word in the preset buffer having a waiting time exceeding a preset waiting time threshold, the word is directly scheduled so that the hybrid expert model can perform operations on the word.

[0102] In response to the fact that the waiting time of multiple terms exceeds the preset waiting time threshold at the current moment, the current load rate of each expert in the hybrid expert model and the remaining capacity of the preset buffer are obtained.

[0103] If the current load rate is less than the preset low load threshold and the remaining capacity is greater than the preset capacity threshold, then the rollback ratio will be set to the first preset ratio.

[0104] If the current load rate is greater than the preset high load threshold or the remaining capacity is less than the preset capacity threshold, the rollback ratio will be set to the second preset ratio.

[0105] The current load rate refers to the ratio of the actual processing load of each expert in the hybrid expert model to the maximum processing capacity at a certain moment.

[0106] Among them, the preset / high load threshold refers to the critical value for judging the overall load status of experts. The low load threshold, such as 0.3, means that the experts are in an idle state, and the high load threshold, such as 0.8, means that the experts are close to saturation.

[0107] The remaining capacity refers to the unoccupied storage space of the preset buffer, which is the upper limit of the capacity minus the number of currently stored words.

[0108] The preset capacity threshold refers to the critical value used to determine whether the remaining space in the buffer is sufficient.

[0109] The rollback ratio refers to the proportion of timeout words that need to be directly scheduled.

[0110] The first / second preset ratio refers to two pre-set rollback ratio values. The first preset ratio, such as 0.2-0.4, is a low ratio, and the second preset ratio, such as 0.8-1.0, is a high ratio, which adapts to different system loads and buffer states.

[0111] Specifically, the system monitors the waiting time of each word in the preset buffer in real time, continuously calculating the difference between the current system time and the word arrival time to obtain the real-time waiting time of a single word. When the waiting time of any word exceeds the preset waiting time threshold, the word rollback mechanism is immediately triggered. Skipping the conventional clustering and priority calculation process, the word is directly marked as an urgent execution word, generating the smallest execution unit containing only that word, and sending it to the hybrid expert model.

[0112] Furthermore, when it is detected that the waiting time of multiple tokens exceeds the preset waiting time threshold at the current moment, the regular batch scheduling process is paused. The experts in the hybrid expert model are traversed, and the number of tokens currently being processed and the length of the pending task queue for each expert are counted. Combined with the expert's maximum processing capacity, the real-time load rate of each expert is calculated. The preset buffer capacity limit is retrieved, and the number of all unscheduled tokens currently stored in the buffer is counted. The remaining buffer capacity is obtained by subtracting the stored number from the capacity limit.

[0113] Furthermore, if the calculated current expert load rate is less than the preset low load threshold, it indicates that the experts are generally idle, and the remaining buffer capacity is greater than the preset capacity threshold, indicating that the buffer space is sufficient. In this case, there is no need for large-scale rollback, and the rollback ratio is set to the first preset ratio. Only a small number of timeout words are selected for direct scheduling, and the remaining timeout words can still participate in clustering and grouping.

[0114] Furthermore, if the current expert load rate is greater than the preset high load threshold, it indicates that the experts are nearing saturation. If a large number of words are waiting, it may lead to subsequent congestion. Or, if the remaining buffer capacity is less than the preset capacity threshold, it indicates that the buffer is about to overflow and cannot accommodate new words. In this case, the rollback ratio is set to the second preset ratio. Most or even all timed-out words are selected for direct scheduling to quickly release buffer space.

[0115] The word grouping and scheduling method for the hybrid expert model provided in this application directly schedules any word whose waiting time in the preset buffer exceeds a preset waiting time threshold. This avoids the waiting time of timed-out words from being delayed due to normal scheduling processes, ensuring timely system response to urgent or key words and improving system real-time performance. When multiple words exceed their waiting time limits, the current load rate of each expert in the hybrid expert model and the remaining capacity of the preset buffer are obtained, and the rollback ratio is flexibly set based on these two key indicators. By considering the current load rate of the experts when setting the rollback ratio, the load among the experts can be balanced. This avoids performance degradation of some experts due to excessive task processing, while ensuring that other experts are also fully utilized, thus improving the overall operating efficiency of the hybrid expert model.

[0116] As an optional implementation, based on any of the above embodiments, after clustering and grouping each word according to the target feature vector to generate at least one task group, the method further includes:

[0117] In response to reaching a preset re-clustering time point or the number of unscheduled words in the preset cache reaching a preset re-clustering threshold, the latest target feature vector of the unscheduled words is obtained.

[0118] The clustering grouping operation is re-executed based on the latest target feature vector to generate at least one updated task group.

[0119] The corresponding set of prediction experts is re-predicted based on the updated task groups, and the priority of each updated task group is updated.

[0120] Among them, the preset re-clustering time point refers to the fixed time node that is set in advance to trigger re-clustering, such as every 500ms or every second.

[0121] The preset re-clustering threshold refers to the critical number of lexical units that trigger re-clustering. When the cumulative number of unscheduled lexical units in the buffer reaches this value, re-clustering is immediately initiated.

[0122] The latest target feature vector refers to the feature vector updated based on the latest state of the lexical unit when re-clustering is triggered. It may differ from the initial feature vector due to factors such as context expansion and changes in routing trends.

[0123] Understandably, the built-in timing module starts timing from the moment the last clustering / re-clustering was completed. When a preset re-clustering time point is reached, such as every 300ms, a time condition is triggered. The number of unscheduled terms in the buffer is counted in real time, and when the number reaches a preset re-clustering threshold, such as 50, a quantity condition is triggered.

[0124] Specifically, after re-clustering is triggered, unscheduled terms in the buffer are selected, and their target feature vectors are updated one by one. For newly arrived terms that have not participated in the first clustering, their embedding vector features, routing history, and arrival time are directly extracted and concatenated into the initial target feature vector, i.e., the latest target feature vector. For terms that have participated in the first clustering but have not been scheduled, their embedding vector features are re-extracted, and combined with the newly added context information, updated routing history, and refreshed waiting time features corresponding to the arrival time, they are re-concatenated into the latest target feature vector.

[0125] Furthermore, the old task group information before re-clustering is cleared. Based on the latest target feature vectors of unscheduled words, the complete clustering and grouping process is re-executed, ultimately generating several task groups with compact features that fit the current word state; these are the updated task groups. For each updated task group, the latest target feature vectors of the words within the group are extracted, input into a preset expert predictor, and the probability value of each word activating each expert is re-predicted, thus determining the new set of prediction experts corresponding to each updated task group.

[0126] Furthermore, the latest average waiting time for the terms within the update task group is calculated. Then, based on the new set of prediction experts, the historical call frequency of each expert is recalculated, the updated historical hotspot index is calculated, and the final priority of the update task group is determined by combining the new average waiting time and the historical hotspot index.

[0127] The hybrid expert model lexical grouping scheduling method provided in this application, when reaching a preset re-clustering time point or when the number of unscheduled lexical units in the preset cache reaches a preset re-clustering threshold, obtains the latest target feature vectors of the unscheduled lexical units and re-clusters them. This allows for timely capture of dynamic changes in lexical features, making the clustering results more accurately reflect the current similarity and correlation of lexical units, and avoiding inaccuracies in the original clustering due to feature changes. By periodically or conditionally triggering re-clustering, it can adapt to changes in data distribution, ensuring that the generated updated task groups better match the actual situation of the current data and improving the quality of clustering. After regenerating the updated task groups, the corresponding prediction expert set is re-predicted based on these new task groups, making the matching between experts and task groups more accurate and improving the utilization efficiency of expert resources. Updating the priority of each updated task group makes the scheduling strategy more in line with the current situation, improving the system's response speed and processing efficiency.

[0128] As an optional implementation, based on any of the above embodiments, it further includes:

[0129] Obtain the execution performance of each task group, including lexical processing latency, expert utilization, and system throughput.

[0130] The preset waiting time threshold, capacity limit, clustering parameters, and priority weighting coefficient are updated based on execution performance.

[0131] Among them, the word processing latency refers to the total time taken from when a single word enters the buffer until the model completes inference and outputs the result, including the time for the entire process such as waiting for scheduling, expert execution, and data transmission.

[0132] Among them, expert utilization rate refers to the ratio of the actual processing time or number of words processed by each expert in the hybrid expert model within a certain period of time to the total time of that period or the maximum number of words that expert can process.

[0133] Clustering parameters are the core configuration items that affect the clustering grouping effect, including similarity threshold, initial number of clusters in the clustering algorithm, upper / lower limit of task group size, etc.

[0134] The priority weighting coefficient refers to the weight value assigned to the average waiting time and the historical hot spot index when calculating the priority index.

[0135] Understandably, a full lifecycle performance monitoring link should be established for each task group to collect real-time data on lexical processing latency, expert utilization, and system throughput. Each lexical within each task group should be uniquely identified, with timing starting from the moment the lexical enters the buffer and ending when the model outputs the processing result for that lexical, recording the processing latency of each individual lexical. A fixed statistical period should be set, recording the actual working time of each expert within that period. The utilization rate of an individual expert is obtained by dividing the actual working time by the total duration of the statistical period. Within the same statistical period, the cumulative number of all lexicals successfully processed by the system is calculated, and the system throughput per unit time is obtained by dividing the total number by the duration of the statistical period.

[0136] Optionally, preset performance benchmark thresholds, such as upper limit for word processing latency, lower limit for expert utilization, and lower limit for throughput, are used to compare the collected execution performance data with the benchmark thresholds and dynamically update the configuration parameters according to different performance problem scenarios.

[0137] Specifically, if the word processing latency consistently exceeds the baseline threshold, it indicates that the current waiting time threshold is set too long, causing excessive word waiting time. The waiting time threshold should be appropriately reduced, such as from 200ms to 150ms, to expedite the rollback scheduling of timed-out words. If the latency is significantly lower than the threshold but expert utilization is low, it indicates that the waiting time threshold is too short, causing words to be scheduled before sufficient aggregation. The threshold should be appropriately increased to allow more time for clustering and grouping.

[0138] Optionally, the current request pattern can be incorporated into the strategy for dynamically adjusting the preset waiting time threshold of the buffer. Based on the average arrival rate, the adjustment direction of the preset waiting time threshold is determined. If it's a high-rate range, the waiting threshold is lowered to the corresponding value for that range, such as 120ms, and the number of tokens in the buffer is monitored in real time. If the number continues to increase rapidly, the threshold is further lowered slightly. If it's a low-rate range, the waiting threshold is raised to the corresponding value for that range, such as 280ms. If the rate continues to not improve, fine-tuning can continue within a flexible range.

[0139] Furthermore, if the system throughput is lower than the baseline threshold and the buffer has sufficient remaining capacity, it indicates that the current capacity limit is set too small and the word aggregation scale is insufficient. The capacity limit needs to be increased, such as from 100 words to 150 words, to improve batch processing efficiency. If the buffer frequently reaches the capacity limit and word processing latency spikes, it indicates that the capacity limit is too large and word accumulation is severe. The capacity limit needs to be reduced to avoid buffer overflow.

[0140] Furthermore, if expert utilization is low and the overlap of expert activations among terms within a task group is low, it indicates that the clustering similarity threshold is set too high, resulting in overly fine grouping. The similarity threshold needs to be lowered, for example, from 0.8 to 0.6, to expand the feature coverage of each group. If term processing latency fluctuates greatly and some task groups are too large, it indicates that the upper limit of task group size is set too high. The upper limit needs to be reduced, and the initial cluster number of the clustering algorithm should be adjusted to adapt to the current term flow density.

[0141] Furthermore, if the latency of word processing exceeds the standard in latency-sensitive scenarios, it indicates that the weighting of the average waiting time is insufficient. Therefore, its weighting coefficient needs to be increased, such as raising α from 0.4 to 0.6, while reducing the weighting of the historical hotspot index, such as lowering β from 0.6 to 0.4, to prioritize the scheduling of task groups with longer waiting times. If the expert utilization rate remains consistently low, it indicates that the weighting of the historical hotspot index is insufficient. Therefore, its weighting coefficient needs to be increased, while reducing the weighting of the average waiting time, prioritizing the scheduling of task groups matched with high-load experts, and improving resource utilization efficiency.

[0142] The lexical grouping scheduling method for hybrid expert models provided in this application can understand the processing time of different task groups in the system by obtaining the lexical processing latency of each task group, reducing additional scheduling operations triggered by waiting timeouts, lowering system overhead, and improving overall operating efficiency. Combining system throughput and expert utilization to update the capacity limit allows the system to better balance processing capacity and resource load. Updating clustering parameters based on expert utilization makes the clustering results more consistent with the actual processing capacity of expert resources, enabling a more reasonable allocation of expert resources, avoiding situations where some experts are overloaded while others are idle, and improving overall resource utilization efficiency.

[0143] As an optional implementation, based on any of the above embodiments, each word is clustered and grouped according to the target feature vector to generate at least one task group, including:

[0144] A pre-defined task classification model is used to determine the task type corresponding to the current word stream based on the target feature vector.

[0145] The target clustering algorithm is determined based on the task type, and the target clustering algorithm is used to cluster and group each word to generate at least one task group.

[0146] Among them, the pre-trained task classification model refers to a lightweight classification model that can identify the business task type to which the current streaming input word belongs based on the target feature vector of the word.

[0147] Among them, the word stream refers to the sequence of all words that arrive in the buffer continuously in a streaming form, representing the overall input characteristics and business requirements within a certain time period.

[0148] Among them, task type refers to the category divided according to business scenario and word processing characteristics. Different types correspond to different expert activation modes, latency requirements and clustering requirements.

[0149] Among them, target clustering algorithm refers to a clustering algorithm selected for a specific task type and adapted to its characteristics.

[0150] Specifically, the target feature vectors of each word to be clustered in the current buffer are extracted and integrated into a feature set of the word stream, which is then input into a preset task classification model. After receiving the feature set, the model first extracts the core feature dimensions, and then outputs the task type and confidence level corresponding to the current word stream through classification logic, such as real-time dialogue with a confidence level of 95%.

[0151] Understandably, a pre-configured mapping rule library of task types and clustering algorithms is provided. Different task types correspond to suitable clustering algorithms. For example, real-time dialogue tasks have extremely high latency requirements and need to complete clustering quickly, so the online lightweight K-means algorithm is selected. For code generation tasks, the lexical expert activation mode is highly concentrated and the grouping accuracy requirement is high, so the density clustering algorithm is selected.

[0152] Furthermore, a target clustering algorithm is selected according to the mapping rules, the parameter configuration adapted to the algorithm is loaded and the clustering operation is performed. The feature similarity / distance between words is calculated according to the core logic of the algorithm. According to the algorithm's judgment rules, words with matching features are grouped into the same group, generating at least one task group.

[0153] The hybrid expert model-based word grouping and scheduling method provided in this application's embodiments pre-determines that the task classification model can deeply analyze the current word stream based on the target feature vector, accurately identifying its corresponding task type. By first determining the task type, the essential characteristics of words can be grasped more accurately, providing a more reliable basis for subsequent clustering operations. Selecting a target clustering algorithm based on the task type makes the clustering process more closely match the actual characteristics of the task. Choosing a clustering algorithm that matches the task type can improve the accuracy and effectiveness of clustering, making the generated task groups more reasonable and representative.

[0154] As an optional implementation, based on any of the above embodiments, the embedding vector features, routing history, and arrival time corresponding to each word are obtained and concatenated to form the target feature vector corresponding to each word, including:

[0155] The embedded vector features are reduced in dimension, and the routing history is encoded and converted into numerical features.

[0156] The dimensionality-reduced embedded vector features, the encoded routing history features, and the arrival time features are concatenated to obtain the target feature vector.

[0157] Dimension reduction refers to the operation of reducing the dimension of high-dimensional embedded vector features using specific methods.

[0158] Encoding conversion refers to the process of transforming non-numerical routing history information into standardized numerical features.

[0159] Numerical features refer to features presented in numerical form, which can be directly used in operations such as vector concatenation and similarity calculation.

[0160] Among them, arrival time feature refers to converting the time information of word units arriving in the buffer into numerical features.

[0161] Specifically, the original embedding vector features of the lexical units are extracted. Methods such as Principal Component Analysis (PCA) are chosen to map the high-dimensional embedding vectors to a low-dimensional space through linear transformation, retaining the principal components with the largest variance, such as reducing the dimensionality from 1024 to 256. If the embedding vectors exhibit non-linear features, such as multilingual semantics, t-SNE (t-distributed stochastic neighbor embedding) or an autoencoder is used for non-linear dimensionality reduction. Core information from the routing history is extracted, including a list of expert IDs assigned to the lexical units, the matching weights of each expert, and the number of assignments. If the number of experts is small, one-hot encoding is used to convert the expert IDs into binary vectors; if the number of experts is large, label encoding is used to map the expert IDs to continuous integers, followed by normalization. Finally, all encoded routing information is integrated into a fixed-length numerical feature vector. The original arrival time, such as the timestamp 1735689600000, is converted into concatenable numerical features. Then, following a preset order, such as embedding vector features, routing history features, and arrival time features, the vectors are concatenated. The values ​​of the three types of features are sequentially linked according to their dimensions to form a complete high-dimensional vector.

[0162] The word grouping scheduling method of the hybrid expert model provided in this application performs dimensionality reduction processing on the embedded vector features. This reduces the data dimensionality and complexity while retaining key information, lowering the dimensionality to tens of dimensions and greatly simplifying subsequent data processing. Routing history is discrete and unstructured data, making direct processing difficult. Encoding it into numerical features gives the data a unified format and structure, facilitating subsequent calculations and analysis. Concatenating the dimensionality-reduced embedded vector features, the encoded routing history features, and the arrival time features allows for the fusion of multi-source information, resulting in a more comprehensive and richer target feature vector.

[0163] Figure 3 A flowchart illustrating a word grouping scheduling method for a hybrid expert model provided in another embodiment of this application is shown below. Figure 3 As shown, the word grouping scheduling method of the hybrid expert model provided in this embodiment includes the following steps:

[0164] S301. In response to the arrival of words in the preset buffer in the form of a stream, the embedded vector features, routing history and arrival time of each word are obtained and concatenated into the target feature vector corresponding to each word. The preset buffer is configured with a preset waiting time threshold and a capacity limit value.

[0165] S302. A preset clustering algorithm is used to calculate the feature similarity between each word based on the target feature vector.

[0166] S303. Group words whose feature distance is less than a preset similarity threshold according to feature similarity to generate at least one task group.

[0167] S304. A preset expert predictor is used to predict the probability value of each word in each task group activating each expert based on the target feature vector.

[0168] S305. Aggregate the probability values ​​of each word activating each expert in each task group to obtain the average probability value of each expert activating each task group.

[0169] S306. Determine the set of prediction experts corresponding to each task group based on the average probability value of each activated expert and the preset prediction rules.

[0170] S307. Obtain the historical call frequency of each expert in the prediction expert set corresponding to each task group.

[0171] S308. Calculate the historical hotspot index for each task group based on historical call frequency.

[0172] S309. A preset weighted algorithm is used to calculate the priority index of each task group based on the average waiting time of each word in each task group and the historical hot spot index.

[0173] S310. Determine the priority of each task group based on the priority index and the preset priority threshold.

[0174] S311. Under the condition of satisfying the preset buffer constraints, schedule each task group according to priority so that the hybrid expert model can perform operations on each word.

[0175] In this embodiment, the implementation method and technical effect of S301-S311 are similar to those of the corresponding solutions in the above embodiments, and will not be repeated here.

[0176] Figure 4 This is a schematic diagram of a word grouping and scheduling device based on a hybrid expert model, provided in an embodiment of this application. Figure 4 As shown, the word grouping scheduling device for the hybrid expert model provided in this embodiment is located in an electronic device. The word grouping scheduling device 40 for the hybrid expert model provided in this embodiment includes: an acquisition module 41, a generation module 42, a determination module 43, and a scheduling module 44.

[0177] Specifically, the acquisition module 41 is used to acquire the embedding vector features, routing history, and arrival time of each word in response to the arrival of words in the preset buffer in the form of a stream, so as to concatenate them into the target feature vector corresponding to each word. The preset buffer is configured with a preset waiting time threshold and a capacity limit. The generation module 42 is used to cluster and group each word according to the target feature vector to generate at least one task group. The determination module 43 is used to determine the priority of each task group based on the average waiting time of each word in each task group and the historical hotspot index. The scheduling module 44 is used to schedule each task group according to priority under the condition of satisfying the constraints of the preset buffer so that the hybrid expert model can perform operations on each word.

[0178] Optionally, the generation module 42, when clustering and grouping each word according to the target feature vector to generate at least one task group, specifically performs the following: using a preset clustering algorithm to calculate the feature similarity between each word based on the target feature vector; grouping words with feature distance less than a preset similarity threshold into one group according to the feature similarity to generate at least one task group; using a preset expert predictor to predict the probability value of each word in each task group activating each expert according to the target feature vector; aggregating the probability values ​​of each word in each task group activating each expert to obtain the average probability value of activating each expert corresponding to each task group; and determining the prediction expert set corresponding to each task group according to the average probability value of activating each expert and the preset prediction rule.

[0179] Optionally, the determining module 43, when determining the priority of each task group based on the average waiting time of each word in each task group and the historical hot spot index, is specifically used for: obtaining the historical call frequency of each expert in the prediction expert set corresponding to each task group; calculating the historical hot spot index corresponding to each task group based on the historical call frequency; calculating the priority index of each task group based on the average waiting time of each word in each task group and the historical hot spot index using a preset weighted algorithm; and determining the priority of each task group according to the priority index and the preset priority threshold.

[0180] Optionally, the scheduling module 44, when scheduling each task group according to priority so that the hybrid expert model can perform operations on each word, specifically performs the following: sorting each task group in descending order of priority to generate a scheduling sequence; based on the prediction expert set corresponding to each task group, merging adjacent tasks in the scheduling sequence with a prediction expert set overlap greater than a preset overlap threshold into an aggregated task group; generating mini-batch execution groups from the aggregated task group and the unmerged task group respectively, sending the mini-batch execution groups to the hybrid expert model, and instructing the model to only activate the corresponding prediction expert set for forward propagation.

[0181] Optionally, the word grouping scheduling device for the hybrid expert model provided in this application also includes a setting module.

[0182] Accordingly, the scheduling module 44 is further configured to, in response to any word in the preset buffer having a waiting time exceeding a preset waiting time threshold, directly schedule the word so that the hybrid expert model can perform operations on the word; the acquisition module 41 is further configured to, in response to multiple words having a waiting time exceeding a preset waiting time threshold at the current time, acquire the current load rate of each expert in the hybrid expert model and the remaining capacity of the preset buffer; the setting module is configured to, if the current load rate is less than a preset low load threshold and the remaining capacity is greater than a preset capacity threshold, set the backoff ratio to a first preset ratio; if the current load rate is greater than a preset high load threshold or the remaining capacity is less than a preset capacity threshold, set the backoff ratio to a second preset ratio.

[0183] Optionally, the word grouping scheduling device for the hybrid expert model provided in this application also includes an update module.

[0184] Accordingly, the acquisition module 41 is further configured to acquire the latest target feature vector of unscheduled words in response to reaching the preset re-clustering time point or the number of unscheduled words in the preset cache reaching the preset re-clustering threshold; the generation module 42 is further configured to re-execute the clustering grouping operation based on the latest target feature vector to generate at least one updated task group; the update module is configured to re-predict the corresponding prediction expert set according to the updated task group and update the priority of each updated task group.

[0185] Optionally, the acquisition module 41 is also used to acquire the execution performance of each task group, including word processing latency, expert utilization and system throughput; the update module is also used to update the preset waiting time threshold, capacity limit, clustering parameters and priority weighting coefficient based on the execution performance.

[0186] Optionally, when generating module 42 clusters each word element according to the target feature vector to generate at least one task group, it is specifically used to: determine the task type corresponding to the current word element stream based on the target feature vector using a preset task classification model; determine the target clustering algorithm according to the task type, and use the target clustering algorithm to cluster each word element to generate at least one task group.

[0187] Optionally, the acquisition module 41, when acquiring the embedding vector features, routing history, and arrival time corresponding to each word, and concatenating them into the target feature vector corresponding to each word, is specifically used to: perform dimensionality reduction processing on the embedding vector features and encode the routing history into numerical features; and concatenate the dimension-reduced embedding vector features, the encoded routing history features, and the arrival time features to obtain the target feature vector.

[0188] It should be noted that the technical effects of the hybrid expert model word grouping scheduling device in this embodiment have been explained in the embodiments of the hybrid expert model word grouping scheduling method described above, and therefore will not be repeated in this embodiment.

[0189] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 51 and a memory 52. ​​Optionally, the electronic device 50 further includes a communication component 53. The processor 51, memory 52, and communication component 53 are connected via a bus.

[0190] In the specific implementation process, at least one processor 51 executes computer execution instructions stored in memory 52, causing at least one processor 51 to execute the above-described embodiment of the hybrid expert model word grouping scheduling method.

[0191] The specific implementation process of processor 51 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0192] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0193] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0194] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0195] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the hybrid expert model lexical grouping scheduling method at runtime.

[0196] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0197] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the word grouping scheduling method for hybrid expert models.

[0198] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the hybrid expert model word grouping scheduling method.

[0199] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0200] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0201] The foregoing has provided a detailed description of the word grouping scheduling method and electronic device based on a hybrid expert model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A word grouping and scheduling method using a hybrid expert model, characterized in that, include: In response to the arrival of tokens in a stream in a preset buffer, the preset buffer is configured with a preset waiting time threshold and a capacity limit value; Obtain the embedding vector features, routing history, and arrival time corresponding to each word, and concatenate them to form the target feature vector corresponding to each word; Before making expert activation routing decisions for each word, each word is clustered and grouped according to the target feature vector to generate at least one task group. Obtain the historical call frequency of each expert in the prediction expert set corresponding to each task group; calculate the historical hot spot index corresponding to each task group based on the historical call frequency; calculate the priority index of each task group based on the average waiting time of each word in each task group and the historical hot spot index using a preset weighted algorithm; determine the priority of each task group according to the priority index and the preset priority threshold. Under the constraints of the preset buffer, each task group is scheduled according to priority so that the hybrid expert model can perform operations on each word. The step of clustering and grouping each word based on the target feature vector to generate at least one task group includes: calculating the feature similarity between each word based on the target feature vector using a preset clustering algorithm; grouping words with feature distances less than a preset similarity threshold into one group based on the feature similarity to generate at least one task group; predicting the probability value of each word activating each expert in each task group based on the target feature vector using a preset expert predictor; aggregating the probability values ​​of each word activating each expert in each task group to obtain the average probability value of activating each expert corresponding to each task group; and determining the prediction expert set corresponding to each task group based on the average probability value of activating each expert and a preset prediction rule.

2. The method according to claim 1, characterized in that, The scheduling of task groups according to priority, so that the hybrid expert model can perform operations on each lexical unit, includes: The task groups are sorted in descending order of priority to generate a scheduling sequence; Based on the prediction expert set corresponding to each task group, tasks that are adjacent in the scheduling sequence and whose prediction expert set overlap is greater than a preset overlap threshold are grouped into aggregated task groups. The aggregated task group and the unmerged task group are respectively generated into mini-batch execution groups, which are then sent to the hybrid expert model, and the model is instructed to activate only the corresponding set of prediction experts for forward propagation.

3. The method according to claim 1, characterized in that, Also includes: In response to any word in the preset buffer having a waiting time exceeding a preset waiting time threshold, the word is directly scheduled so that the hybrid expert model can perform operations on the word. In response to the fact that the waiting time of multiple terms exceeds the preset waiting time threshold at the current moment, the current load rate of each expert in the hybrid expert model and the remaining capacity of the preset buffer are obtained. If the current load rate is less than the preset low load threshold and the remaining capacity is greater than the preset capacity threshold, then the rollback ratio is set to the first preset ratio. If the current load rate is greater than the preset high load threshold or the remaining capacity is less than the preset capacity threshold, the rollback ratio will be set to the second preset ratio.

4. The method according to claim 1, characterized in that, After clustering and grouping each word according to the target feature vector to generate at least one task group, the method further includes: In response to reaching a preset re-clustering time point or the number of unscheduled words in the preset cache reaching a preset re-clustering threshold, the latest target feature vector of the unscheduled words is obtained; Based on the latest target feature vector, the clustering grouping operation is re-executed to generate at least one updated task group; The corresponding set of prediction experts is re-predicted based on the updated task groups, and the priority of each updated task group is updated.

5. The method according to claim 1, characterized in that, Also includes: The execution performance of each task group is obtained, including lexical processing latency, expert utilization, and system throughput. The preset waiting time threshold, capacity limit, clustering parameters, and priority weighting coefficient are updated based on the execution performance.

6. The method according to claim 1, characterized in that, The step of clustering and grouping each word based on the target feature vector to generate at least one task group includes: The task type corresponding to the current word stream is determined by using a preset task classification model based on the target feature vector; The target clustering algorithm is determined based on the task type, and the target clustering algorithm is used to cluster and group each word to generate at least one task group.

7. The method according to claim 1, characterized in that, The step of obtaining the embedded vector features, routing history, and arrival time corresponding to each word element, and concatenating them into the target feature vector corresponding to each word element, includes: The embedded vector features are reduced in dimensionality, and the routing history is encoded and converted into numerical features. The dimensionality-reduced embedded vector features, the encoded routing history features, and the arrival time features are concatenated to obtain the target feature vector.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 7.

Images