A collaborative reasoning method and device based on a multi-expert model and a medium

Abstract

The embodiment of the specification discloses a kind of collaborative reasoning method, equipment and medium based on multi-expert model, it is related to collaborative reasoning technical field, method includes: receiving multiple reasoning task requests, to obtain the task characteristic information of each reasoning task;Based on the task characteristic information of each reasoning task and the model registration information obtained in advance, task allocation is carried out to determine at least one allocation expert model corresponding to each reasoning task;The model load of each allocation expert model is predicted, to determine at least one target expert model corresponding to each reasoning task according to the predicted load rate, at least one target expert model corresponding to each reasoning task is adjusted;Each reasoning task is issued to corresponding target expert model, the single model reasoning data of each target expert model is collected by asynchronous message queue, to determine the collaborative reasoning result corresponding to each reasoning task by the single model reasoning data of each target expert model.

Description

Technical Field

[0001] This specification relates to the field of collaborative reasoning technology, and in particular to a collaborative reasoning method, device and medium based on a multi-expert model. Background Technology

[0002] Multi-model collaboration typically employs static routing strategies to allocate inference tasks when handling complex inference tasks. During task scheduling, traditional routing strategies are usually based on preset rules or fixed priorities, failing to dynamically adjust routing directions according to the real-time characteristics of the inference tasks. This leads to unreasonable task allocation and impacts the overall inference efficiency of the system. Furthermore, when processing multiple inference tasks simultaneously, these tasks may be assigned to the same expert model. In actual task allocation, the lack of a dynamic load prediction mechanism for the target model can trigger systemic congestion when tasks are concentrated on the same model instance, resulting in soaring inference latency or even task dropout.

[0003] Therefore, in concurrent reasoning scenarios, the traditional multi-model collaborative reasoning process relies on fixed rules when allocating tasks, ignoring the task characteristics of the reasoning task and the processing capabilities of the models, resulting in unreasonable task allocation and affecting reasoning efficiency. Summary of the Invention

[0004] This specification provides one or more embodiments of a collaborative reasoning method, device, and medium based on a multi-expert model to solve the following technical problem: In concurrent reasoning task scenarios, the traditional multi-model collaborative reasoning process relies on fixed rules when allocating tasks, ignoring the task characteristics of the reasoning task and the processing capabilities of the model, resulting in unreasonable task allocation and affecting reasoning efficiency.

[0005] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a collaborative inference method based on multiple expert models. The method includes: receiving multiple inference task requests to obtain task feature information for each inference task, wherein the task feature information includes data modality type features, complexity feature scores, and real-time feature scores; performing task allocation based on the task feature information of each inference task and pre-acquired model registration information to determine at least one allocation expert model corresponding to each inference task; predicting the model load of each allocation expert model to obtain the prediction load rate corresponding to each allocation expert model, and adjusting at least one allocation expert model corresponding to each inference task according to the prediction load rate to determine at least one corresponding target expert model; distributing each inference task to the corresponding target expert model, and collecting single-model inference data of each target expert model through an asynchronous message queue to determine the collaborative inference result corresponding to each inference task through the single-model inference data of each target expert model.

[0006] This specification provides one or more embodiments of a collaborative reasoning device based on a multi-expert model, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0007] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0008] The above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: Through the technical solutions of the embodiments of this specification, the multi-dimensional analysis of task feature information enables modal features, complexity scores and real-time constraints to jointly form the task's capability requirement vector, avoiding the homogeneous allocation logic that relies on traditional rules, ensuring that high-complexity tasks are matched with precise models, and high-real-time tasks are bound to low-latency resources, effectively reducing resource mismatch and idleness of professional models; secondly, the predictive load control mechanism upgrades static allocation to a continuous elastic adaptation process, predicts the model load before the model instance actually bears it, actively identifies potential overload nodes and adjusts tasks, avoids the chain reaction of systemic congestion, and solves the long-tail latency caused by resource competition in concurrent scenarios; through the dual anchoring of task sub-identifiers and model sub-identifiers, it ensures that the spatiotemporal deviation of discrete outputs does not affect the integrity and accuracy of the final collaborative inference results. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a collaborative reasoning method based on a multi-expert model, provided as an embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a collaborative reasoning device based on a multi-expert model, provided as an embodiment of this specification. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0011] This specification provides a collaborative reasoning method based on a multi-expert model. It should be noted that the execution entity in this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart illustrating a collaborative reasoning method based on a multi-expert model, provided as an embodiment of this specification, is shown below. Figure 1 As shown, the main steps include the following: Step S101: Receive multiple inference task requests to obtain task feature information for each inference task.

[0012] The task's feature information includes data modality type features, complexity feature scores, and real-time feature scores.

[0013] In one embodiment of this specification, multiple inference task requests are received, and the input model inference task is analyzed using a convolutional neural network feature extraction algorithm. The original data features of the inference task are extracted using a pre-trained convolutional neural network. The convolutional neural network integrates a channel attention module and a spatial attention module after the residual block. The classification probability of the data modality type, the task complexity score, and the real-time requirement score are generated through a multi-branch output layer.

[0014] Specifically, the Convolutional Neural Network (CNN) architecture design can be based on an improved ResNet-50 network structure, adding attention mechanism modules to the shallow layers of the network to enhance the extraction capability of key task features. Specifically, a Channel Attention (CAM) module and a Spatial Attention (SAM) module are added after each residual block. By learning the weights of different channels and spatial locations, task-relevant features are highlighted while irrelevant information is suppressed. In the network's output layer, a multi-branch output structure is designed according to different types of task features. For example, for determining the data modality type, a softmax classifier is used to output probability values ​​for modalities such as text, image, and time series; for evaluating task complexity and real-time requirements, a regression layer outputs corresponding values. During training, the network is pre-trained using a large-scale inference task dataset containing labeled samples of different modalities, complexity scores, and real-time requirement scores. After pre-training, the network is fine-tuned using a small-scale labeled dataset for specific application scenarios to improve the model's feature extraction accuracy on specific tasks. In the feature extraction stage, the input model inference task data is preprocessed according to the specified format. For example, image data is normalized and scaled, and text data is segmented and vectorized. Then, the preprocessed data is input into the convolutional neural network. After forward propagation, the key feature vector of the task is obtained.

[0015] Step S102: Based on the task feature information of each inference task and the pre-acquired model registration information, task allocation is performed to determine at least one allocation expert model corresponding to each inference task.

[0016] Based on the task feature information of each inference task and the pre-acquired model registration information, task allocation is performed to determine at least one allocation expert model corresponding to each inference task. Specifically, this includes: determining the task type corresponding to each inference task based on the task feature information of each inference task, and performing type matching in a pre-registered model library according to the model registration information to determine the model resource group corresponding to each inference task; and using a preset dynamic programming algorithm, determining at least one allocation expert model corresponding to each inference task in the model resource group based on the task feature information of each inference task and the model registration information. Using a pre-defined dynamic programming algorithm, based on the task feature information and model registration information of each inference task, at least one allocation expert model is determined for each inference task within the model resource group. Specifically, this includes: calculating at least one planning decision cost data for each inference task based on the task feature information and model registration information; wherein the planning decision cost data consists of cost parameters and accuracy benefit parameters corresponding to allocating each inference task to each model resource within the model resource group, and the cost parameters include task processing time cost and model load increment cost; determining the optimization objective of the dynamic programming algorithm, and based on the optimization objective and at least one planning decision cost data for each inference task, determining the initial task allocation strategy for each inference task within the model resource group; wherein the optimization objective is task completion time and model inference accuracy.

[0017] In one embodiment of this specification, based on the task feature information of each inference task, the task type corresponding to each inference task is determined, such as text task, image task, point cloud task. According to the model registration information, type matching is performed in the pre-registered model library to determine the model resource group of the same type corresponding to the inference task. In the subsequent task allocation process, task allocation is only performed within the model resource group of the same type.

[0018] In one embodiment of this specification, the task allocation problem is modeled as a multi-stage decision-making process, with task completion time and inference accuracy as optimization objectives. State variables are defined as the set of currently assigned tasks and the set of remaining unassigned tasks, and decision variables are which expert model to assign the task to. At each decision stage, based on task characteristics and model information in the model registry, the costs (including task processing time, increased model load, etc.) and benefits (such as improved inference accuracy) resulting from assigning the current task to each model are calculated. Through a recursive formula of dynamic programming, the optimal task allocation strategy is progressively calculated, minimizing the total cost and maximizing inference accuracy while completing all tasks.

[0019] Specifically, the task allocation problem is modeled as a multi-stage decision-making process, with each stage corresponding to a reasoning task to be assigned. The state variable is defined as a binary tuple S. k =(A k U k Among them, A k For the set of currently assigned tasks (including task ID, assignment model ID, and estimated cost), U k The set of remaining unassigned tasks (sorted by task priority); define the decision variable as... ∈{0,1}, xim=1 when task i is assigned to expert model m, otherwise it is 0. The optimization objective is defined as: ,in, The task processing time cost when task i is assigned to expert model m. The incremental cost of model load when task i is assigned to expert model m. The inference accuracy gain when task i is assigned to expert model m. This is the minimum accuracy threshold required by the system.

[0020] Based on task characteristics and model registry information, decision parameters are calculated according to the following rules. Here, we take an industrial quality inspection system as an example, focusing on task processing time cost. ,in For models obtained from the registry m The theoretical latency for processing the baseline task; Models acquired for real-time monitoring m Current task queue length; For the model m The average processing rate can be determined using historical statistics. α Queuing penalty coefficient, defaulted to 0.8. Model load incremental cost. , This represents the estimated memory increment for task i running on model m. The task complexity level can be determined here based on the task complexity score, which is determined by looking up a table. For real-time data acquisition models m The available video memory of the node in question; β Used to represent the resource scarcity coefficient, when <2 GB hour, β =1.5, otherwise β =1.0. Inference accuracy gain , The basic accuracy of model m on the standard test set can be obtained through the model registry. For the task i With model mThe adaptation gain coefficient is determined here by calculating the matching degree between the task feature vector and the model capability vector using cosine similarity.

[0021] Specifically, the task complexity level is determined based on the task complexity score. The estimated GPU memory increment for task i running on model m is determined by looking up a table as follows: Before system deployment, a mapping relationship between task complexity and GPU memory usage is established using a benchmark dataset. First, the complexity dimension is graded according to the core features of the task input data. For example, image tasks are graded according to resolution and number of targets: resolution ≤ 720p and number of targets ≤ 3 are set to level 1; 720p ≤ resolution ≤ 1080p and number of targets ≤ 10 are set to level 2. Levels 3, 4, etc., are not discussed here and can be set according to actual needs. Similarly, point cloud tasks are graded according to the number of points and reconstruction level, and text tasks are graded according to the number of words. Based on practical experience, standard test data for each level is set. Standard test data for each level is input for each expert model, and the peak GPU memory is recorded. A model-specific lookup table is generated based on the test results. The model-specific lookup table includes the peak GPU memory for each model corresponding to the standard test data at different levels and the benchmark GPU memory for this model. The difference between the peak GPU memory and the benchmark GPU memory is used to determine the GPU memory increment. To estimate the memory increment for task i running on model m, first determine the task complexity level corresponding to task i. Then, in the model-specific lookup table, based on model m and task complexity level, look up the peak memory corresponding to the standard test data at different levels to determine the corresponding peak memory. The difference between the peak memory and the baseline memory is determined as the memory increment.

[0022] In one embodiment of this specification, the optimal strategy is solved recursively. Here, the Bellman equation can be solved in reverse order to calculate the optimal task allocation strategy step by step, so that the cost is minimized and the inference accuracy is optimal in the process of completing the task, so as to determine the initial task allocation strategy corresponding to each inference task.

[0023] The above technical solution achieves multi-objective optimization decision-making within the model resource group through dynamic programming algorithms. By simultaneously calculating time cost, incremental load cost, and accuracy gain parameters, it avoids the risk of model overload or accuracy reduction caused by resource-task mismatch in traditional methods while ensuring that inference accuracy meets the target. Finally, the elastic adaptation under constraints incorporates the accuracy threshold as a hard constraint into the optimization objective, dynamically adjusting the resource allocation weights while strictly meeting the accuracy requirements of high-value tasks. This significantly improves the robustness of decision-making in complex production environments compared to fixed rule matching or single-objective optimization methods.

[0024] Step S103: Predict the model load of each allocation expert model, obtain the prediction load rate corresponding to each allocation expert model, and adjust at least one allocation expert model corresponding to each inference task according to the prediction load rate to determine at least one target expert model.

[0025] When multiple inference tasks are requested, among the at least one expert model assigned to each inference task as determined in step S102, there may be cases where a model needs to handle a large number of inference tasks. In this case, under high-concurrency inference task scenarios, it may affect the actual inference load of the model. When tasks are concentrated and allocated to the same model instance, systemic congestion is triggered, leading to a surge in inference latency or even task dropout. For example, model A has different loads under different task planning scenarios. Scenario 1 is that model A plans to execute only one inference task 1, and scenario 2 is that model A plans to execute one inference task 1, one inference task 2, and one inference task 3. In the above two scenarios, the actual load of model A when executing inference tasks is significantly different, which is due to the change in resource requirements caused by the increase in the number of tasks. When model A plans to execute inference task 1, the load is relatively light and concentrated. Since there is only one task, the computing resources of model A (such as GPU / CPU utilization and memory usage) are mainly determined by the requirements of task 1. During inference, resource usage exhibits a single peak shape, and the peak load is entirely determined by the complexity of task 1. After the task is completed, the load quickly drops to an idle state. Furthermore, there is no resource contention; a single task exclusively uses the resources of model A, avoiding concurrency conflicts. Case 2 corresponds to a significantly increased and more complex load. The three tasks may execute concurrently or sequentially. If tasks execute concurrently, the resource consumption of model A may have a cumulative effect, with the peak load higher than in case 1. However, due to resource sharing and contention, and the efficiency loss caused by resource contention, the actual peak load may be lower than the simple arithmetic sum of the load contributions of the three tasks. If tasks execute sequentially, the load duration is prolonged, and the overall resource consumption curve is flatter, but the cumulative load is higher. In addition, multiple tasks sharing the limited resources of model A may cause conflicts; for example, memory access by task 2 may increase the computational latency of task 1, thereby reducing overall efficiency. Ultimately, the total load of model A includes not only the independent contribution of each task but also the additional overhead caused by resource contention. Therefore, before distributing inference tasks to the expert model, it is necessary to consider whether the existing allocation method will lead to an overload of the model. The embodiments in this specification adjust the existing allocation method through load prediction.

[0026] The model load for each allocation expert model is predicted to obtain the prediction load rate for each allocation expert model. Specifically, this includes: based on at least one allocation expert model for each inference task, calculating the allocation inference task set corresponding to each allocation expert model; determining the task priority for each allocation inference task in the allocation inference task set, and determining the task execution strategy for the allocation expert model based on the task priority, wherein the task execution strategy includes any one or more of parallel execution and non-parallel execution; and predicting the model load through the task execution strategy for the allocation expert model to determine the prediction load rate for each allocation expert model executing the allocation inference task set.

[0027] In one embodiment of this specification, based on the at least one allocation expert model corresponding to each inference task obtained in the above steps, a set of allocation inference tasks corresponding to each allocation expert model is statistically analyzed. The set of allocation inference tasks includes at least one allocation expert model. When multiple allocation expert models exist in the set, the task priority corresponding to each allocation inference task in the allocation inference task set is determined. Here, the task priority can be a priority identifier in the inference task request, used to indicate the user's execution time requirement for this inference task. When determining the task execution strategy corresponding to the allocation expert model based on the task priority, if the task priorities are all different, it is determined that this model needs to be executed sequentially in descending order of priority, that is, the task execution strategy is parallel execution; if the task priorities are the same, it is determined that the task execution strategy of this model is non-parallel execution.

[0028] By using the task execution strategy corresponding to the allocation expert model, model load prediction is performed to determine the predicted load rate for each allocation expert model executing the allocation inference task set. Specifically, this includes: when the task execution strategy is parallel execution, obtaining the complexity features of each allocation inference task in the allocation inference task set, wherein the complexity features include input data parameters and operation steps; based on the complexity features of each allocation inference task and the pre-acquired single-task memory usage benchmark data of the allocation expert model, performing task-level load prediction to determine the load contribution index corresponding to each allocation inference task; and using the load contribution index corresponding to each allocation inference task, performing model-level load aggregation to determine the predicted load rate for the allocation expert model executing the allocation inference task set.

[0029] In one embodiment of this specification, the load on the prediction model is adjusted before the actual inference task is executed to avoid resource overload and performance degradation. Prediction is based on task-specific features and model architecture information. The predicted load contribution of each task is aggregated, and an adjustment factor for resource contention is added based on the number of tasks. The prediction process consists of two steps: task-level load prediction and model-level load aggregation.

[0030] When the task execution strategy is parallel execution, firstly, the complexity features of each assigned inference task in the assigned inference task set are obtained. These complexity features include input data parameters and the number of operation steps. It should be noted that these complexity features differ from the complexity feature scoring in step S101. These complexity features can be obtained through the metadata of the inference task request, including the input data size and predicted operation steps. These features are parsed from the request header or configuration file when the task is received. Then, based on the complexity features of the assigned inference tasks and the baseline data of the single-task memory usage of the assigned expert model, task-level load prediction is performed, calculating the load contribution of each task (task 1, task 2, task 3). The complexity features of each task and the baseline data of the single-task memory usage of the model are input, and the contribution value of each task is output through a preset basic load prediction function.

[0031] It should be noted that the single-task memory usage benchmark refers to the minimum memory resources required by this model to process a single inference task under typical workloads, obtained through offline benchmarking. The offline benchmarking process is as follows: the target expert model is deployed in an isolated environment, a standard test dataset is input to perform inference, peak memory usage is recorded using memory monitoring tools, and the median memory usage from multiple tests is taken as the single-task memory usage benchmark.

[0032] In one embodiment of this specification, the expression for the basic load prediction function is as follows: Where C is the complexity feature score, which can be obtained through the task feature information in step S101, i.e. the complexity feature score, and α and β represent resource weight coefficients, which are obtained by regression fitting of historical data, α+β=1; This serves as a baseline for model memory usage, referring to the steady-state memory consumption when the model has completed initialization but has not yet processed any tasks. This serves as the benchmark for model computation, derived from hardware performance reports, and represents the baseline computational cost for the model to handle tasks of unit complexity. and These are the reference baseline values ​​for the memory baseline and the computation baseline. After obtaining the load contribution metric corresponding to each assigned inference task (i.e., the predicted load contribution value for each assigned inference task), model-level load aggregation is performed, aggregating the contributions of all tasks and adding a resource contention adjustment factor.

[0033] The predicted load obtained after aggregation ,in, This refers to the basic load prediction corresponding to the model when executing task i, where K(N) is the concurrency adjustment factor. When the task execution strategy is parallel execution, Where γ is the maximum contention loss, typically 0.1~0.3, and here we can take the maximum value of 0.3; λ is the attenuation coefficient, which is determined by the hardware architecture, such as 0.15 for the NVIDIA Ampere architecture; θ is the basic parallel efficiency, which can be set to θ=0.7; and the exponential term... The simulation addresses the nonlinear decay caused by resource contention. If the task execution strategy is sequential processing, i.e., non-parallel execution, K(N) is set to 1, meaning the predicted load is the sum of the task load contributions, with no concurrent contention and no adjustment factor is added.

[0034] Through the above technical solution, the task-level load contribution decouples the resource consumption of each inference task into two orthogonal dimensions: memory and computation. A prediction function is constructed based on single-task memory usage and computation benchmarks, making the process of transforming the task complexity characteristics corresponding to the input data scale and number of operation steps into resource requirements have interpretable physical meaning, avoiding the systematic bias caused by traditional empirical estimation methods that ignore the inherent static load of the model. Secondly, the explicit quantification of the concurrency competition effect innovatively introduces an exponential adjustment factor K(N), whose mathematical form characterizes the nonlinear decay law of resource contention during multi-task parallelism. As the number of tasks N increases, the exponential term simulates the efficiency loss caused by hardware resource conflicts, while the constant term θ retains the basic efficiency of the parallel architecture, based on hardware architecture characteristics (such as NVIDIA). Ampere's dynamic modeling (λ=0.15) makes the prediction results more accurate. Finally, in parallel mode, the competition factor is activated to warn of resource overload risk, and in sequential mode, the linear superposition of load superposition is maintained, which can meet the needs of various heterogeneous scenarios. Resource bottlenecks can be predicted before task scheduling, providing a basis for dynamically adjusting the model allocation strategy, thereby reducing the risk of chain system collapse caused by load prediction distortion.

[0035] Based on the predicted load rate, at least one assigned expert model corresponding to each inference task is adjusted to determine at least one target expert model. Specifically, this includes: selecting designated assigned expert models with a predicted load rate greater than a preset load rate threshold based on the predicted load rate of each assigned expert model to determine at least one designated inference task corresponding to the designated assigned expert model; determining a target adjusted inference task among the at least one designated inference task according to the pre-acquired task priority of the inference task; determining the model resource group corresponding to the designated assigned expert model, reallocating within the model resource group, and determining the adjusted expert model corresponding to the target adjusted inference task to determine at least one target expert model corresponding to each inference task.

[0036] In one embodiment of this specification, when the prediction load rate corresponding to the assigned expert model is greater than a preset load rate threshold, it indicates that the assigned task corresponding to this assigned expert model is expected to increase the model load rate and affect task execution efficiency, requiring reallocation of the specified inference task corresponding to this assigned expert model. It should be noted that the preset load rate threshold here can be set through the model rules of each model, using the maximum load rate that does not affect inference efficiency as the preset load rate threshold. During the reallocation process, the reallocation task is determined according to the task priority of the multiple inference tasks originally assigned to this model. The multiple specified inference tasks are sorted in ascending order of priority, and the lowest-priority inference task at the top is selected as the target adjustment inference task. Load prediction is then performed on the model according to the remaining inference tasks. If the prediction load rate is still greater than the preset load rate threshold, the second-ranked inference task and the lowest-priority inference task are selected as the target adjustment inference tasks. Load prediction is then performed on the model again for the remaining inference tasks, and so on, until the prediction load rate of the remaining inference tasks is no greater than the preset load rate threshold. If multiple specified inference tasks have the same priority, the target adjustment inference task is determined from at least one specified inference task according to the complexity feature score from high to low. Following step S102, the target adjustment inference task is reallocated among the other models in the model resource group corresponding to the specified allocation expert model, and the target expert model corresponding to the target adjustment inference task is determined.

[0037] The above technical solution uses low-priority tasks as a safety buffer for initial release, ensuring high-value tasks have execution privileges in the original model and avoiding core business quality fluctuations caused by traditional polling or random migration. Secondly, it approaches the load threshold through atomic operations of migrating single tasks, verifying the load, and migrating again if the limit is exceeded. At the same time, it utilizes the capability isomorphism within the model resource group to ensure that the migrated tasks still meet the original accuracy and timeliness constraints on the replacement model.

[0038] Step S104: Distribute each inference task to the corresponding target expert model, collect single-model inference data of each target expert model through an asynchronous message queue, and determine the collaborative inference result corresponding to each inference task through the single-model inference data of each target expert model.

[0039] In one embodiment of this specification, each inference task is distributed to the corresponding target expert model. After distribution, a reinforcement learning algorithm can be introduced to continuously monitor the real-time environment state. The agent in the reinforcement learning environment makes dynamic routing decisions. Its state includes the current task queue, model load, and task characteristics. Its action is to assign tasks to a specific expert model or model combination. The reward function is designed based on task completion time, inference accuracy, and model load balancing. For example, if a task is completed within a specified time with high inference accuracy and balanced model load, a higher reward is given; otherwise, a lower reward is given. A Deep Q-Network (DQN) algorithm is used as the reinforcement learning implementation method. By continuously interacting with the reinforcement learning environment, it learns the optimal task allocation strategy. During training, an experience replay mechanism is used to address data correlation issues, improving the algorithm's stability and convergence speed. Reinforcement learning continuously learns from environment feedback to fine-tune decisions. Through dynamic programming and reinforcement learning, a collaborative decision-making framework of offline coarse screening and online optimization is constructed, achieving dynamic complementarity through state sharing and policy fusion.

[0040] Specifically, GPU utilization, queue backlog, and remaining task deadlines for each model node are collected via data acquisition, and the state vector is updated every second. When environmental fluctuations exceed a threshold, reinforcement learning intervenes, inputting the current state into the DQN network. The network structure includes a 128-unit LSTM layer to capture temporal dependencies and outputs a policy correction Δ. For example, the original policy assigns the gear size measurement task to model M1, but DQN detects that the real-time latency of M1 has spiked by 50%, so it generates an action Δ to dynamically switch the task to the backup model M2, reducing accuracy but decreasing latency. After the policy is executed, actual metrics are collected, such as task completion time of 108ms, accuracy of 96.5%, and load variance of 0.07. Multi-dimensional reward values ​​are calculated, such as time reward +1.0, accuracy reward +0.8, and balance reward +0.9. These are stored in the training pool through a priority experience replay mechanism to drive the Q-network parameter updates.

[0041] Furthermore, during model loading, an association mapping table can be established between task features and expert models. For example, if the task is image classification, the association mapping table will list expert models proficient in image classification. When a new inference task is received, the association mapping table is queried to activate only the expert models related to that task. To improve the flexibility and adaptability of the activation strategy, a rule engine-based approach is adopted. Custom rules for the association between task features and models can also be defined according to actual needs. Rules can be written based on logical expressions, such as "when the task's data modality is image and the task complexity is high, activate model A and model B." While activating relevant models, inactive models undergo resource release or freezing operations. For model parameters in memory, the parameter data of inactive models is moved to the disk cache to release memory space; for resources such as threads and processes during model runtime, resources related to inactive models are frozen to avoid resource waste. Simultaneously, a resource monitoring mechanism is established to monitor system resource usage in real time. When system resources are scarce, the activation status of models is dynamically adjusted according to task priority and model importance to ensure that critical tasks receive sufficient resource support.

[0042] The collaborative inference result for each inference task is determined using the single-model inference data of each target expert model. Specifically, this includes: acquiring inference result data and inference result identifiers from the single-model inference data, where the inference result identifier includes an expert model sub-identifier and an inference task sub-identifier; integrating the inference result data based on at least one target expert model and the inference result identifier for each inference task to determine the collaborative inference result for each inference task. The process of integrating the inference result data based on at least one target expert model and the inference result identifier for each inference task to determine the collaborative inference result for each inference task specifically includes: filtering multiple inference result data sets that match the inference task sub-identifier to obtain the expert model sub-identifier corresponding to the inference result data; determining the model execution list for each inference task based on at least one target expert model; and integrating the inference result data when the model execution list and the expert model sub-identifier corresponding to the inference result data meet preset requirements to determine the collaborative inference result for each inference task.

[0043] In one embodiment of this specification, each inference task is distributed to the corresponding target expert model, and single-model inference data for each target expert model is collected through an asynchronous message queue. First, inference result data carrying dual sub-identifiers is extracted from the message queue, including the inference result and the model sub-identifier and task sub-identifier that generated the result. Assuming the model sub-identifier is PointNet++ model identifier M007 and the task sub-identifier is TS-202309, then a task sub-identifier matcher filters all data packets related to the current target inference task. For example, from 200 concurrent tasks, 10 results carrying the TS-202309 identifier are precisely separated, and the corresponding expert model sub-identifier set is extracted. Simultaneously, the task scheduling center synchronously generates the actual model execution list for the inference task, such as the initial allocation scheme determined after load adjustment as [KPConv_M005, PointNet++_M007]. The core verification logic lies in the completeness matching of the execution list and result identifiers. Only when the collected result data package covers all models in the list (e.g., simultaneously containing the outputs of M005 and M007) and there is no redundant data from models outside the list is the preset requirement satisfied. At this point, the fusion engine starts and performs fusion according to a preset method, such as integrating multiple model outputs using a weighted voting mechanism. In financial risk control tasks, the fraud detection results of the NLP model (confidence 0.92) and the graph neural network model (confidence 0.88) are fused using Bayesian methods to ultimately generate the TS-202309 task final-state inference result carrying a collaborative decision identifier. If a missing model identifier is detected during this process (e.g., KPConv_M005 does not return a result due to node failure), cross-resource group model reallocation and result recalculation are immediately triggered to ensure a complete closed loop in the decision-making chain.

[0044] The above technical solution utilizes expert model sub-identifiers and inference task sub-identifiers to construct a data lineage traceability system. By strictly binding the generation relationship between the result data and the original task / model through sub-identifiers, it ensures that the fusion process is not interfered with by other concurrent tasks, reducing the risk of data cross-contamination caused by asynchronous delays in traditional timestamp matching mechanisms. Secondly, the precise verification of the dynamic execution list is based on a real-time updated model execution list, recording the set of target expert models actually participating in a certain inference task, and filtering valid inference data that meets preset requirements. This closed-loop verification mechanism can still maintain the legitimacy verification of the result source when the identifier drifts due to model instance failure reconstruction, ensuring the integrity constraints of multi-model collaborative conclusions. By decoupling the strong binding relationship between result identifiers and physical instances, when a task is migrated to a backup model in the same group, the newly generated inference results can still be seamlessly integrated into the fusion process because they inherit the original task sub-identifier, ensuring the integrity of collaborative conclusions even in distributed node failure or elastic scaling scenarios.

[0045] Through the technical solutions of the embodiments in this specification, the multi-dimensional analysis of task feature information enables modal features, complexity scores, and real-time constraints to jointly form a task capability requirement vector. This avoids the homogeneous allocation logic that relies on traditional rules, ensuring that high-complexity tasks are matched with precise models and high-real-time tasks are bound to low-latency resources, effectively reducing resource mismatch and idleness of professional models. Secondly, the predictive load control mechanism upgrades static allocation to a continuous elastic adaptation process. By predicting the model load before the model instance actually bears the load, it actively identifies potential overload nodes and adjusts tasks, avoiding the chain reaction of systemic congestion and solving the long-tail latency caused by resource competition in concurrent scenarios. Through the dual anchoring of task sub-identifiers and model sub-identifiers, it ensures that the spatiotemporal deviation of discrete outputs does not affect the integrity and accuracy of the final collaborative inference results.

[0046] This specification also provides a multi-model collaborative architecture. First, it defines a modular multi-expert model fusion framework and collaborative interface protocol, including a model access layer, an intermediate adaptation layer, and a collaborative inference layer. The model access layer provides a unified registration interface for different types of expert models, such as natural language, multimodal, temporal, and mechanistic models, supporting dynamic updates and deregistration of models. Each expert model must adhere to the framework's predefined interface specifications, including input / output formats and model description information (such as supported data modality types, inference task types, etc.).

[0047] The registration interface for the model access layer is designed as follows: A unified registration interface based on RESTful style is defined, and the interface uses JSON format for data interaction. Model developers submit a registration request containing basic model information through the interface. Specific information includes the model name, unique identifier, supported data modality types (e.g., natural language processing models need to specify supported languages ​​and text length ranges; image models need to specify supported image formats and resolution ranges), inference task type (e.g., classification, regression, generation), input / output format (detailed description of input data field names, data types, dimensions, and output data structure and meaning), and version number. When the expert model is updated, the registration interface is called again, passing in the updated model description information and the new version number. The system verifies the legality of the updated content, such as whether the input / output formats are compatible. If the verification passes, the relevant information in the model registry is updated, and a model update notification is sent to the intermediate adaptation layer and collaborative inference layer via a message queue. For model deregistration, developers call the deregistration interface, passing in the model's unique identifier. The system deletes the model information from the model registry, stops all task scheduling related to that model, and clears relevant cache resources.

[0048] To address the heterogeneity of different model outputs, a standardized mapping rule library is constructed. Taking natural language processing (NLP) and image recognition models as examples, the intermediate adaptation layer maps the original output format of the NLP model (which outputs text classification labels and probability values, and the image recognition model (which outputs object coordinates and category information) to a unified JSON format structure, such as {"task_id": "xxx", "model_name": "xxx", "result_type": "classification", "results": [ {"label": "xxx", "probability": "xxx"} ]}. The specific mapping process is implemented by writing extensible adapter plugins, each corresponding to a specific type of model output format conversion. Developers can add or modify plugins as needed. When encountering situations where the model output format cannot be directly adapted, the intermediate adaptation layer uses a combination of data conversion and semantic mapping. First, data conversion tools are used to convert the raw data into a common data format, such as converting a special image encoding format into a common JPEG or PNG format. Then, semantic mapping rules are used to match the semantic information in the converted data with the corresponding fields in the standardized format. For semantic information that cannot be directly mapped, the system logs the information and triggers a manual intervention process, whereby the administrator formulates temporary mapping rules based on the specific situation to ensure the integrity and accuracy of the data.

[0049] The collaborative inference layer maintains a task queue to store pending inference tasks. When a new task is received, it is retrieved from the queue and assigned to the corresponding expert model based on the decision of the dynamic routing and scheduling engine. To ensure fairness and efficiency in task allocation, a priority-based task scheduling algorithm is used, assigning higher priority to tasks with high real-time requirements and prioritizing their allocation to models that are fast and proficient in that task type. Simultaneously, the collaborative inference layer monitors the task processing progress of each model in real time, and automatically reassigns some tasks to other models with lower workloads when a model experiences task backlog.

[0050] A binary data communication format based on Protocol Buffers is defined, which features efficient serialization and deserialization performance, reducing bandwidth consumption and processing time during data transmission. Regarding communication rules, data transmission must include fields such as task identifier, source model identifier, target model identifier, data type, data content, and verification information. During data transmission, an encrypted transmission protocol (such as TLS) is used to ensure data security, and a timeout retransmission mechanism is implemented. If data does not reach the target model within a specified time, it is retransmitted to ensure reliable data transmission. Furthermore, the collaborative inference layer adopts an asynchronous communication method, using an event-driven model to achieve non-blocking task processing and improve the system's concurrent processing capabilities. Specifically, an asynchronous message queue (such as RabbitMQ) is used for task and data transfer. After the model completes the inference task, it sends the result to the message queue, and the collaborative inference layer retrieves and integrates the results from the message queue.

[0051] Secondly, a dynamic routing and scheduling engine module for inference tasks is constructed. This module is used to automatically classify and route inference task features, and includes three sub-modules: task feature extraction, dynamic routing decision, and routing distribution. The task feature extraction module uses convolutional neural network feature extraction algorithms to analyze the input model inference task, extracting key features such as data modality type (text, image, time series, etc.), task complexity, and real-time requirements. The dynamic routing decision module, based on task features and model information in the model registry, uses dynamic programming and reinforcement learning algorithms to select the most suitable expert model or model combination for each task. It needs to comprehensively consider factors such as model load, preferred modality types, and inference accuracy to achieve reasonable task allocation. The routing distribution module accurately distributes tasks to the corresponding expert models based on the decision results of the dynamic routing decision module, and collects and returns the inference results to the collaborative inference layer after task processing. During task distribution, a unique task identifier is generated for each task, and the task identifier, input data, and relevant task parameters are sent to the model. After the model completes the inference task, it packages the inference results and task identifier and returns them to the routing distribution module via a message queue. The routing and distribution module matches the results with the original tasks based on the task identifier and stores the collected results in a result cache, awaiting further processing by the collaborative inference layer. During task distribution and result collection, if network failures or model service anomalies occur, the routing and distribution module records error information and adopts different handling strategies based on the error type. For recoverable errors, such as brief network interruptions, an exponential backoff algorithm is used for retries, meaning the interval between each retry increases exponentially to avoid network congestion caused by frequent retries. If multiple retries fail, the task is marked as failed, and the error information is returned to the collaborative inference layer, which decides whether to reassign the task or notify the user of the task failure.

[0052] Furthermore, a sparse activation mechanism and a cold start acceleration module are constructed. The sparse activation mechanism is defined so that, during model loading, only expert models relevant to the current inference task are activated based on the characteristics of that task, reducing unnecessary computation and resource consumption. Secondly, during the model cold start phase, preloading and caching techniques are used to preload the basic parameters of commonly used models and cache them in computer memory. When the model is needed, the parameters can be quickly retrieved from the cache, reducing model initialization time. Finally, incremental loading is adopted, loading only the incremental model parameters required for processing the current inference task, further improving model loading speed.

[0053] During system startup, a list of commonly used models is determined by analyzing historical task data and model usage frequency. For these commonly used models, their basic parameters are pre-loaded into computer memory and managed using an LRU (Least Recently Used) caching strategy. When a model parameter is accessed, it is moved to the head of the cache queue; when cache space is insufficient, model parameters at the tail of the queue are evicted. To further improve caching efficiency, model parameters are compressed for storage and decompressed upon loading into memory. Simultaneously, a cache invalidation mechanism is established; when a model is updated, the corresponding model's cache is automatically cleared to ensure the use of the latest model parameters. When the model processes inference tasks, in addition to the pre-loaded basic parameters, only the incremental model parameters required for the current inference task are loaded, based on the specific needs of the task. The determination of incremental parameters is achieved by analyzing task characteristics and model structure. For example, for image classification tasks, if the resolution and number of categories of the input image are known, the parameters of specific convolutional and fully connected layers that need to be loaded can be calculated as incremental parameters based on the model design. Incremental parameters are loaded asynchronously in the background, while the model begins processing the task, avoiding delays in task processing due to parameter loading. Meanwhile, the loading process of incremental parameters is monitored, and if any abnormality occurs during the loading process, corresponding remedial measures are taken in a timely manner, such as reloading or using backup parameters.

[0054] By defining a heterogeneous model fusion framework and collaborative interface protocol, pluggable integration of different types of expert models, such as natural language, multimodal, temporal, and mechanistic models, is achieved. Users can easily add or replace models according to actual needs without large-scale modifications to the system architecture, greatly improving the system's scalability and adaptability. The dynamic routing and scheduling engine can automatically classify and route tasks based on their real-time characteristics, achieving reasonable task allocation and avoiding the limitations of static routing strategies. By considering factors such as model load and domain expertise, it ensures that each task is assigned to the most suitable model or model combination, improving the overall inference efficiency of the system and reducing resource waste and inference latency. The sparse activation mechanism and cold start acceleration module effectively reduce the time and resource consumption required during model loading. Especially during the cold start phase, the model initialization time is significantly shortened, improving the system's response speed and better meeting the real-time requirements of applications such as equipment fault prediction.

[0055] This specification also provides an embodiment of a collaborative reasoning device based on a multi-expert model, such as... Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0056] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0057] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0058] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A collaborative reasoning method based on a multi-expert model, characterized in that, The method includes: Receive multiple inference task requests to obtain task feature information for each inference task, wherein the task feature information includes data modality type features, complexity feature scores, and real-time feature scores. Based on the task feature information of each inference task and the pre-acquired model registration information, task allocation is performed to determine at least one allocation expert model corresponding to each inference task; The model load of each of the assigned expert models is predicted, and the prediction load rate corresponding to each of the assigned expert models is obtained. Based on the prediction load rate, at least one assigned expert model corresponding to each inference task is adjusted to determine at least one target expert model. Each inference task is sent to the corresponding target expert model, and the single-model inference data of each target expert model is collected through an asynchronous message queue, so as to determine the collaborative inference result corresponding to each inference task through the single-model inference data of each target expert model; Based on the task feature information of each inference task and the pre-acquired model registration information, task allocation is performed to determine at least one allocation expert model corresponding to each inference task, specifically including: Based on the task feature information of each inference task, the task type corresponding to each inference task is determined, and type matching is performed in the pre-registered model library according to the model registration information to determine the model resource group corresponding to each inference task. Using a preset dynamic programming algorithm, based on the task feature information of each inference task and the model registration information, at least one allocation expert model corresponding to each inference task is determined in the model resource group; The optimization objective is defined as follows: ,in, The task processing time cost when task i is assigned to expert model m. The incremental cost of model load when task i is assigned to expert model m. The inference accuracy gain when task i is assigned to expert model m. This is the minimum accuracy threshold required by the system. Task processing time cost ,in For models obtained from the registry m The theoretical latency for processing the baseline task; Models acquired for real-time monitoring m Current task queue length; For the model m The average processing rate is determined through historical statistical values; α This is the queuing penalty coefficient; Model load incremental cost , This represents the estimated memory increment for task i running on model m. Here, the task complexity level is determined based on the task complexity score, which is determined by looking up a table. For real-time data acquisition models m The available video memory of the node in question; β Used to represent the resource scarcity coefficient; Inference accuracy gains , This is used to represent the basic accuracy of model m on the standard test set, and is obtained through the model registry. For the task i With model m The adaptation gain coefficient is determined here by calculating the matching degree between the task feature vector and the model capability vector using cosine similarity. Based on the task execution strategy corresponding to the allocation expert model, model load prediction is performed to determine the predicted load rate corresponding to the allocation inference task set executed by each allocation expert model, specifically including: When the task execution strategy is parallel execution, task-level load prediction is performed based on the complexity characteristics of each assigned inference task and the pre-acquired single-task memory usage benchmark data of the assigned expert model to determine the load contribution index corresponding to each assigned inference task. The single-task memory usage benchmark refers to the minimum memory resources required by this model to process a single inference task under typical workload. By using the load contribution metrics corresponding to each of the allocation inference tasks, model-level load aggregation is performed to determine the predicted load rate corresponding to the allocation expert model executing the set of allocation inference tasks.

2. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that, Using a pre-defined dynamic programming algorithm, based on the task feature information of each inference task and the model registration information, at least one expert model is assigned to each inference task from the model resource group, specifically including: Based on the task feature information and the model registration information, at least one planning and decision cost data corresponding to each inference task is calculated. The planning and decision cost data consists of cost parameters and accuracy benefit parameters corresponding to allocating each inference task to each model resource in the model resource group. The cost parameters include task processing time cost and model load increment cost. The optimization objective of the dynamic programming algorithm is determined, and based on the optimization objective and at least one planning decision cost data corresponding to each inference task, an initial task allocation strategy corresponding to each inference task is determined in the model resource group, wherein the optimization objective is the task completion time and the model inference accuracy.

3. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that, The model load of each of the allocation expert models is predicted to obtain the predicted load rate corresponding to each allocation expert model, specifically including: Based on at least one allocation expert model corresponding to each of the inference tasks, a set of allocation inference tasks corresponding to each of the allocation expert models is statistically analyzed. The task priority corresponding to each allocation reasoning task in the allocation reasoning task set is determined, so as to determine the task execution strategy corresponding to the allocation expert model according to the task priority, wherein the task execution strategy includes any one or more of parallel execution and non-parallel execution; By using the task execution strategy corresponding to the allocation expert model, model load prediction is performed to determine the predicted load rate corresponding to the allocation inference task set executed by each allocation expert model.

4. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that, Based on the predicted load rate, at least one allocation expert model corresponding to each inference task is adjusted to determine at least one target expert model, specifically including: Based on the predicted load rate corresponding to each of the allocation expert models, select a specified allocation expert model that is greater than a preset load rate threshold to determine at least one specified inference task corresponding to the specified allocation expert model. According to the pre-acquired task priority of the inference task, the target is determined and the inference task is adjusted in the at least one specified inference task; Determine the model resource group corresponding to the specified allocation expert model, perform reallocation within the model resource group, determine the adjustment expert model corresponding to the target adjustment inference task, and determine at least one target expert model corresponding to each inference task.

5. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that, Using the single-model inference data of each of the target expert models, the collaborative inference result corresponding to each inference task is determined, specifically including: Obtain the reasoning result data and reasoning result identifier from the single-model reasoning data, wherein the reasoning result identifier includes an expert model sub-identifier and a reasoning task sub-identifier; Based on at least one target expert model corresponding to each reasoning task and the reasoning result identifier, the reasoning result data is integrated to determine the collaborative reasoning result corresponding to each reasoning task.

6. The collaborative reasoning method based on a multi-expert model according to claim 5, characterized in that, Based on at least one target expert model corresponding to each inference task and the inference result identifier, the inference result data is integrated to determine the collaborative inference result corresponding to each inference task, specifically including: Filter multiple inference result data that match the inference task sub-identifier to obtain the expert model sub-identifier corresponding to the inference result data; Based on at least one target expert model corresponding to each inference task, determine the model execution list corresponding to each inference task; When the expert model sub-identifiers corresponding to the model execution list and the inference result data meet the preset requirements, the inference result data is integrated to determine the collaborative inference result corresponding to each inference task.

7. A collaborative reasoning device based on a multi-expert model, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-6.

Images