Cooperative reasoning method and device based on multi-expert model and medium

Abstract

The embodiment of the invention discloses a collaborative reasoning method and device based on a multi-expert model and a medium, and relates to the technical field of collaborative reasoning, and the method comprises the steps: receiving a plurality of reasoning task requests, so as to obtain the task feature information of each reasoning task; based on the task feature information of each reasoning task and pre-acquired model registration information, performing task allocation to determine at least one allocation expert model corresponding to each reasoning task; predicting the model load of each distributed expert model, adjusting at least one distributed expert model corresponding to each reasoning task according to the predicted load rate, and determining at least one corresponding target expert model; and issuing each reasoning task to the corresponding target expert model, collecting single model reasoning data of each target expert model through an asynchronous message queue, and determining a collaborative reasoning result corresponding to each reasoning task through the single model reasoning data of each target expert model.

Description

Technical Field

[0001] This specification relates to the field of collaborative reasoning technology, and in particular to a collaborative reasoning method, device, and medium based on a multi-expert model. Background Art

[0002] When handling complex inference tasks, multi-model collaboration typically uses a static routing strategy to allocate inference tasks. During the task scheduling process, traditional routing scheduling strategies are typically based on preset rules or fixed priorities and are unable to dynamically adjust routing directions based on the real-time characteristics of the inference tasks. This leads to irrational task allocation and affects the overall inference efficiency of the system. Furthermore, when processing multiple inference tasks simultaneously, multiple inference tasks may be assigned to the same expert model. In the actual task allocation process, there is a lack of a dynamic load prediction mechanism for the target model, which causes system congestion when tasks are concentratedly assigned to the same model instance, leading to a surge in inference latency and even task abandonment.

[0003] Therefore, in the concurrent reasoning task scenario, the traditional multi-model collaborative reasoning process relies on fixed rules when allocating tasks, ignoring the task characteristics of the reasoning task and the processing capabilities of the model, resulting in unreasonable task allocation and affecting reasoning efficiency. Summary of the Invention

[0004] One or more embodiments of this specification provide a collaborative reasoning method, device, and medium based on multiple expert models to solve the following technical problems: In concurrent reasoning task scenarios, the traditional multi-model collaborative reasoning process relies on fixed rules when allocating tasks, ignoring the task characteristics of the reasoning tasks and the processing capabilities of the models, resulting in unreasonable task allocation and affecting reasoning efficiency.

[0005] One or more embodiments of this specification adopt the following technical solutions: One or more embodiments of the present specification provide a collaborative reasoning method based on multiple expert models, the method comprising: receiving multiple reasoning task requests to obtain task feature information of each reasoning task, wherein the task feature information includes data modality type features, complexity feature scores, and real-time feature scores; performing task allocation based on the task feature information of each reasoning task and pre-acquired model registration information to determine at least one assigned expert model corresponding to each reasoning task; predicting the model load of each assigned expert model to obtain the predicted load rate corresponding to each assigned expert model, and adjusting the at least one assigned expert model corresponding to each reasoning task according to the predicted load rate to determine the corresponding at least one target expert model; sending each reasoning task to the corresponding target expert model, collecting the single model reasoning data of each target expert model through an asynchronous message queue, and determining the collaborative reasoning result corresponding to each reasoning task through the single model reasoning data of each target expert model.

[0006] One or more embodiments of this specification provide a collaborative reasoning device based on a multi-expert model, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0007] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above method.

[0008] At least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the technical solutions of the embodiments of this specification, multi-dimensional analysis of task feature information enables modal features, complexity scores and real-time constraints to jointly form the capability requirement vector of the task, avoiding the homogenized allocation logic that relies on traditional rules, ensuring that high-complexity tasks are matched with precise models, and high-real-time tasks are bound with low-latency resources, effectively reducing resource mismatch and idleness of professional models; secondly, the predictive load regulation mechanism upgrades static allocation to a continuous elastic adaptation process, by predicting the model load before the model instance is actually loaded, actively identifying potential overloaded nodes and adjusting tasks, avoiding the chain reaction of systemic congestion, and solving the long-tail delay caused by resource competition in concurrent scenarios; through the dual anchoring of task sub-identifiers and model sub-identifiers, it ensures that the spatiotemporal deviation of discrete outputs does not affect the integrity and accuracy of the final collaborative reasoning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings: Figure 1 A flowchart of a collaborative reasoning method based on a multi-expert model provided in an embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a collaborative reasoning device based on a multi-expert model provided in an embodiment of this specification. DETAILED DESCRIPTION

[0010] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0011] The embodiments of this specification provide a collaborative reasoning method based on a multi-expert model. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart of a collaborative reasoning method based on a multi-expert model is provided in the embodiment of this specification, such as Figure 1 As shown, it mainly includes the following steps: Step S101: receiving multiple reasoning task requests to obtain task feature information of each reasoning task.

[0012] Among them, the task feature information includes data modality type features, complexity feature scores and real-time feature scores.

[0013] In one embodiment of the present specification, multiple inference task requests are received, the input model inference task is analyzed using a convolutional neural network feature extraction algorithm, the original data features of the inference task are extracted using a pre-trained convolutional neural network, and the convolutional neural network integrates a channel attention module and a spatial attention module after the residual block; the classification probability of the data modality type, the task complexity score and the real-time requirement score are generated through a multi-branch output layer.

[0014] Specifically, the convolutional neural network (CNN) architecture can be designed based on a modified ResNet-50 network structure, with an attention mechanism added to the shallow layers of the network to enhance the ability to extract key task features. Specifically, a channel attention module (CAM) and a spatial attention module (SAM) are added after each residual block. By learning weights for different channels and spatial locations, they highlight task-relevant features and suppress irrelevant information. In the network's output layer, a multi-branch output structure is designed based on the different types of task features. For example, to determine the data modality type, a softmax classifier outputs probabilities for modalities such as text, image, and time series. To assess task complexity and real-time requirements, a regression layer outputs corresponding numerical values. During training, the network is pre-trained using a large-scale inference task dataset containing annotated examples of tasks across different modalities, complexity scores, and real-time requirements. After pre-training, the network is fine-tuned using a small-scale annotated dataset for specific application scenarios to improve the model's feature extraction accuracy for specific tasks. In the feature extraction stage, the input model inference task data is preprocessed according to the prescribed format, such as normalization and scaling of image data, and word segmentation and vectorization of text data. The preprocessed data is then input into the convolutional neural network, and the key feature vector of the task is obtained through forward propagation calculation.

[0015] Step S102 : performing task allocation based on the task feature information of each reasoning task and the pre-acquired model registration information to determine at least one allocated expert model corresponding to each reasoning task.

[0016] Based on the task feature information of each reasoning task and the pre-acquired model registration information, task allocation is performed to determine at least one allocation expert model corresponding to each reasoning task, specifically including: determining the task type corresponding to each reasoning task based on the task feature information of each reasoning task, and performing type matching in the pre-registered model library according to the model registration information to determine the model resource group corresponding to each reasoning task; using a preset dynamic programming algorithm, based on the task feature information of each reasoning task and the model registration information, determining at least one allocation expert model corresponding to each reasoning task in the model resource group. Utilizing a preset dynamic programming algorithm, based on the task feature information of each reasoning task and the model registration information, at least one allocation expert model corresponding to each reasoning task is determined in the model resource group, specifically including: calculating at least one planning decision cost data corresponding to each reasoning task according to the task feature information and the model registration information, wherein the planning decision cost data is the cost parameter and accuracy benefit parameter corresponding to each reasoning task after allocating each model resource in the model resource group, and the cost parameter includes the task processing time cost and the model load incremental cost; determining the optimization target of the dynamic programming algorithm, and determining the initial task allocation strategy corresponding to each reasoning task in the model resource group based on the optimization target and the at least one planning decision cost data corresponding to each reasoning task, wherein the optimization target is the task completion time and the model reasoning accuracy.

[0017] In one embodiment of the present specification, based on the task feature information of each reasoning task, the task type corresponding to each reasoning task is determined, such as a text task, an image task, or a point cloud task. According to the model registration information, type matching is performed in a pre-registered model library to determine a model resource group of the same type corresponding to the reasoning task. In the subsequent task allocation process, task allocation is performed only within the model resource group of the same type.

[0018] In one embodiment of the present specification, the task allocation problem is modeled as a multi-stage decision-making process, with task completion time and reasoning accuracy as optimization objectives. State variables are defined as the set of currently assigned tasks and the set of remaining unassigned tasks, and the decision variable is the choice of which expert model to assign the task to. At each decision stage, the costs (including task processing time, increased model load, etc.) and benefits (such as improved reasoning accuracy) of assigning the current task to each model are calculated based on the task characteristics and the model information in the model registry. Using the recursive formula of dynamic programming, the optimal task allocation strategy is gradually calculated, minimizing the total cost and achieving optimal reasoning accuracy while completing all tasks.

[0019] Specifically, the task allocation problem is modeled as a multi-stage decision process, where each decision stage corresponds to a reasoning task to be assigned. The state variable is defined as a binary tuple S k =(A k ,U k ) Among them, A k is the set of currently assigned tasks (including task ID, assigned model ID, and estimated cost), U k is the set of remaining unassigned tasks (sorted by task priority); define the decision variable as ∈{0,1}, when task i is assigned to expert model m, xim=1, otherwise it is 0. The optimization objective is defined as: ,in, The task processing time cost when task i is assigned to expert model m, The incremental cost of model load when task i is assigned to expert model m, The inference accuracy gain when task i is assigned to expert model m, The minimum accuracy threshold required by the system.

[0020] Based on the task characteristics and model registry information, the decision parameters are calculated according to the following rules. Here, the industrial quality inspection system is taken as an example. The task processing time cost ,in For models obtained from the registry m Theoretical latency for processing benchmark tasks; Models acquired for real-time monitoring m Current task queue length; For the model m The average processing rate can be determined through historical statistics; α The default value is 0.8. , is the estimated memory increment for task i running on model m. Here, the task complexity level can be determined based on the task complexity score by looking up the table. Model for real-time acquisition m The free video memory of the node; β Used to express resource strain coefficient, when <2 GB hour, β =1.5, otherwise β =1.0. Inference accuracy gain , It is used to indicate the basic accuracy of model m on the standard test set, which can be obtained through the model registry; For the task i With model mThe adaptation gain coefficient is determined here by calculating the matching degree between the task feature vector and the model capability vector through cosine similarity.

[0021] Specifically, the task complexity level is determined based on the task complexity score. The estimated memory usage increase for task i running on model m is determined through a table lookup as follows: Before system deployment, a mapping between task complexity and memory usage is established using a benchmark test dataset. First, the complexity dimension is graded based on the core features of the task input data. For example, image tasks are graded based on resolution and number of objects: resolution ≤ 720p and number of objects ≤ 3 is set to level 1, 720p ≤ resolution ≤ 1080p and number of objects ≤ 10 is set to level 2. Levels 3 and 4 are not discussed in detail here and can be set based on actual needs. Similarly, point cloud tasks are graded based on number of points and reconstruction level, and text tasks are graded based on number of tokens. Based on actual experience, corresponding standard test data for each level is set. Each expert model is fed with the standard test data for each level, and the peak memory usage is recorded. Based on the test results, a model-specific lookup table is generated. This table contains the test peak memory usage and the baseline memory usage of each model corresponding to the standard test data at each level. The memory usage increase is determined by the difference between the test peak memory and the baseline memory usage. To estimate the memory increment for task i running on model m, first determine the task complexity level corresponding to task i. Then, in the model-specific lookup table, look up the test peak memory corresponding to standard test data at different levels based on model m and task complexity level, determine the corresponding test peak memory, and determine the memory increment by the difference between the test peak memory and the benchmark memory.

[0022] In one embodiment of the present specification, the optimal strategy is solved recursively. Here, the Bellman equation can be solved recursively in reverse order to gradually calculate the optimal task allocation strategy so that the cost is minimized and the reasoning accuracy is optimized in the process of completing the task, so as to determine the initial task allocation strategy corresponding to each reasoning task.

[0023] Through the above technical solution, multi-objective optimization decision-making is realized within the model resource group through a dynamic programming algorithm. By synchronously calculating the time cost, load incremental cost and accuracy benefit parameters, the risk of model overload or accuracy reduction caused by resource-task mismatch in traditional methods is avoided while ensuring that the reasoning accuracy meets the standards. Finally, elastic adaptation under constraint conditions incorporates the accuracy threshold as a hard constraint into the optimization goal, and dynamically adjusts the resource allocation weight on the basis of strictly meeting the accuracy of high-value tasks. Compared with fixed rule matching or single-objective optimization methods, it significantly improves the robustness of decision-making in complex production environments.

[0024] Step S103 , predicting the model load of each assigned expert model, obtaining the predicted load rate corresponding to each assigned expert model, and adjusting at least one assigned expert model corresponding to each reasoning task according to the predicted load rate, and determining at least one corresponding target expert model.

[0025] When multiple inference task requests are received, the at least one assigned expert model corresponding to each inference task determined in step S102 may need to handle a large number of inference tasks. In this case, in scenarios with high concurrency of inference tasks, the actual inference load of the model may be affected. When tasks are concentrated on the same model instance, system congestion may be triggered, leading to a surge in inference latency or even task abandonment. For example, model A experiences varying loads under different task scheduling scenarios. Case 1: Model A plans to execute only one inference task, 1; Case 2: Model A plans to execute one inference task, one inference task, and one inference task, 3. In these two cases, the actual load on model A when executing inference tasks differs significantly, due to changes in resource requirements caused by the increase in the number of tasks. When model A plans to execute inference task 1, the load is relatively light and concentrated. Because there is only one task, model A's computing resources (such as GPU / CPU utilization and memory usage) are primarily determined by the requirements of task 1. During inference, resource usage exhibits a single peak, which is entirely determined by the complexity of task 1. After the task is completed, the load quickly drops to an idle state. There is no resource competition phenomenon, and a single task monopolizes the resources of model A, avoiding concurrency conflicts. The load corresponding to situation 2 increases significantly and becomes more complicated. The three tasks may be executed concurrently or sequentially. If the tasks are executed concurrently, the resource occupancy of model A may have a superposition effect, and the peak load is higher than situation 1. However, due to resource sharing contention and the efficiency loss caused by resource competition, the actual peak may be lower than the simple arithmetic sum of the load contributions of the three tasks. If the tasks are executed sequentially, the load duration is extended, the overall resource occupancy curve is flatter, but the cumulative load is higher. In addition, multiple tasks sharing the limited resources of model A may cause conflicts. For example, memory access of task 2 may increase the calculation delay of task 1, thereby reducing overall efficiency. Ultimately, the total load of model A includes not only the independent contribution of each task, but also the additional overhead caused by resource competition. Therefore, before sending the inference task to the expert model, it is necessary to consider whether the model load will exceed the norm according to the existing distribution method. The embodiments of this specification adjust the existing distribution method through load prediction.

[0026] The model load of each of the allocation expert models is predicted to obtain the predicted load rate corresponding to each of the allocation expert models, specifically including: based on at least one allocation expert model corresponding to each of the reasoning tasks, counting the allocation reasoning task set corresponding to each of the allocation expert models; determining the task priority corresponding to each of the allocation reasoning tasks in the allocation reasoning task set, and determining the task execution strategy corresponding to the allocation expert model according to the task priority, wherein the task execution strategy includes any one or more of parallel execution and non-parallel execution; performing model load prediction through the task execution strategy corresponding to the allocation expert model, and determining the predicted load rate corresponding to each of the allocation expert models executing the allocation reasoning task set.

[0027] In one embodiment of the present specification, at least one allocation expert model corresponding to each reasoning task obtained according to the above steps is used to count the allocation reasoning task set corresponding to each allocation expert model. The allocation reasoning task set includes at least one allocation expert model. When there are multiple allocation expert models in the set, the task priority corresponding to each allocation reasoning task in the allocation reasoning task set is determined. The task priority here can be the priority identifier in the reasoning task request, which is used to represent the user's execution time requirement for this reasoning task. When determining the task execution strategy corresponding to the allocation expert model based on the task priority, if the task priorities are all different, it is determined that this model needs to be executed in order from high to low priority, that is, the task execution strategy is parallel execution; if the task priorities are the same, it is determined that the task execution strategy of this model is non-parallel execution.

[0028] The task execution strategy corresponding to the allocation expert model is used to perform model load prediction and determine the predicted load rate corresponding to each allocation expert model executing the allocation reasoning task set, specifically including: when the task execution strategy is parallel execution, obtaining the complexity characteristics of each allocation reasoning task in the allocation reasoning task set, wherein the complexity characteristics include input data parameters and the number of operation steps; performing task-level load prediction based on the complexity characteristics of each allocation reasoning task and the pre-acquired single-task memory occupancy benchmark data of the allocation expert model to determine the load contribution index corresponding to each allocation reasoning task; performing model-level load aggregation based on the load contribution index corresponding to each allocation reasoning task to determine the predicted load rate corresponding to the allocation expert model executing the allocation reasoning task set.

[0029] In one embodiment of this specification, the model load is predicted before actually executing an inference task to avoid resource overload and performance degradation. This prediction is based on the task's feature information and the model's architecture. The predicted load contribution of each task is aggregated, and an adjustment factor for resource contention is added based on the number of tasks. The prediction process consists of two steps: task-level load prediction and model-level load aggregation.

[0030] When the task execution strategy is parallel execution, first, the complexity characteristics of each assigned reasoning task in the assigned reasoning task set are obtained. The complexity characteristics include input data parameters and the number of operation steps. It should be noted that the complexity characteristics here are different from the complexity feature score in step S101. The complexity characteristics here can be obtained through the metadata of the reasoning task request, including the input data size and the predicted value of the number of operation steps. These characteristics are parsed from the request header or configuration file when the task is received. Then, task-level load prediction is performed based on the complexity characteristics of the assigned reasoning tasks and the single-task memory usage benchmark data of the assigned expert model, and the load contribution of each task (task 1, task 2, task 3) is calculated respectively. The complexity characteristics of each task and the single-task memory usage benchmark data of the model are input, and the contribution value of each task is output through the preset basic load prediction function.

[0031] It should be noted that the single-task memory usage benchmark refers to the minimum memory resources required by this model to process a single inference task under a typical workload. This is obtained through offline benchmark testing. The offline benchmark testing process is as follows: the target expert model is deployed in an isolated environment, and a standard test dataset is input to perform inference. The peak memory usage is recorded using a memory monitoring tool. The median memory usage of multiple tests is taken as the single-task memory usage benchmark.

[0032] In one embodiment of the present specification, the basic load prediction function is expressed as follows: , where C is the complexity feature score, which can be obtained through the task feature information in step S101, i.e., the complexity feature score, α and β represent resource weight coefficients, which are obtained through historical data regression fitting, α+β=1; is the model memory benchmark, which refers to the steady-state memory usage when the model is initialized but no tasks are processed. It is the model calculation benchmark, which comes from the hardware performance report and is used to represent the benchmark calculation consumption of the model processing unit complexity task. and The reference baseline values ​​for the memory benchmark and the compute benchmark. After obtaining the load contribution indicator corresponding to each assigned inference task, that is, the predicted load contribution value corresponding to each assigned inference task, model-level load aggregation is performed to aggregate all task contributions and add a resource contention adjustment factor.

[0033] The predicted load after aggregation ,in, It refers to the basic load prediction corresponding to the model when executing task i, K(N) is the concurrency adjustment factor. When the task execution strategy is parallel execution, , where γ is the maximum contention loss, usually 0.1~0.3, and here the maximum value can be 0.3, λ is the attenuation coefficient, which is determined by the hardware architecture. For example, NVIDIA Ampere architecture λ is 0.15, θ is the basic parallel efficiency, which can be set to θ=0.7, and the exponential term Simulates nonlinear attenuation caused by resource contention. If the task execution strategy is sequential processing, that is, non-parallel execution, K(N) takes the value of 1, which means that the predicted load is the sum of the task load contributions, there is no concurrency contention, and no adjustment factor is added.

[0034] Through the above technical solution, the task-level load contribution decouples the resource consumption of each inference task into two orthogonal dimensions: memory and computing. A prediction function is constructed based on the single-task memory usage benchmark and computing benchmark, so that the process of converting the task complexity characteristics corresponding to the input data scale and the number of operation steps into resource demand has an interpretable physical meaning, avoiding the systematic deviation caused by the traditional empirical valuation method due to ignoring the inherent static load of the model; secondly, the explicit quantification of the concurrent competition effect innovatively introduces the exponential adjustment factor K(N), whose mathematical form describes the nonlinear attenuation law of resource contention when multiple tasks are parallel. When the number of tasks N increases, the exponential term simulates the efficiency loss caused by hardware resource conflicts, while the constant term θ retains the basic efficiency of the parallel architecture. Based on the characteristics of the hardware architecture (such as NVIDIA Ampere's dynamic modeling with λ=0.15 makes the prediction results more accurate. Finally, in parallel mode, the competition factor is activated to warn of resource overload risks, and in sequential mode, the linear superposition of load superposition is maintained. This can meet the needs of various heterogeneous scenarios, predict resource bottlenecks before task scheduling, and provide a basis for dynamically adjusting the model allocation strategy, thereby reducing the risk of cascading system crashes caused by load forecast distortion.

[0035] According to the predicted load rate, at least one allocation expert model corresponding to each reasoning task is adjusted to determine at least one corresponding target expert model, specifically including: according to the predicted load rate corresponding to each allocation expert model, screening the designated allocation expert model that is greater than the preset load rate threshold to determine at least one designated reasoning task corresponding to the designated allocation expert model; according to the task priority of the reasoning task obtained in advance, determining the target adjustment reasoning task in the at least one designated reasoning task; determining the model resource group corresponding to the designated allocation expert model, reallocating within the model resource group, and determining the adjustment expert model corresponding to the target adjustment reasoning task to determine at least one target expert model corresponding to each reasoning task.

[0036] In one embodiment of the present specification, when the predicted load rate corresponding to an assigned expert model is greater than a preset load rate threshold, it indicates that the assigned task corresponding to this assigned expert model is expected to increase the model load rate and affect task execution efficiency, and the designated reasoning tasks corresponding to this assigned expert model need to be reallocated. It should be noted that the preset load rate threshold here can be set by the model rules of each model, with the maximum load rate that does not affect reasoning efficiency being used as the preset load rate threshold. During the reallocation process, the reallocated tasks are determined based on the task priorities corresponding to the multiple reasoning tasks originally assigned to this model. The multiple designated reasoning tasks are sorted in ascending order of priority, and the first-ranked reasoning task with the lowest priority is designated as the target adjustment reasoning task. Load prediction is then performed for the remaining reasoning tasks for this model. If the predicted load rate is still greater than the preset load rate threshold, the second-ranked reasoning task and the lowest-priority reasoning task are determined as the target adjustment reasoning task. Load prediction is then performed for the remaining reasoning tasks for this model, and so on, until the predicted load rate of the remaining reasoning tasks is no greater than the preset load rate threshold. If multiple designated reasoning tasks have the same task priority, a target adjustment reasoning task is determined in at least one designated reasoning task in descending order of complexity feature scores. According to step S102, the target adjustment reasoning task is reallocated to other models in the model resource group corresponding to the designated allocation expert model, excluding the designated allocation expert model, to determine a target expert model corresponding to the target adjustment reasoning task.

[0037] Through the above technical solution, low-priority tasks are released as a safety buffer zone in the first round, ensuring the execution privileges of high-value tasks in the original model and avoiding fluctuations in core business quality caused by traditional polling or random migration. Secondly, the load threshold is approached through atomic operations such as migrating single tasks, verifying the load, and migrating again if the limit is exceeded. At the same time, the capability isomorphism within the model resource group is utilized to ensure that the migrated tasks still meet the original accuracy and timeliness constraints on the alternative model.

[0038] In step S104, each reasoning task is sent to the corresponding target expert model, and the single model reasoning data of each target expert model is collected through the asynchronous message queue to determine the collaborative reasoning result corresponding to each reasoning task through the single model reasoning data of each target expert model.

[0039] In one embodiment of this specification, each inference task is assigned to a corresponding target expert model. After this assignment, a reinforcement learning algorithm can be introduced to continuously monitor the real-time environment state. The agent in the reinforcement learning environment makes dynamic routing decisions. Its state is the current task queue, model load, and task characteristics. Its action is to assign tasks to a specific expert model or model combination. The reward function is designed based on task completion time, inference accuracy, and model load balance. For example, if a task is completed within the specified timeframe with high inference accuracy and balanced model load, a higher reward is given; otherwise, a lower reward is given. The Deep Q-Network (DQN) algorithm is used as a reinforcement learning implementation. It continuously interacts with the reinforcement learning environment to learn the optimal task allocation strategy. During training, an experience replay mechanism is used to address data correlation issues, improving algorithm stability and convergence speed. Decisions are fine-tuned through continuous feedback from the learning environment through reinforcement learning. By combining dynamic programming and reinforcement learning, a collaborative decision-making framework with offline coarse screening and online optimization is constructed, achieving dynamic complementarity through state sharing and policy fusion.

[0040] Specifically, the GPU utilization, queue backlog, and remaining task deadlines of each model node are collected through tracking, and the state vector is updated every second. When environmental fluctuations exceed the threshold, reinforcement learning intervenes and inputs the DQN network based on the current state. The network structure contains a 128-unit LSTM layer to capture timing dependencies and output a policy correction value Δ. For example, the original policy assigned the gear size measurement task to model M1, but DQN detected that the real-time latency of M1 had soared by 50%. It then generated an action Δ to dynamically switch the task to the backup model M2. Although the accuracy was reduced, the latency was reduced. After the policy is executed, actual indicators are collected, such as task completion time of 108ms, accuracy of 96.5%, and load variance of 0.07. Multi-dimensional reward values ​​are calculated, such as time reward +1.0, accuracy reward +0.8, and balance reward +0.9. These are stored in the training pool through the priority experience replay mechanism to drive the update of the Q network parameters.

[0041] Furthermore, during the model loading process, an association mapping table can be established between task features and expert models. For example, if the task is image classification, the association mapping table will list expert models that excel at image classification. When a new inference task is received, only the expert models relevant to that task are activated by querying the association mapping table. To enhance the flexibility and adaptability of the activation strategy, a rule-based engine is implemented. The association rules between task features and models can be customized based on actual needs. Rules can be written based on logical expressions, such as "When the task's data modality is image and the task complexity is high, activate Model A and Model B." While activating relevant models, resources for inactive models are released or frozen. For in-memory model parameters, parameter data for inactive models is moved to the disk cache to free up memory space. For runtime resources such as threads and processes, resources associated with inactive models are frozen to avoid resource waste. Furthermore, a resource monitoring mechanism is established to monitor system resource usage in real time. When system resources are limited, the activation status of models is dynamically adjusted based on task priority and model importance to ensure sufficient resource support for critical tasks.

[0042] The collaborative reasoning result corresponding to each reasoning task is determined by the single-model reasoning data of each target expert model, specifically including: obtaining the reasoning result data and the reasoning result identifier in the single-model reasoning data, wherein the reasoning result identifier includes the expert model sub-identifier and the reasoning task sub-identifier; integrating the reasoning result data according to the at least one target expert model corresponding to each reasoning task and the reasoning result identifier to determine the collaborative reasoning result corresponding to each reasoning task. The collaborative reasoning result corresponding to each reasoning task is determined by integrating the reasoning result data according to the at least one target expert model corresponding to each reasoning task and the reasoning result identifier, specifically including: screening multiple reasoning result data that match the reasoning task sub-identifier with the reasoning task to obtain the expert model sub-identifier corresponding to the reasoning result data; determining the model execution list corresponding to each reasoning task according to the at least one target expert model corresponding to each reasoning task; when the model execution list and the expert model sub-identifier corresponding to the reasoning result data meet the preset requirements, integrating the reasoning result data to determine the collaborative reasoning result corresponding to each reasoning task.

[0043] In one embodiment of the present specification, each reasoning task is sent to the corresponding target expert model, and the single model reasoning data of each target expert model is collected through an asynchronous message queue. First, the reasoning result data carrying dual sub-identifiers is extracted from the message queue, including the reasoning result and the model sub-identifier and task sub-identifier that generate the result. Assuming that the model sub-identifier is PointNet++ model identifier M007 and the task sub-identifier is TS-202309, all data packets related to the current target reasoning task are then filtered through the task sub-identifier matcher, such as accurately separating 10 results carrying the TS-202309 identifier from 200 concurrent tasks, and extracting the corresponding expert model sub-identifier set; at the same time, the task scheduling center synchronously generates the actual model execution list of the reasoning task, such as the initial allocation plan is determined to be [KPConv_M005, PointNet++_M007] after load adjustment. The core validation logic ensures completeness matching between the execution list and the result identifier. Only when the collected result data package covers all models in the list (for example, including the outputs of both M005 and M007) and contains no redundant data from models outside the list, is the pre-defined requirements considered met? At this point, the fusion engine activates and performs fusion according to the pre-defined method. For example, a weighted voting mechanism is used to integrate the outputs of multiple models. In a financial risk control task, a Bayesian fusion is performed on the fraud detection results of an NLP model (with a confidence level of 0.92) and a graph neural network model (with a confidence level of 0.88), ultimately generating the final state inference result for the TS-202309 task, which carries a collaborative decision identifier. If a missing model identifier is detected during this process (for example, if KPConv_M005 does not return a result due to a node failure), the system immediately triggers model redistribution and result recalculation across resource groups, ensuring a complete closed-loop decision chain.

[0044] Through the above technical solution, the expert model sub-identifier and the reasoning task sub-identifier are used to build a data lineage tracing system. The sub-identifier is used to strictly bind the generation relationship between the result data and the original task / model, ensuring that the fusion process is not interfered with by other concurrent tasks, and reducing the risk of data cross-contamination caused by asynchronous delays in the traditional timestamp matching mechanism; secondly, the precise verification of the dynamic execution list is based on the real-time updated model execution list, recording the set of target expert models actually participating in a certain reasoning task, and screening valid reasoning data that meet the preset requirements. This closed-loop verification mechanism can maintain the legitimacy verification of the result source when the model instance undergoes identity drift due to fault reconstruction, ensuring the integrity constraints of multi-model collaborative conclusions; by decoupling the strong binding relationship between the result identifier and the physical instance, when the task is migrated to the same group of backup models, the newly generated reasoning result can still be seamlessly integrated into the fusion process because it inherits the original task sub-identifier, and the integrity of the collaborative conclusion is still guaranteed in the case of distributed node failure or elastic scaling scenarios.

[0045] Through the technical solutions of the embodiments of this specification, multi-dimensional analysis of task feature information enables modal features, complexity scores and real-time constraints to jointly form the capability requirement vector of the task, avoiding the homogenized allocation logic that relies on traditional rules, ensuring that high-complexity tasks are matched with precise models, and high-real-time tasks are bound with low-latency resources, effectively reducing resource mismatch and idleness of professional models; secondly, the predictive load regulation mechanism upgrades static allocation to a continuous elastic adaptation process, by predicting the model load before the model instance is actually loaded, actively identifying potential overloaded nodes and adjusting tasks, avoiding the chain reaction of systemic congestion, and solving the long-tail delay caused by resource competition in concurrent scenarios; through the dual anchoring of task sub-identifiers and model sub-identifiers, it ensures that the spatiotemporal deviation of discrete outputs does not affect the integrity and accuracy of the final collaborative reasoning results.

[0046] The embodiments of this specification also provide a multi-model collaborative architecture. This architecture first defines a modular multi-expert model fusion framework and collaborative interface protocol, including a model access layer, an intermediate adaptation layer, and a collaborative reasoning layer. The model access layer provides a unified registration interface for different types of expert models, such as natural language, multimodal, time series, and mechanism models, supporting dynamic model updates and deregistration. Each expert model must adhere to the framework's predefined interface specifications, including input and output formats and model description information (such as supported data modality types and reasoning task types).

[0047] The registration interface for the model access layer is designed as follows: a unified registration interface based on the RESTful style is defined, using JSON format for data exchange. Model developers use this interface to submit a registration request containing basic model information, including the model name, unique identifier, supported data modality types (e.g., natural language processing models must specify supported languages ​​and text length ranges; image models must specify supported image formats and resolution ranges), inference task types (e.g., classification, regression, generation), input and output formats (detailed description of the input data's field names, data types, and dimensions, and the output data's structure and meaning), and version number. When an expert model is updated, the registration interface is called again, passing in the updated model description and new version number. The system verifies the validity of the updated content, including compatibility of the input and output formats. If the verification passes, the relevant information in the model registry is updated, and a model update notification is sent to the middle adaptation layer and collaborative inference layer via a message queue. To deregister a model, developers call the deregistration interface and pass in the model's unique identifier. The system then deletes the model information from the model registry, stops all task scheduling associated with the model, and cleans up any associated cached resources.

[0048] To address the heterogeneity of model outputs, a standardized mapping rule library is constructed. Taking natural language processing and image recognition models as examples, the intermediate adaptation layer maps the model outputs into a unified JSON structure based on their original formats. For example, {"task_id": "xxx", "model_name": "xxx", "result_type": "classification", "results": [ {"label": "xxx", "probability": "xxx"} ]}} This mapping process is implemented by writing extensible adapter plugins, each corresponding to a specific model output format conversion. Developers can add or modify plugins as needed. When the model output format cannot be directly adapted, the intermediate adaptation layer uses a combination of data conversion and semantic mapping to resolve the issue. First, data conversion tools are used to convert raw data into a common data format, such as converting a specific image encoding format into the common JPEG or PNG formats. Then, semantic mapping rules are used to match the semantic information in the converted data with the corresponding fields in the standardized format. For semantic information that cannot be directly mapped, the system logs it and triggers a manual intervention process, where administrators develop temporary mapping rules based on the specific situation to ensure data integrity and accuracy.

[0049] The collaborative inference layer maintains a task queue to store pending inference tasks. When a new task is received, it is removed from the task queue and assigned to the corresponding expert model based on the decision made by the dynamic routing and scheduling engine. To ensure fair and efficient task allocation, a priority-based task scheduling algorithm is employed. Tasks with high real-time requirements are given higher priority, and are preferentially assigned to models with high processing speed and expertise in these tasks. Furthermore, the collaborative inference layer monitors the task processing progress of each model in real time. If a model experiences a task backlog, it automatically reallocates some tasks to other models with lower loads.

[0050] A binary data communication format based on Protocol Buffers is defined. This format offers efficient serialization and deserialization, reducing bandwidth usage and processing time during data transmission. Communication rules stipulate that data transmission must include fields such as task identifier, source model identifier, target model identifier, data type, data content, and validation information. During data transmission, encrypted transmission protocols (such as TLS) are used to ensure data security. A timeout retransmission mechanism is also implemented. If data does not reach the target model within the specified time, it is resent to ensure reliable data transmission. Furthermore, the collaborative reasoning layer utilizes asynchronous communication, implementing non-blocking task processing through an event-driven model, thereby improving the system's concurrent processing capabilities. In its implementation, asynchronous message queues (such as RabbitMQ) are used to transmit tasks and data. When the model completes an inference task, it sends the results to the message queue, from which the collaborative reasoning layer retrieves and integrates the results.

[0051] Next, a dynamic routing and scheduling engine module for inference tasks is constructed. This module automatically classifies and routes inference task features. It consists of three submodules: task feature extraction, dynamic routing decision making, and routing distribution. The task feature extraction module uses a convolutional neural network feature extraction algorithm to analyze the input model inference task and extract key task features, including data modality type (text, image, time series, etc.), task complexity, and real-time requirements. The dynamic routing decision module, based on task features and model information stored in the model registry, employs dynamic programming and reinforcement learning algorithms to select the most appropriate expert model or model combination for each task. This module considers factors such as the model's load, preferred modality, and inference accuracy to ensure optimal task allocation. The routing distribution module accurately distributes tasks to the appropriate expert model based on the decision results of the dynamic routing decision module. After task processing is complete, the inference results are collected and returned to the collaborative inference layer. During task distribution, a unique task identifier is generated for each task and sent to the model along with the task identifier, input data, and relevant task parameters. Once the model completes the inference task, it packages the inference results and task identifier and returns them to the routing distribution module via a message queue. The routing distribution module matches the results with the original tasks based on the task identifier and stores the collected results in the result cache, waiting for subsequent processing by the collaborative reasoning layer. During the task distribution and result collection process, if there are problems such as network failures and model service anomalies, the routing distribution module will record the error information and adopt different processing strategies according to the error type. For recoverable errors, such as short network interruptions, an exponential backoff algorithm is used for retrying, that is, the interval between each retry increases exponentially to avoid frequent retries causing network congestion. If multiple retries fail, the task will be marked as failed and the error information will be returned to the collaborative reasoning layer, which will decide whether to reallocate the task or notify the user of the task processing failure.

[0052] In addition, a sparse activation mechanism and cold start acceleration module are constructed. A sparse activation mechanism is defined. During the model loading process, based on the characteristics of the current inference task, only the expert model related to the task is activated, reducing unnecessary computation and resource usage. Secondly, during the model cold start phase, preloading and caching techniques are used to preload the basic parameters of commonly used models and cache them in computer memory. When the model is needed, the parameters can be quickly retrieved from the cache, reducing model initialization time. Finally, an incremental loading method is used to load only the incremental model parameters required to process the current inference task, further improving model loading and speed.

[0053] During system startup, a list of frequently used models is determined by analyzing historical task data and model usage frequency. For these frequently used models, their basic parameters are pre-loaded into computer memory and managed using an LRU (least recently used) cache strategy. When a model parameter is accessed, it is moved to the head of the cache queue; when cache space is insufficient, the model parameters at the tail of the queue are eliminated. To further improve cache efficiency, model parameters are compressed and stored, then decompressed when loaded into memory. Furthermore, a cache invalidation mechanism is implemented to automatically clear the cache for the corresponding model when a model is updated, ensuring the latest model parameters are used. When the model processes inference tasks, in addition to the pre-loaded basic parameters, only the incremental model parameters required for the current inference task are loaded based on the specific requirements of the task. Incremental parameters are determined by analyzing task characteristics and model structure. For example, for image classification tasks, if the input image resolution and number of categories are known, the parameters of the specific convolutional and fully connected layers that need to be loaded can be calculated based on the model design as incremental parameters. Incremental parameter loading is asynchronous: when the model begins processing the task, the required incremental parameters are asynchronously loaded in the background, avoiding delays in task processing due to parameter loading. At the same time, the loading process of incremental parameters is monitored. If any abnormality occurs during the loading process, appropriate remedial measures are taken in a timely manner, such as reloading or using backup parameters.

[0054] By defining a heterogeneous model fusion framework and collaborative interface protocol, pluggable integration of different types of expert models, such as natural language, multimodal, time series, and mechanism models, is achieved. Users can easily add or replace models based on actual needs without making large-scale changes to the system architecture, greatly improving the system's scalability and adaptability. The dynamic routing scheduling engine can automatically classify and route inference tasks based on their real-time characteristics, achieving reasonable task allocation and avoiding the limitations of static routing strategies. By considering factors such as the model's load and areas of expertise, it ensures that each task is assigned to the most appropriate model or model combination, improving the system's overall inference efficiency and reducing resource waste and inference latency. The sparse activation mechanism and cold start acceleration module effectively reduce the time and resource consumption required during model loading. Especially during the cold start phase, the model initialization time is significantly shortened, improving the system's response speed and better meeting the needs of application scenarios with high real-time requirements, such as equipment failure prediction.

[0055] The embodiment of this specification also provides a collaborative reasoning device based on a multi-expert model, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0056] The embodiments of this specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0057] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0058] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A collaborative reasoning method based on a multi-expert model, characterized in that: The method comprises: Receive multiple reasoning task requests to obtain task feature information of each reasoning task, wherein the task feature information includes data modality type features, complexity feature scores, and real-time feature scores; Performing task allocation based on task feature information of each of the reasoning tasks and pre-acquired model registration information to determine at least one allocation expert model corresponding to each of the reasoning tasks; Predicting the model load of each of the allocation expert models, obtaining a predicted load rate corresponding to each of the allocation expert models, and adjusting at least one allocation expert model corresponding to each of the inference tasks according to the predicted load rate to determine at least one corresponding target expert model; Each of the reasoning tasks is sent to the corresponding target expert model, and the single-model reasoning data of each target expert model is collected through an asynchronous message queue to determine the collaborative reasoning result corresponding to each of the reasoning tasks through the single-model reasoning data of each target expert model.

2. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that: Performing task allocation based on the task feature information of each reasoning task and the pre-acquired model registration information to determine at least one allocation expert model corresponding to each reasoning task, specifically comprising: Based on the task feature information of each reasoning task, determining the task type corresponding to each reasoning task, performing type matching in a pre-registered model library according to the model registration information, and determining the model resource group corresponding to each reasoning task; By utilizing a preset dynamic programming algorithm, at least one allocation expert model corresponding to each of the reasoning tasks is determined in the model resource group based on the task feature information of each of the reasoning tasks and the model registration information.

3. The collaborative reasoning method based on a multi-expert model according to claim 2, characterized in that: Using a preset dynamic programming algorithm, based on the task feature information of each reasoning task and the model registration information, determining at least one allocation expert model corresponding to each reasoning task in the model resource group, specifically including: Calculate at least one planning decision cost data corresponding to each of the inference tasks based on the task feature information and the model registration information, wherein the planning decision cost data is a cost parameter and an accuracy benefit parameter corresponding to each of the inference tasks after allocating each of the model resources in the model resource group, and the cost parameters include a task processing time cost and a model load increment cost; Determine the optimization target of the dynamic programming algorithm to determine the initial task allocation strategy corresponding to each reasoning task in the model resource group based on the optimization target and at least one planning decision cost data corresponding to each reasoning task, wherein the optimization target is the task completion time and the model reasoning accuracy.

4. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that: Predicting the model load of each of the allocation expert models to obtain the predicted load rate corresponding to each of the allocation expert models specifically includes: Based on at least one allocation expert model corresponding to each of the reasoning tasks, counting a set of allocation reasoning tasks corresponding to each of the allocation expert models; Determining a task priority corresponding to each assigned reasoning task in the assigned reasoning task set, and determining a task execution strategy corresponding to the assigned expert model according to the task priority, wherein the task execution strategy includes any one or more of parallel execution and non-parallel execution; The model load prediction is performed through the task execution strategy corresponding to the allocation expert model to determine the predicted load rate corresponding to the execution of the allocation reasoning task set by each allocation expert model.

5. The collaborative reasoning method based on a multi-expert model according to claim 4 is characterized in that: Performing model load prediction based on the task execution strategy corresponding to the assigned expert model to determine the predicted load rate corresponding to each assigned expert model executing the assigned reasoning task set specifically includes: When the task execution strategy is parallel execution, obtaining a complexity feature of each of the distributed reasoning tasks in the distributed reasoning task set, wherein the complexity feature includes input data parameters and the number of operation steps; Performing task-level load prediction based on the complexity characteristics of each of the assigned reasoning tasks and pre-acquired single-task memory usage benchmark data of the assigned expert model to determine a load contribution index corresponding to each of the assigned reasoning tasks; Model-level load aggregation is performed through the load contribution index corresponding to each of the allocated reasoning tasks to determine the predicted load rate corresponding to the allocated reasoning task set executed by the allocated expert model.

6. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that: Adjusting at least one allocation expert model corresponding to each of the inference tasks according to the predicted load rate to determine at least one corresponding target expert model specifically includes: According to the predicted load rate corresponding to each of the allocation expert models, a designated allocation expert model having a load rate greater than a preset threshold is screened to determine at least one designated reasoning task corresponding to the designated allocation expert model; determining a target adjustment reasoning task in the at least one designated reasoning task according to a task priority of the reasoning task obtained in advance; Determine the model resource group corresponding to the designated allocation expert model, perform reallocation within the model resource group, determine the adjustment expert model corresponding to the target adjustment reasoning task, and determine at least one target expert model corresponding to each reasoning task.

7. The collaborative reasoning method based on a multi-expert model according to claim 1, characterized in that: Determining the collaborative reasoning result corresponding to each of the reasoning tasks through the single-model reasoning data of each of the target expert models specifically includes: Acquire inference result data and an inference result identifier in the single model inference data, wherein the inference result identifier includes an expert model sub-identifier and an inference task sub-identifier; The reasoning result data are integrated according to at least one target expert model corresponding to each reasoning task and the reasoning result identifier to determine the collaborative reasoning result corresponding to each reasoning task.

8. The collaborative reasoning method based on a multi-expert model according to claim 7 is characterized in that: Integrating the reasoning result data according to at least one target expert model corresponding to each reasoning task and the reasoning result identifier to determine the collaborative reasoning result corresponding to each reasoning task specifically includes: Filtering a plurality of inference result data whose inference task sub-identifier matches the inference task to obtain an expert model sub-identifier corresponding to the inference result data; Determining a model execution list corresponding to each reasoning task according to at least one target expert model corresponding to each reasoning task; When the model execution list and the expert model sub-identifier corresponding to the reasoning result data meet preset requirements, the reasoning result data is integrated to determine the collaborative reasoning result corresponding to each of the reasoning tasks.

9. A collaborative reasoning device based on a multi-expert model, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images