Method and device for collaborative reasoning of multiple heterogeneous models

Abstract

The invention discloses a method and device for collaborative reasoning of multiple heterogeneous models, and relates to the technical field of artificial intelligence reasoning prediction.The method comprises the steps that a reasoning request and a cue word submitted by a client side are received; according to the context information of the reasoning request and a preset condition, selecting a plurality of matched heterogeneous models and performing Token prediction at the same time; and selecting an optimal prediction result from the prediction results of all the heterogeneous models, splicing the optimal prediction result and the cue word, if reasoning of the splicing result is not completed, submitting the splicing result as a new cue word to a plurality of matched heterogeneous models, and continuing iterative reasoning until reasoning is completed. According to the method, the overall reasoning performance and efficiency are improved, and the task execution capability of the model in a specific field is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence reasoning and prediction technology, and in particular to a method and device for collaborative reasoning of multiple heterogeneous models. Background Art

[0002] In the prior art, each AI model of architecture (e.g., large language model LLM) usually runs independently, and each model performs reasoning based on its weights and architectural characteristics. However, a single model may not perform well on a specific task, or be inefficient in processing a specific type of query. In addition, a single model may not be able to fully utilize the expertise of other models in a specific field or task. Therefore, AI models of a single architecture have low overall reasoning performance and efficiency when processing reasoning tasks, and are not capable of performing specific tasks. Summary of the invention

[0003] In view of the above defects or deficiencies in the prior art, the present invention provides a method and device for collaborative reasoning of multiple heterogeneous models. Through the collaborative work between models of different architectures, the overall reasoning performance and efficiency are improved, and the ability of the model to perform tasks in specific fields is enhanced.

[0004] One aspect of the present invention provides a method for collaborative reasoning of multiple heterogeneous models, including: receiving an inference request and a prompt word submitted by a client; selecting multiple matching heterogeneous models to perform token prediction simultaneously according to the context information and preset conditions of the inference request; selecting the best prediction result from the prediction results of all heterogeneous models, splicing the best prediction result with the prompt word, and if the splicing result does not complete the reasoning, submitting the splicing result as a new prompt word to the multiple matching heterogeneous models, and continuing the iterative reasoning until the reasoning is completed.

[0005] Furthermore, the preset conditions include: at least one of the expertise or areas of expertise of the model, inference cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, and marking of specific tokens.

[0006] Furthermore, according to the context information and preset conditions of the inference request, multiple matching heterogeneous models are selected to perform Token prediction simultaneously, including: if the label information indicating the model's expertise in the inference request matches the labels declared by multiple heterogeneous models, the inference request is sent to the multiple heterogeneous models, and the multiple heterogeneous models that receive the inference request perform inference prediction simultaneously, and each model generates no more than a predetermined number of Tokens.

[0007] Furthermore, according to the context information and preset conditions of the inference request, multiple matching heterogeneous models are selected to perform Token prediction simultaneously, including: if the inference request does not contain label information indicating the model's expertise, the prompt word is predicted and classified through a language model, and the inference request is sent to heterogeneous models corresponding to multiple declaration labels matching the classification result, and multiple heterogeneous models perform inference prediction simultaneously, and each model generates no more than a predetermined number of Tokens.

[0008] Further, the selecting the best prediction result from the prediction results of all heterogeneous models includes: recording the number of predictions generated by each model. The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token ; Record the first to the second generation of each model In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token ; If one of the following conditions is met, the model stops generating tokens and ends this round of reasoning and prediction: Condition 1: Deviation Reach the set threshold; Condition 2: Deviation Reach the set threshold; Condition 3: Deviation Reach the set threshold; Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; Calculate the average deviation of each heterogeneous model in Condition 1, Condition 2, and Condition 3 respectively, and take the prediction result of the model with the smallest average deviation as the best prediction result.

[0009] On the other hand, the present invention also provides a device for collaborative reasoning of multiple heterogeneous models, including: a receiving module, used to receive reasoning requests and prompt words submitted by a client; a scheduling module, used to select multiple matching heterogeneous models to perform token prediction simultaneously according to the context information and preset conditions of the reasoning request; an inference module, used to select the best prediction result from the prediction results of all heterogeneous models, and splice the best prediction result with the prompt word. If the splicing result does not complete the reasoning, the splicing result is submitted as a new prompt word to the multiple matching heterogeneous models, and the iterative reasoning is continued until the reasoning is completed.

[0010] Furthermore, the preset conditions include: at least one of the expertise or areas of expertise of the model, inference cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, and marking of specific tokens.

[0011] Furthermore, the scheduling module is also used to: if the label information indicating the model's expertise in the inference request matches the labels declared by multiple heterogeneous models, the inference request is sent to the multiple heterogeneous models, and the multiple heterogeneous models that receive the inference request perform inference prediction at the same time, and each model generates no more than the predetermined number of tokens.

[0012] Furthermore, the scheduling module is also used for: if the inference request does not contain label information indicating the model's expertise, the prompt word is predicted and classified through a language model, and the inference request is sent to heterogeneous models corresponding to multiple declaration labels matching the classification results. Multiple heterogeneous models perform inference prediction simultaneously, and each model generates no more than a predetermined number of tokens.

[0013] Furthermore, the inference module is also used to: record each model generates the first The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token ; Record the first to the second generation of each model In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token ; If one of the following conditions is met, the model stops generating tokens and ends this round of reasoning and prediction: Condition 1: Deviation Reach the set threshold; Condition 2: Deviation Reach the set threshold; Condition 3: Deviation Reach the set threshold; Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; Calculate the average deviation of each heterogeneous model in Condition 1, Condition 2, and Condition 3 respectively, and take the prediction result of the model with the smallest average deviation as the best prediction result.

[0014] The present invention provides a method and device for collaborative reasoning of multiple heterogeneous models, which innovatively integrates multiple models of different architectures into an inference system, supports collaborative reasoning of heterogeneous models, gives full play to the advantages of each model, and dynamically selects the most suitable model for token generation according to preset conditions. The present invention improves the overall reasoning performance and efficiency, and enhances the ability of the model to perform tasks in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a flowchart of a method for collaborative reasoning of multiple heterogeneous models provided by an embodiment of the present application; Figure 2 It is a schematic diagram of the structure of a device for collaborative reasoning of multiple heterogeneous models provided by an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0018] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0019] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0020] It should be noted that the directional words such as "upper", "lower", "left", and "right" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is formed "on" or "under" another element, it can not only be formed directly "on" or "under" another element, but also be formed "on" or "under" another element indirectly through an intermediate element.

[0021] The present application provides a method and device for collaborative reasoning of multiple heterogeneous models, which can be implemented based on the following system: Multiple language models with different architectures, each service can perform reasoning independently. These model services can be large open source models such as Qwen, GPT-3, BERT, XLNet, etc., or small self-trained models that focus on specific fields or tasks;

[0022] The scheduler is responsible for deciding which model to use to generate the next token based on preset conditions. The scheduler has real-time analysis capabilities and can dynamically select the most appropriate model service based on the model's feedback information and prediction results, as well as the current reasoning context. The method of the present application can implement the functions of the above scheduler by means of a computer program;

[0023] Communication interface, which allows information exchange between models and between models and scheduler. This interface supports efficient and low-latency data transmission, ensuring that models can quickly respond to scheduler requests;

[0024] The API gateway provides a unified service interface to the outside world, distributes reasoning requests to the scheduler internally, and returns the scheduler's decision results to the user.

[0025] See also Figure 1 An embodiment of the present application provides a method for collaborative reasoning of multiple heterogeneous models, including the following steps: Step S101, receiving an inference request and a prompt word submitted by a client; Specifically, the client submits an inference request and a prompt word through the API gateway. The inference request contains the text to be inferred and possible context information. The API gateway forwards the inference request to the scheduler module. The scheduler module receives the inference request and prompt word submitted by the client, and uses the method of collaborative inference of multiple heterogeneous models in this embodiment to implement the corresponding function. Optionally, the scheduler can be one or more, for example, multiple heterogeneous models with scheduling modules installed, or a separately set scheduler module.

[0026] Step S102: According to the context information of the reasoning request and preset conditions, multiple matching heterogeneous models are selected to perform token prediction simultaneously.

[0027] In natural speech processing, context information in an inference request refers to text information surrounding a specific word or phrase, including the words before (above) and after (below) the target word. This information is essential for understanding the meaning of the target word, eliminating ambiguity, and performing semantic reasoning. Preset conditions are conditions pre-set in the scheduler algorithm, including but not limited to: the model's expertise or areas of expertise, inference cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, tags of specific tokens, etc. The scheduler selects multiple heterogeneous models to perform token predictions simultaneously based on the current inference request context information and preset conditions. Specifically, each inference node (i.e., model service) can declare one or more tags to indicate the type of reasoning that the model is good at, such as news facts, security knowledge, medical diagnosis, mathematical logic, code generation, etc. If the label information indicating the domain in which the model is good at in the inference request matches the labels declared by multiple heterogeneous models, the inference request is sent to the multiple heterogeneous models, and the multiple heterogeneous models that receive the inference request perform inference prediction at the same time, and each model generates no more than a predetermined number of tokens; if the inference request does not contain label information indicating the domain in which the model is good at, the scheduler uses a trained small language model of its own to determine which capability labels are needed according to the prompt word prompt. For example, the scheduler is based on a trained language model for classification, and the language model predicts the classification according to the prompt word prompt. If the prompt word is "Please give a weekly meal recipe for a 70-year-old diabetic patient", it should be predicted as "medical health" classification, and the inference request is sent to the heterogeneous models corresponding to the multiple declared labels that match the classification result. Multiple heterogeneous models perform inference prediction at the same time, and each model generates no more than a predetermined number of tokens. It should be pointed out that the generation of tokens can be stopped before the predetermined number is reached, that is, the generation can be completed in advance. At this time, it means that the scheduler determines through its own algorithm that a complete paragraph has been generated and no longer needs to generate tokens.

[0028] Step S103, selecting the best prediction result from the prediction results of all heterogeneous models, concatenating the best prediction result with the prompt word, and if the concatenated result does not complete the reasoning, submitting the concatenated result as a new prompt word to the multiple matching heterogeneous models, and continuing the iterative reasoning until the reasoning is completed.

[0029] First, record the number of times each model generates The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token For example, the prompt word is "What is the capital of France?", and the model predicts the generated text "The capital of France is Paris". Let's assume that the first token is "France", the second token is "of", the third token is "capital", and the fourth token is "Paris". After the third token is generated, the fourth token may be "Paris", "Beijing", or "London". The probabilities of the various possibilities are not the same. Assuming that the probability of "London" is the highest, the probability value of the token "London" with the highest probability is recorded as , and the fourth token selected by the model is "Paris", so the probability value of "Paris" is recorded as .

[0030] Next, record the first to the second In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token For example, two tokens have been generated, and the probability value of the token selected by the scheduler algorithm is =40% =60%, but in fact the probability value of the maximum probability token is =90%, =90%, then =50% is the first two probabilities , The average value of =90% is the first two probabilities , The average value of . Now we need to generate the third token. =20%, =60%, we can calculate =40%, which is the average of 40%, 60%, and 20%. We can also calculate =80%, which is the average of 90%, 90%, 60%, and so on.

[0031] Next, determine whether one of the following conditions is met. If any of the conditions is met, the model stops generating tokens and ends this round of reasoning and prediction: Condition 1: Deviation Reach the set threshold (experience value is 10%); Condition 2: Deviation Reach the set threshold (experience value is 10%); Condition 3: Deviation Reach the set threshold (experience value is 15%); Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; It should be noted that only the token generation of a model that meets the above conditions is stopped. If other model inference services have not reached the stopping conditions, they will continue to generate tokens until the preset number is reached, or if a model inference service determines through its own algorithm that a complete paragraph has been generated and no token generation is needed anymore, it will also stop generating tokens.

[0032] Next, the average deviation of each heterogeneous model in conditions 1, 2 and 3 is calculated respectively, and the prediction result of the model with the smallest average deviation is taken as the best prediction result.

[0033] For example: there are two heterogeneous models, Model 1 and Model 2. The deviation of Model 1 in Condition 1 is 10%, the deviation of Model 1 in Condition 2 is 5%, and the deviation of Model 1 in Condition 3 is 15%. The average deviation of Model 1 is (10%+5%+15%) / 3=15%; the deviation of Model 2 in Condition 1 is 5%, the deviation of Model 2 in Condition 2 is 5%, and the deviation of Model 2 in Condition 3 is 10%. The average deviation of Model 2 is (5%+5%+10%) / 3=6.67%. The prediction result of Model 2 is the best prediction result.

[0034] Finally, the best prediction result is concatenated with the prompt word prompt. If the concatenated result is marked as completed, the prediction result is returned to the client. If the concatenated result has not completed the reasoning, the concatenated result is submitted as a new prompt word to the multiple matching heterogeneous models, and the iterative reasoning is continued until the reasoning is completed.

[0035] For example, if the prompt word prompt is "What is the capital of France?", and it is assumed that "The capital of France is Paris" will be generated, and the best prediction result of the first step generates "The capital of France", then the prompt word prompt and the best prediction result of the first step are spliced ​​to get "What is the capital of France? The capital of France", and the generation is not completed at this time, so this splicing result is submitted to the scheduler again as a new prompt word prompt, and the reasoning continues from step S101; assuming that the second step generates "It is Paris", the generation is completed at this time, then "It is Paris" and the prompt word "What is the capital of France? The capital of France" are directly spliced ​​together as the final result "What is the capital of France? The capital of France is Paris". Among them, the model reasoning service uses its own algorithm to identify whether a complete paragraph has been generated to determine whether the generation is completed.

[0036] The above-mentioned embodiments of the present application innovatively integrate multiple models of different architectures into one reasoning system, support collaborative reasoning of heterogeneous models, give full play to the advantages of each model, dynamically select the most suitable model for token generation according to preset conditions, improve the overall reasoning performance and efficiency, and enhance the model's ability to perform tasks in specific fields.

[0037] See also Figure 2 Another embodiment of the present invention further provides a device 200 for collaborative reasoning of multiple heterogeneous models, including a receiving module 201, a scheduling module 202 and a reasoning module 203. The device 200 can execute the method for collaborative reasoning of multiple heterogeneous models in the method embodiment.

[0038] Specifically, the device 200 includes: The receiving module 201 is used to receive the reasoning request and prompt word submitted by the client; The scheduling module 202 is used to select multiple matching heterogeneous models to perform token prediction simultaneously according to the context information of the reasoning request and preset conditions; The reasoning module 203 is used to select the best prediction result from the prediction results of all heterogeneous models, and splice the best prediction result with the prompt word. If the splicing result does not complete the reasoning, the splicing result is submitted as a new prompt word to the multiple matching heterogeneous models, and the iterative reasoning is continued until the reasoning is completed.

[0039] Furthermore, the preset conditions include: at least one of the expertise or areas of expertise of the model, inference cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, and marking of specific tokens.

[0040] Furthermore, the scheduling module 202 is also used to: if the label information indicating the model's expertise in the inference request matches the labels declared by multiple heterogeneous models, the inference request is sent to the multiple heterogeneous models, and the multiple heterogeneous models that receive the inference request perform inference prediction at the same time, and each model generates no more than a predetermined number of tokens.

[0041] Furthermore, the scheduling module 202 is also used for: if the inference request does not contain label information indicating the model's expertise, predicting and classifying the prompt word through a language model, and sending the inference request to heterogeneous models corresponding to multiple declaration labels matching the classification results, and multiple heterogeneous models perform inference prediction simultaneously, and each model generates no more than a predetermined number of tokens.

[0042] Furthermore, the inference module 203 is also used to: record each model generates the first The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token ; Record the first to the second generation of each model In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token ; If one of the following conditions is met, the model stops generating tokens and ends this round of reasoning and prediction: Condition 1: Deviation Reach the set threshold; Condition 2: Deviation Reach the set threshold; Condition 3: Deviation Reach the set threshold; Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; Calculate the average deviation of each heterogeneous model in Condition 1, Condition 2, and Condition 3 respectively, and take the prediction result of the model with the smallest average deviation as the best prediction result.

[0043] It should be noted that the device 200 for collaborative reasoning of multiple heterogeneous models provided in this embodiment corresponds to a technical solution that can be used to execute each method embodiment. Its implementation principle and technical effect are similar to the method and will not be repeated here.

[0044] See also Figure 3 Another embodiment of the present invention provides a schematic diagram of the structure of an electronic device 300, which is used to implement the method of collaborative reasoning of multiple heterogeneous models in the method embodiment. The electronic device 300 in the embodiment of the present invention may include but is not limited to a PC and a server. Figure 3 The electronic device 300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0045] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to implement the method of the embodiment of the present invention. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0046] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0047] The above description is only a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A method for collaborative reasoning of multiple heterogeneous models, characterized in that: include: Receive the reasoning request and prompt word submitted by the client; According to the context information and preset conditions of the reasoning request, multiple matching heterogeneous models are selected to perform token prediction simultaneously; The best prediction result is selected from the prediction results of all heterogeneous models, and the best prediction result is spliced ​​with the prompt word. If the spliced ​​result does not complete the reasoning, the spliced ​​result is submitted as a new prompt word to the multiple matching heterogeneous models, and the iterative reasoning is continued until the reasoning is completed.

2. The method for collaborative reasoning of multiple heterogeneous models according to claim 1, characterized in that: The preset conditions include: at least one of the model's expertise or areas of expertise, reasoning cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, and labels of specific tokens.

3. The method for collaborative reasoning of multiple heterogeneous models according to claim 2, characterized in that: According to the context information and preset conditions of the reasoning request, multiple matching heterogeneous models are selected to perform token prediction simultaneously, including: If the label information indicating the model's expertise in the inference request matches the labels declared by multiple heterogeneous models, the inference request will be sent to the multiple heterogeneous models. The multiple heterogeneous models that receive the inference request will perform inference prediction at the same time, and each model will generate no more than a predetermined number of tokens.

4. The method for collaborative reasoning of multiple heterogeneous models according to claim 2, characterized in that: According to the context information and preset conditions of the reasoning request, multiple matching heterogeneous models are selected to perform token prediction simultaneously, including: If the inference request does not contain label information indicating the model's expertise, the prompt word is predicted and classified through the language model, and the inference request is sent to heterogeneous models corresponding to multiple declaration labels that match the classification results. Multiple heterogeneous models perform inference and prediction simultaneously, and each model generates no more than a predetermined number of tokens.

5. A method for collaborative reasoning of multiple heterogeneous models according to claim 3 or 4, characterized in that: The selecting the best prediction result from the prediction results of all heterogeneous models includes: Record each model generation The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token ; Record the first to the second generation of each model In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token ; If one of the following conditions is met, the model stops generating tokens and ends this round of inference prediction: Condition 1: Deviation Reaching a set threshold; Condition 2: Deviation Reaching a set threshold; Condition 3: Deviation Reaching a set threshold; Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; The average deviation of each heterogeneous model in conditions 1, 2 and 3 is calculated respectively, and the prediction result of the model with the smallest average deviation is taken as the best prediction result.

6. A device for collaborative reasoning of multiple heterogeneous models, characterized in that include: A receiving module, used to receive the reasoning request and prompt word submitted by the client; The scheduling module is used to select multiple matching heterogeneous models for simultaneous token prediction based on the context information and preset conditions of the inference request; The reasoning module is used to select the best prediction result from the prediction results of all heterogeneous models, and splice the best prediction result with the prompt word. If the splicing result does not complete the reasoning, the splicing result is submitted as a new prompt word to the multiple matching heterogeneous models, and the iterative reasoning is continued until the reasoning is completed.

7. The device for collaborative reasoning of multiple heterogeneous models according to claim 6, characterized in that: The preset conditions include: at least one of the model's expertise or areas of expertise, reasoning cost and resource consumption, response time and real-time requirements, probability distribution and confidence of predicted tokens, and labels of specific tokens.

8. The device for collaborative reasoning of multiple heterogeneous models according to claim 7, characterized in that: The scheduling module is also used for: If the label information indicating the model's expertise in the inference request matches the labels declared by multiple heterogeneous models, the inference request will be sent to the multiple heterogeneous models. The multiple heterogeneous models that receive the inference request will perform inference prediction at the same time, and each model will generate no more than a predetermined number of tokens.

9. The device for collaborative reasoning of multiple heterogeneous models according to claim 7, characterized in that: The scheduling module is also used for: If the inference request does not contain label information indicating the model's expertise, the prompt word is predicted and classified through the language model, and the inference request is sent to heterogeneous models corresponding to multiple declaration labels that match the classification results. Multiple heterogeneous models perform inference and prediction simultaneously, and each model generates no more than a predetermined number of tokens.

10. The device for collaborative reasoning of multiple heterogeneous models according to claim 8 or 9, characterized in that: The reasoning module is also used to: Record each model generation The probability value of the token with the highest probability in the process of Token And the probability value of the selected Token ; Record the first to the second generation of each model In the process of Token, the average probability of the Token with the largest probability and the average probability of the selected Token ; If one of the following conditions is met, the model stops generating tokens and ends this round of inference prediction: Condition 1: Deviation Reaching a set threshold; Condition 2: Deviation Reaching a set threshold; Condition 3: Deviation Reaching a set threshold; Condition 4: The predetermined number of tokens has been generated; Condition 5: The model has marked this reasoning as finished; The average deviation of each heterogeneous model in conditions 1, 2 and 3 is calculated respectively, and the prediction result of the model with the smallest average deviation is taken as the best prediction result.

Images