Routing method and device for large model reasoning service and computer equipment
Through the routing methods and devices of large-model inference services, intelligently judge and distribute user questions to the most suitable inference services, solving the problem of users frequently switching inference services, achieving a simpler user experience and cost reduction.
Patent Information
- Application Number
- CN202411188572.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-20
AI Technical Summary
In the prior art, users need to frequently switch different inference services to deal with problems in different fields, resulting in cumbersome use process.
Provide a routing method and device for large-model inference services. By receiving question information, obtaining inference service list and routing policy plug-in list, using routing policy plug-in to filter the inference service list, obtaining target inference services, and using target inference services to pair Ask questions for inference and realize intelligent routing.
It realizes intelligent judgment of which inference service should be handed over to the backend for processing of each question, giving full play to the characteristics and capabilities of each large model, reducing the cost of use, and improving the experience of use.
Smart Images
Figure CN119166772A_ABST
Abstract
Description
Technical Field
[0001] The present specification relates to the field of computer technology, and in particular to a routing method, apparatus, and computer equipment for large model reasoning services. Background Art
[0002] With the continuous advancement of artificial intelligence technology, big models have developed rapidly, and a large number of big models have emerged in the market, which is called the "hundred-model war" by the industry. These big models not only cover multiple modalities, such as vision, voice, text, etc., but also develop specific vertical industry big models for different industry fields, such as finance, medical care, education, law, etc. Each model has its own unique advantages and capabilities, so that it can meet the needs of different segmented application scenarios and provide more accurate and efficient services.
[0003] The related technology configures a corresponding reasoning service for each field. When users need to ask questions in different fields, they need to frequently switch between different reasoning services, which is cumbersome. Summary of the invention
[0004] To overcome the problems existing in the related art, this specification provides a routing method, apparatus and computer equipment for large model reasoning services.
[0005] According to a first aspect of an embodiment of the present specification, a routing method for a large model reasoning service is provided, the method comprising: receiving question information; obtaining an inference service list and a routing policy plug-in list, the inference service list comprising at least one inference service, the routing policy plug-in list comprising at least one routing policy plug-in; using each routing policy plug-in in the routing policy list, filtering the inference service list to obtain a target reasoning service; using the target reasoning service to reason on the question information to obtain an output result.
[0006] According to the second aspect of the embodiments of this specification, a routing device for a large model reasoning service is provided, including: a receiving module for receiving question information; an acquisition module for acquiring an inference service list and a routing policy plug-in list, the inference service list including at least one inference service, and the routing policy plug-in list including at least one routing policy plug-in; a screening module for using each routing policy plug-in in the routing policy list to screen the inference service list to obtain a target reasoning service; and a reasoning module for using the target reasoning service to reason on the question information to obtain an output result.
[0007] According to a third aspect of an embodiment of this specification, a computer device is provided, including:
[0008] processor;
[0009] a memory for storing processor-executable instructions;
[0010] The processor is configured to: receive question information; obtain an inference service list and a routing policy plug-in list, wherein the inference service list includes at least one inference service, and the routing policy plug-in list includes at least one routing policy plug-in; use each routing policy plug-in in the routing policy list to filter the inference service list to obtain a target inference service; use the target inference service to infer the question information to obtain an output result.
[0011] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:
[0012] In the embodiments of this specification, intelligent routing of large model reasoning services is implemented. By intelligently judging which backend reasoning service should be handed over to each question, the characteristics and capabilities of each large model are fully utilized, effectively reducing the cost of use and improving the user experience.
[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.
[0015] Figure 1 It is a flowchart of a routing method for a large model reasoning service according to an exemplary embodiment of this specification.
[0016] Figure 2 It is a schematic diagram of an inference service registration center shown in this specification according to an exemplary embodiment.
[0017] Figure 3 It is a schematic diagram of a routing policy configuration center according to an exemplary embodiment of this specification.
[0018] Figure 4 It is a schematic diagram of a routing policy plug-in according to an exemplary embodiment of this specification.
[0019] Figure 5 The diagram is a schematic diagram of the priority and enabling status of a routing policy plug-in according to an exemplary embodiment of the present specification.
[0020] Figure 6 It is a schematic diagram of a routing method for a large model reasoning service according to an exemplary embodiment of the present specification.
[0021] Figure 7It is a hardware structure diagram of the computer device where the routing device of the large model reasoning service of the embodiment of this specification is located.
[0022] Figure 8 It is a block diagram of a routing device for a large model reasoning service according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION
[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.
[0024] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0025] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0026] Next, the embodiments of this specification are described in detail.
[0027] The routing method of the large model reasoning service provided by the embodiment of this specification is described in detail below. Figure 1 As shown, Figure 1 This is a flowchart of a routing method for a large model reasoning service according to an exemplary embodiment of this specification. The routing method for a large model reasoning service provided by the embodiment of this specification may include the following steps.
[0028] In step 110, question information is received.
[0029] According to an embodiment of the present specification, the question information may include, for example, a question asked by a user.
[0030] In step 120, an inference service list and a routing policy plug-in list are obtained.
[0031] According to an embodiment of the present specification, the inference service list may include at least one inference service, and the routing strategy plug-in list may include at least one routing strategy plug-in. The inference service may utilize the inference capability of the large model to infer based on the input data and output a prediction or decision result. The inference service may include, for example, a natural language processing service, a recommendation system service, a speech recognition service, etc. The routing strategy plug-in may be used to filter out available inference services from the inference service list.
[0032] In step 130, each routing policy plug-in in the routing policy list is used to filter the inference service list to obtain the target inference service.
[0033] According to an embodiment of the present specification, a routing policy plug-in may include a filtering rule, and each routing policy plug-in may filter the inference service list based on the corresponding filtering rule.
[0034] In step 140, the target reasoning service is used to reason about the question information to obtain an output result.
[0035] According to the routing method of the large model reasoning service of the embodiment of this specification, the intelligent routing of the large model reasoning service is realized, and each question is intelligently judged to which reasoning service of the backend should be handed over to process, so as to give full play to the characteristics and capabilities of each large model. Therefore, questions can be received at a unified question-and-answer interaction entrance, and the questions can be distributed to the most suitable reasoning service of the backend to answer, so as to give full play to the characteristics and capabilities of each large model, and there is no need to frequently switch reasoning services, which is more convenient. In addition, it can effectively reduce the cost of use and improve the user experience.
[0036] Optionally, reasoning service registration information may be received, and the reasoning service registration information may include at least one of a capability tag, a permission tag, a speed tag, and a price tag. Then, the reasoning service may be registered according to the reasoning service registration information. Among them, the capability tag may be used to indicate the capabilities of the reasoning service, such as knowledge question and answer, creative writing, document summarization, logical reasoning, NL2SQL (natural language conversion to SQL statements), code generation, etc. The permission tag may be used to indicate a role that has permission to use the reasoning service. For example, if the permission tag contains Role_1, it means that the user role Role_1 has permission to use it. The speed tag is used to indicate the speed of the reasoning service. The price tag is used to indicate the price of the reasoning service. It is understandable that, in addition to the tag, the reasoning service registration information may also include other information of the reasoning service, such as the ID (identification code) and name of the reasoning service.
[0037] Optionally, a routing policy plug-in registration information may be received, the routing policy plug-in registration information including the ID, name, type, processing logic, etc. of the routing policy plug-in. Then, the routing policy plug-in may be registered according to the routing policy plug-in registration information. By defining routing policies of different dimensions through plug-ins and registration methods, flexible expansion of routing policies may be dynamically achieved.
[0038] Optionally, you can also set the priority and enabled state of each routing policy plug-in. Routing policy plug-ins with high priority are applied first. The enabled state can include, for example, enabled and disabled. Only routing policy plug-ins in the enabled state are applied.
[0039] Optionally, when the routing policy list includes multiple routing policy plug-ins, the routing policy plug-ins in the routing policy list can be combined according to the serial relationship between the routing policy plug-ins. Then, the inference service list is filtered based on the combined routing policy plug-ins to obtain the target inference service that meets the requirements.
[0040] For example, the routing policy list may include routing policy plug-ins p1, p2, and p3. The serial relationship may be p1-p2-p3, that is, the output of p1 is used as the input of p2, and the output of p2 is used as the input of p3. Based on this, the inference service list may be input into p1, and p1 filters the inference service list to obtain the inference service list filtered by p1. Then, the inference service list filtered by p1 may be input into p2, and p2 filters the input inference service list to obtain the inference service list filtered by p2. Next, the inference service list filtered by p2 may be input into p3, and p3 filters the input inference service list to obtain the target inference service.
[0041] Optionally, when the reasoning service is configured with a capability tag, the routing policy plug-in list may include a first routing policy plug-in. The input of the first routing policy plug-in may include question information and a reasoning service list, hereinafter referred to as the first reasoning service list, and the output of the first routing policy plug-in may include a filtered reasoning service list, hereinafter referred to as the second reasoning service list.
[0042] The first routing strategy plug-in can be configured to: receive a first inference service list and question information; determine type information based on the question information. Then, filter out inference services whose capability labels match the type information from the inference service list. Next, output a second inference service list, which includes inference services whose capability labels match the type information. Optionally, for example, a BERT (Bidirectional Encoder Based on Transformer) model can be used to classify the question information and obtain type information.
[0043] Optionally, when the reasoning service is configured with a permission tag, the routing policy plug-in list may include a second routing policy plug-in. The input of the second routing policy plug-in may include a user role and a reasoning service list, hereinafter referred to as a third reasoning service list, and the output of the second routing policy plug-in may include a filtered reasoning service list, hereinafter referred to as a fourth reasoning service list.
[0044] The second routing strategy plug-in can be configured to: receive a third inference service list and a user role; obtain the user role; filter out inference services whose permission tags match the user role from the inference service list; and output a fourth inference service list, the fourth inference service list including inference services whose permission tags match the user role.
[0045] Optionally, when the reasoning service is configured with a price tag, the routing strategy plug-in list may include a third routing strategy plug-in. The input of the third routing strategy plug-in may include a user role and a reasoning service list, hereinafter referred to as a fifth reasoning service list, and the output of the third routing strategy plug-in may include a filtered reasoning service list, hereinafter referred to as a fifth reasoning service list.
[0046] The third routing strategy plug-in can be configured to: receive the fifth inference service list; filter out the inference services whose price tags match the price requirements and whose cost performance meets the preset requirements from the inference service list; and output the sixth inference service list, which includes the inference services whose price tags match the price requirements. The price requirements can include, for example, a price less than a price threshold or a cost performance higher than a cost performance threshold, wherein the price threshold and the cost performance threshold can be set according to actual needs.
[0047] Optionally, when the inference service is configured with a speed label, the routing policy plug-in list may include a fourth routing policy plug-in. The input of the fourth routing policy plug-in may include an inference service list, hereinafter referred to as a seventh inference service list, and the output of the fourth routing policy plug-in may include a filtered inference service list, hereinafter referred to as an eighth inference service list.
[0048] The fourth routing strategy plug-in can be configured to: receive the seventh inference service list; filter out the inference service whose speed label matches the speed requirement from the inference service list; and output the eighth inference service list, wherein the eighth inference service list includes the inference service whose speed label matches the speed requirement. The speed requirement may include, for example, a speed higher than a speed threshold, wherein the speed threshold may be set according to actual needs.
[0049] Optionally, in addition to the first routing policy plug-in, the second routing policy plug-in, the third routing policy plug-in and the fourth routing policy plug-in, the routing policy plug-in list may also include a default routing policy plug-in, which is used to select one of the multiple inference services as a target inference service when there are multiple inference services left after screening by other routing policy plug-ins.
[0050] The default routing strategy plug-in may be configured to: receive a ninth inference service list, the ninth inference service list including multiple inference services obtained after being screened by other routing strategy plug-ins. Then, according to a preset strategy, select an inference service from the seventh inference service list as a target inference service. Then, output the target inference service. The preset strategy may include, for example, a round-robin strategy.
[0051] Exemplarily, in this embodiment, the system architecture of the routing method and device of the large model reasoning service provided in the embodiment of this specification can be applied to the intelligent routing module. The system architecture of the intelligent routing module can include, for example, an inference service registration center, a routing policy configuration center, a routing policy plug-in, and a routing policy plug-in registration center.
[0052] Figure 2 It is a schematic diagram of an inference service registration center shown in this specification according to an exemplary embodiment.
[0053] like Figure 2 As shown in the figure, the reasoning service registration center is one of the core components of the intelligent routing module, and its responsibilities include:
[0054] 1. Manage and maintain a list of candidate inference services;
[0055] 2. Define the capability label of each inference service;
[0056] 3. Monitor the running status of each inference service in real time;
[0057] 4. Support new reasoning services. New reasoning services are registered with the reasoning service center through active reporting.
[0058] Exemplarily, in this embodiment, the following reasoning services may be registered: reasoning service_1, reasoning service_2, ..., reasoning service_n, where n is a positive integer.
[0059] Among them, the reasoning service_1 may include, for example, the following tag list:
[0060] 1. Type: Vincent's text.
[0061] 2. Abilities: Knowledge quiz, creative writing, document summary.
[0062] 3. Permissions: Role_1, Role_2.
[0063] 4. Price: 0.01 yuan / 1,000 Tokens.
[0064] 5. Speed: 30 Token / s.
[0065] Inference service_2 may include, for example, the following tag list:
[0066] 1. Type: Vincent's text.
[0067] 2. Abilities: logical reasoning, NL2SQL, code generation.
[0068] 3. Permissions: Role_2.
[0069] 4. Price: 0.05 yuan / 1,000 Tokens.
[0070] 5. Speed: 25 Token / s.
[0071] Inference service_n may include, for example, the following tag list:
[0072] 1. Type: Wensheng picture.
[0073] 2. Pixels: 512*512.
[0074] 3. Style: cartoon, two-dimensional.
[0075] 4. Permissions: Role_1, Role_3.
[0076] 5. Price: 0.05 yuan / 1,000 Tokens.
[0077] Figure 3 It is a schematic diagram of a routing policy configuration center according to an exemplary embodiment of this specification.
[0078] like Figure 3 As shown in the figure, the routing policy configuration center is one of the core components of the intelligent routing module. Its responsibilities include:
[0079] 1. Define the overall framework of routing policy management, and support the definition, addition, deletion, and arrangement of multiple routing policies through plug-ins;
[0080] 2. Define the interface format of the plug-in;
[0081] 3. Provide a routing strategy plug-in registration center to manage all plug-ins;
[0082] 4. Support configuration of plugin priority and enable / disable;
[0083] 5. Provide a default routing policy plug-in to handle the bottom line.
[0084] Figure 4It is a schematic diagram of a routing policy plug-in according to an exemplary embodiment of this specification.
[0085] like Figure 4 As shown in the figure, each routing policy plug-in consists of the following three parts:
[0086] 1. Plug-in input: User questions and a list of candidate inference services are used as input parameters of the plug-in;
[0087] 2. Plug-in business logic body: Filter available reasoning services from the list of candidate reasoning services based on user questions and screening rules;
[0088] 3. Plug-in output: The output parameter is the filtered list of available inference services.
[0089] Plugins can be connected serially, that is, the available service list output by the previous plugin can be used as the alternative service list of the next plugin.
[0090] The responsibilities of the routing policy plugin registry can include:
[0091] 1. Manage and maintain the routing strategy plug-in list;
[0092] 2. Provide a default routing policy plug-in to handle the bottom line;
[0093] 3. New routing policy plug-ins can be registered through active reporting.
[0094] Exemplarily, in this embodiment, the routing policy plug-in may include routing policy_1, routing policy_2, routing policy_3, and routing policy_default.
[0095] Figure 5 The diagram is a schematic diagram of the priority and enabling status of a routing policy plug-in according to an exemplary embodiment of the present specification.
[0096] like Figure 5 As shown, it supports configuring the priority and enabling / disabling of plug-ins. Through the combination of multiple plug-ins, various advanced and flexible routing strategies can be implemented.
[0097] Figure 6 FIG. 1 is a schematic diagram of a routing method for a large model reasoning service according to an exemplary embodiment of the present specification. Figure 6 As shown, the routing method for the large model reasoning service provided in the embodiment of this specification may include the following steps.
[0098] Step 1: The user asks questions in natural language on the AIGC (Artificial Intelligence Generated Content) question-and-answer interface, and the question information is submitted to the backend intelligent routing module.
[0099] Step 2: The intelligent routing module obtains a list of all available candidate reasoning services from the reasoning service registration center. The candidate reasoning service list may include the ID, name, status, label list, etc. of each reasoning service. The label list may include labels such as type, capability, authority, price, speed, etc.
[0100] Step 3: The intelligent routing module obtains a routing policy list from the routing policy configuration center. Exemplarily, in this embodiment, the routing policy list may include routing policy_1, routing policy_2, routing policy_3, and routing policy_default.
[0101] Step 4: The intelligent routing module executes the corresponding routing strategies one by one in the order of routing strategy_3——routing strategy_1——routing strategy_2——routing strategy_default, filters and selects the list of candidate inference services based on the routing strategies, and outputs the list of inference services that meet the requirements.
[0102] Step 5: If the remaining candidate service list contains two or more inference services after filtering through the previous series of routing policies, the default routing policy is finally executed, and a inference service is selected as the final preferred inference service, i.e., the target inference service, according to the round-robin policy.
[0103] Step 6: Submit the user's question information to the preferred reasoning service for answering. If the preferred service list is empty, directly output the default fallback response, such as "Sorry, I can't answer this question, let's change the question."
[0104] Step 7: Send the output of the large model back to the user.
[0105] According to the embodiments of this specification, user questions can be received at a unified question-and-answer interaction portal, and the questions can be distributed to the most suitable reasoning service at the back end for answering. Based on the user's question, a comprehensive routing decision is made in combination with various factors such as the type, function, performance, authority, and price of the back-end reasoning service, and each question should be intelligently determined to which back-end reasoning service to handle, giving full play to the characteristics and capabilities of each large model, without the need to frequently switch reasoning services, which is more convenient. In addition, it can effectively reduce the cost of use and improve the user experience.
[0106] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a routing device for a large model reasoning service and a terminal to which it is applied.
[0107] The embodiment of the routing device of the large model reasoning service of this specification can be applied to a computer device, such as a server or a terminal device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 7 As shown, it is a hardware structure diagram of the computer device where the routing device of the large model reasoning service of the embodiment of this specification is located, except Figure 7 In addition to the processor 710, memory 730, network interface 720, and non-volatile memory 740 shown, the server or electronic device where the device 731 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.
[0108] like Figure 8 As shown, Figure 8 1 is a block diagram of a routing device for a large model reasoning service according to an exemplary embodiment of the present specification, the device comprising:
[0109] Receiving module 810, used for receiving question information;
[0110] An acquisition module 820 is used to acquire an inference service list and a routing policy plug-in list, wherein the inference service list includes at least one inference service, and the routing policy plug-in list includes at least one routing policy plug-in;
[0111] A screening module 830 is used to screen the inference service list using each routing policy plug-in in the routing policy list to obtain a target inference service;
[0112] The reasoning module 840 is used to use the target reasoning service to reason about the question information and obtain an output result.
[0113] Correspondingly, the present specification also provides a routing device for a large model reasoning service, the device comprising a processor; a memory for storing processor executable instructions; wherein the processor is configured to: receive question information; obtain a reasoning service list and a routing policy plug-in list, the reasoning service list includes at least one reasoning service, and the routing policy plug-in list includes at least one routing policy plug-in; use each routing policy plug-in in the routing policy list to filter the reasoning service list to obtain a target reasoning service; use the target reasoning service to reason on the question information to obtain an output result.
[0114] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.
[0115] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0116] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0117] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.
[0118] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
[0119] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.
Claims
1. A routing method for a large model reasoning service, characterized in that: The method comprises: Receive question information; Obtaining an inference service list and a routing policy plug-in list, wherein the inference service list includes at least one inference service, and the routing policy plug-in list includes at least one routing policy plug-in; Using each routing strategy plug-in in the routing strategy list, screening the inference service list to obtain a target inference service; The target reasoning service is used to reason about the question information to obtain an output result.
2. The method according to claim 1, characterized in that: The filtering of the inference service list according to each routing policy plug-in in the routing policy list to obtain the target inference service includes: Combining the routing policy plug-ins in the routing policy list according to the serial relationship between the routing policy plug-ins; The inference service list is filtered and screened based on the combined routing strategy plug-in to obtain a target inference service that meets the requirements.
3. The method according to claim 2, characterized in that The reasoning service is configured with a capability label; the routing policy plug-in list includes a first routing policy plug-in, and the first routing policy plug-in is configured as: Receiving a first reasoning service list and question information; Determine type information according to the question information; Filtering out the inference services whose capability tags match the type information from the inference service list; A second inference service list is output, where the second inference service list includes inference services whose capability tags match the type information.
4. The method according to claim 2, characterized in that: The inference service is configured with a permission tag; the routing strategy plug-in list includes a second routing strategy plug-in; The second routing strategy plug-in is configured as follows: receiving a third inference service list; Get user roles; Filter out the inference service whose permission label matches the user role from the inference service list; A fourth inference service list is output, wherein the fourth inference service list includes inference services whose permission labels match the user role.
5. The method according to claim 2, characterized in that: The inference service is configured with a price tag; the routing strategy plug-in list includes a third routing strategy plug-in, and the third routing strategy plug-in is configured as: receiving a fifth inference service list; Filtering out the inference services whose price tags match the price requirement from the inference service list; A sixth reasoning service list is output, wherein the sixth reasoning service list includes reasoning services whose price tags match the price requirement.
6. The method according to claim 1, characterized in that The inference service is configured with a speed tag; the routing strategy plug-in list includes a fourth routing strategy plug-in; The fourth routing strategy plug-in is configured as follows: receiving a seventh inference service list; filtering out inference services whose speed tags match the speed requirement from the inference service list; An eighth inference service list is output, wherein the eighth inference service list includes inference services whose speed tags match the speed requirement.
7. The method according to any one of claims 1 to 6, characterized in that The routing policy plug-in list also includes a default routing policy plug-in; The default routing policy plugin is configured as: receiving a ninth inference service list, wherein the ninth inference service list includes a plurality of inference services obtained after being screened by other routing strategy plug-ins; Using a preset strategy, select a reasoning service from the seventh reasoning service list as a target reasoning service; The target inference service is output.
8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: receiving reasoning service registration information, wherein the reasoning service registration information includes at least one of a capability tag, a permission tag, a speed tag, and a price tag; Register the reasoning service according to the reasoning service registration information.
9. A routing device for a large model reasoning service, characterized in that: The device comprises: A receiving module, used for receiving question information; An acquisition module, used to acquire an inference service list and a routing policy plug-in list, wherein the inference service list includes at least one inference service, and the routing policy plug-in list includes at least one routing policy plug-in; A screening module, used to screen the inference service list using each routing strategy plug-in in the routing strategy list to obtain a target inference service; The reasoning module is used to use the target reasoning service to reason about the question information and obtain an output result.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the following method is implemented: receiving question information; obtaining an inference service list and a routing strategy plug-in list, wherein the inference service list includes at least one inference service, and the routing strategy plug-in list includes at least one routing strategy plug-in; Using each routing strategy plug-in in the routing strategy list, screening the inference service list to obtain a target inference service; The target reasoning service is used to reason about the question information to obtain an output result.