Inference result determination method, device, computer equipment, and storage medium

Abstract

The present application relates to a method, apparatus, computer equipment, and storage medium for determining an inference result. The method comprises: receiving request data carried in a target prompt template and an inference request during an inference service update, and splicing the request data to obtain a target request; determining a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request, so as to determine an inference result based on the target request through the target inference instance. The architecture of the inference instance and the prompt template is dynamically adapted to ensure the matching relationship between the inference instance and the prompt template, so as to avoid the situation where the inference effect of the inference model deployed on the inference instance is poor due to incompatibility between the inference instance and the prompt template, thereby ensuring that a good inference effect can be obtained during the inference service update process.

Description

Technical Field

[0001] The present application relates to the field of model reasoning technology, and in particular to a method, apparatus, computer equipment, and storage medium for determining reasoning results. Background Art

[0002] Large-model reasoning is currently the hottest technology, and a large number of products have begun integrating large-model reasoning capabilities to enhance their services. Prompt engineering is a critical component of the large-model reasoning process, directly impacting the final inference results. Because large-model training relies on large amounts of data, the newly trained inference model often performs poorly with the original prompt template, necessitating the generation of a new prompt template to adapt to the new inference model. However, during large-model inference service updates, the old prompt template and the new inference model may coexist, resulting in poor inference results during this period. Therefore, providing a prompt template that can match the inference model during inference service updates, thereby ensuring good inference results, has become a pressing technical issue in this field. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, device, computer equipment, and storage medium for determining inference results that can obtain a prompt template that matches the inference model during the inference service update process to address the above technical problems, thereby ensuring good inference results.

[0004] In a first aspect, the present application provides a method for determining an inference result, which is applied to a first inference instance. During an inference service update, the method includes:

[0005] Receive the target request sent by the Prompt instance; the target request is obtained by splicing the target Prompt template and the request data carried in the inference request by the Prompt instance. The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. The inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier. The quantity ratio is the ratio sent by the registration center received by the gateway. The quantity ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set. The inference instances in the instance set are inference instances in a healthy state.

[0006] According to the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0007] In one embodiment, determining, based on the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service includes:

[0008] Determine the model version identifier corresponding to the first inference instance according to the topology information of the multiple inference instances corresponding to the inference service; the topology information includes the model version identifier and address identifier corresponding to each inference instance;

[0009] Compare the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result;

[0010] According to the comparison result, the target inference instance corresponding to the model version identifier in the target request is determined from the multiple inference instances corresponding to the inference service.

[0011] In one embodiment, determining, based on the comparison result, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service includes:

[0012] If the comparison result is that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance, the first inference instance is used as the target inference instance.

[0013] If the comparison result shows that the model version identifier in the target request is inconsistent with the model version identifier corresponding to the first inference instance, then determine the second inference instance from each inference instance according to the model version identifier corresponding to each inference instance in the topology information, and use the second inference instance as the target inference instance;

[0014] The second inference instance is an inference instance corresponding to the target model version identifier, and the target model version identifier is a model version identifier among the model version identifiers in the topology information that is consistent with the model version identifier in the target request.

[0015] In a second aspect, the present application further provides a method for determining an inference result, which is applied to a registration center. During the inference service update process, the method includes:

[0016] According to the number of inference instances in the instance set corresponding to each model version identifier, calculate the ratio of the number of instances corresponding to each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0017] The quantity ratio corresponding to each model version identifier is sent to the gateway, and the mapping relationship between each model version identifier and different Prompt templates is sent to the Prompt instance; the quantity ratio is used for the gateway to use the model version identifier to mark the multiple user requests received according to each quantity ratio to obtain the corresponding inference request, and send each inference request to the Prompt instance according to the routing rules. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and to splice the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0018] In a third aspect, the present application further provides a method for determining an inference result, which is applied to a gateway. During the inference service update process, the method includes:

[0019] Receive the number ratios corresponding to multiple model version identifiers sent by the registration center; the number ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state;

[0020] According to the proportion of the number corresponding to each model version identifier, multiple user requests are marked with the model version identifier to obtain the corresponding inference request, and the inference request is sent to the Prompt instance; the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and the target Prompt template and the request data carried in the inference request are spliced ​​to obtain the target request, and the target request is sent to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0021] In a fourth aspect, the present application further provides a method for determining an inference result, which is applied to a Prompt instance and includes:

[0022] The target Prompt template is determined based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by marking multiple user requests received by the gateway according to the number ratio corresponding to each model version identifier, and the number ratio is the ratio sent by the registration center received by the gateway. The number ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0023] Concatenate the target prompt template and the request data carried in the inference request to obtain the target request;

[0024] A target request is sent to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0025] In a fifth aspect, the present application further provides a device for determining an inference result, which is provided in the first inference instance. During the inference service update process, the device includes:

[0026] The first receiving module is used to receive the target request sent by the Prompt instance; the target request is obtained by splicing the target Prompt template and the request data carried in the inference request for the Prompt instance, the target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request, and the mapping relationship between each model version identifier and different Prompt templates, the inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier, the quantity ratio is the ratio sent by the registration center received by the gateway, the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are in healthy state inference instances;

[0027] The determination module is used to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0028] In a sixth aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0029] In a seventh aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0030] The above-mentioned method, apparatus, computer device, and storage medium for determining inference results receive the target prompt template and request data carried in the inference request during the inference service update process, concatenate them, and obtain the target request. Based on the model version identifier in the target request, the method then determines the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, thereby determining the inference result based on the target request through the target inference instance. This method dynamically adapts the architecture of the inference instance and prompt template, ensuring a matching relationship between the inference instance and the prompt template, thereby avoiding situations where the inference model deployed on the inference instance has poor inference results due to incompatibility between the inference instance and the prompt template, thereby ensuring that good inference results can be obtained during the inference service update process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is an application environment diagram of the method for determining the reasoning result provided in an embodiment of the present application;

[0033] Figure 2 This is one of the flowcharts of the method for determining the reasoning result provided in the embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of the Prompt instance, reasoning instance, and registration center communication process provided in the embodiments of the present application;

[0035] Figure 4 This is a flow chart of a method for determining a target reasoning instance provided by an embodiment of the present application;

[0036] Figure 5 This is the second flow chart of the method for determining the reasoning result provided in the embodiment of the present application;

[0037] Figure 6 This is a schematic diagram of the registration center obtaining a mapping relationship and issuing the mapping relationship to the Prompt instance provided in an embodiment of the present application;

[0038] Figure 7Schematic diagram of the method for determining an instance set provided in an embodiment of the present application;

[0039] Figure 8 This is a flow chart of the example health check process provided in the embodiment of the present application;

[0040] Figure 9 This is the third flow chart of the method for determining the reasoning result provided in the embodiment of the present application;

[0041] Figure 10 This is a schematic diagram of the communication between the registration center and the gateway provided in an embodiment of the present application;

[0042] Figure 11 This is the fourth flow chart of the method for determining the reasoning result provided in the embodiment of the present application;

[0043] Figure 12 This is the fifth flowchart of the method for determining the reasoning result provided in the embodiment of the present application;

[0044] Figure 13 This is a structural block diagram of an inference result determination device provided in an embodiment of the present application;

[0045] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] Large-model inference is currently a hot technology, and numerous products are already integrating it to enhance their services. Prompt engineering is a crucial component of large-model inference, directly impacting the final inference performance. Because large-model training relies on large amounts of data, the newly trained inference model often performs poorly with the original prompt template, necessitating the generation of a new prompt template to adapt it to the new inference model. However, during large-model inference service updates, the original prompt template and the new inference model may coexist, or the original prompt template, the original prompt template, the original inference model, and the new inference model may all coexist, resulting in poor inference performance. For example, the original prompt template used during inference model training for version V1 was "Please generate an engaging recommendation based on the following product description: [Product Description]." The inference model was upgraded to version V2. The new inference model incorporates user behavior data (such as click-through rate and conversion rate) during training and is optimized for sensitivity to promotional information. At this point, the prompt template that best matches the new inference model should be adjusted to: "Please generate high-conversion recommendation copy based on the following product description and promotional information: [Product Description] [Promotional Information]." However, during large-model inference service updates, due to the large size of the model image, service deployment uses a batch update strategy (for example, updating 50% of servers first). During the transition between the old and new services, user requests are randomly assigned to the V1 or V2 model. When user requests are routed to the new inference model, the original prompt input, generated using the original prompt template and excluding the promotional information field, results in poor inference performance using the new inference model. Therefore, providing a prompt template that can match the inference model during the inference service update process, thereby ensuring good inference performance, has become a pressing technical issue in this field.

[0048] In order to solve the above technical problems, the present application provides a method for determining an inference result, which can be applied to Figure 1 The application environment shown includes the gateway, registration center, prompt instance and reasoning instance. The reasoning service can correspond to multiple reasoning instances. Figure 1 Figure 1 shows four inference instances: inference instance 11, inference instance 12, inference instance 13, and inference instance 14. Each inference instance includes an inference process and a proxy sidecar. An inference instance is the carrier of an inference model, and the model version identifier is the version identifier of the inference model. Some inference instances deploy the original inference model, while others deploy the new inference model.

[0049] The registration center receives registration information for inference instances corresponding to the inference service, performs health checks on registered inference instances, generates topology information based on the registration information of healthy inference instances, and sends the topology information to each healthy inference instance. The registration information may include the model version identifier and address information of the inference instance.

[0050] The registration center supports binding model version identifiers to Prompt templates, and issues a mapping relationship between model version identifiers and Prompt templates to Prompt instances. The Prompt template can include the original Prompt template and the new Prompt template. The model version identifier can include the model version identifier of the new inference model and the model version identifier of the original inference model that matches the original Prompt template. In the mapping relationship, the original Prompt template corresponds to the model version identifier of the original inference model, and the new Prompt template corresponds to the model version identifier of the new inference model. The configuration center can also calculate the proportion of inference instances corresponding to each model version identifier in real time based on the registration information of each inference instance corresponding to the inference service, and push the proportion to the gateway via a persistent connection.

[0051] The gateway receives the user request and, based on the real-time calculation of the model version ID's corresponding quantity ratio by the registration center, adds the model version information to the user request. This effectively marks healthy inference instances with the model version ID. The gateway then forwards the marked user request to the Prompt instance according to the routing rules.

[0052] The Prompt instance manages Prompt templates: This management is combined with model versioning. The Prompt template for inference, known as the target Prompt template, is determined based on the labeled user request and the mapping relationship. The target Prompt template is concatenated with the labeled user request to generate the target request, which is then sent to the inference instance.

[0053] When an inference instance is started, the basic information of the inference instance is registered with the registration center through the inference service. The inference service can provide a health check interface for the registration center to check the health of the inference instance through heartbeat to determine whether the inference instance is in a healthy state.

[0054] The inference service enables automatic addressing, implementing a service mesh proxy service. It automatically determines the inference instance corresponding to the target request based on the model version identifier, and executes large-model inference using the corresponding inference instance. By adding a sidecar to the inference instance, the inference service's proxy capabilities are implemented, completing service registration and automatic addressing. This is simple and easy to use, requiring no modifications to the inference service.

[0055] In an exemplary embodiment, Figure 2 As shown, Figure 2 This is one of the flow charts of the method for determining the inference result provided in the embodiment of the present application, in which the method is applied to Figure 1 The reasoning example in is used as an example to illustrate the reasoning example. Figure 1 In any inference instance, during the process of updating the inference service, the method includes the following steps S201 to S202:

[0056] S201, receiving a target request sent by a Prompt instance; the target request is obtained by the Prompt instance concatenating the target Prompt template and the request data carried in the inference request.

[0057] The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request, the mapping relationship between each model version identifier and different Prompt templates, the inference request is obtained by the gateway by marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier, the quantity ratio is the ratio sent by the registration center received by the gateway, the quantity ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state.

[0058] Figure 3 This is a schematic diagram of the Prompt instance, reasoning instance, and registration center communication process provided by the embodiment of this application. Figure 1 and Figure 3 For example, when reasoning instance 11, reasoning instance 12, reasoning instance 13, and reasoning instance 14 are started, they send their respective registration information to the registration center, which then stores the registration information, thereby registering the reasoning instance. The registration information may include a model version identifier and address information, which may include an Internet Protocol (IP) address and a Media Access Control (MAC) address. The registration center also stores topology information generated based on the registration information of the reasoning instance and sends the topology information to the reasoning instance.

[0059] Taking the startup of inference instance 11 as an example, after inference instance 11 is registered, the registration center can call the health check interface provided by the inference service to perform a health check on inference instance 11 and obtain its status. If the status of inference instance 11 is healthy and it does not exist in the instance set corresponding to the model version identifier, inference instance 11 is added to the instance set corresponding to the model version identifier. In other words, inference instance 11 participates in the calculation of the number of inference instances under the corresponding model version identifier. Furthermore, assuming that before inference instance 11 is started, the topology information in the registration center includes the registration information for inference instances 11, 12, 13, and 14, that is, the registration information for inference instance 11 exists in the topology information. In this case, there is no need to update the topology information based on the registration information for inference instance 11. If the registration information for inference instance 11 does not exist in the topology information, the topology information needs to be updated based on the registration information for inference instance 11. In other words, if the status of inference instance 11 is healthy and its registration information does not exist in the topology information, the topology information needs to be updated based on the registration information for inference instance 11.

[0060] If the state of the reasoning instance 11 is an unhealthy state and the registration information of the reasoning instance 11 exists in the topology information, the registration information of the reasoning instance 11 in the topology information is removed to obtain new topology information.

[0061] The registration center calculates the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances based on the number of inference instances in the instance set under each model version identifier, and sends the ratio of the number of instances corresponding to each model version identifier to the gateway.

[0062] The gateway tags multiple received user requests based on the corresponding model version IDs, using the model version IDs to determine the inference requests corresponding to the user requests. For example, if the model version IDs include V1 and V2, with V1 accounting for 3 / 10 and V2 accounting for 7 / 10, then if there are 100 user requests, 30 requests can be tagged with V1 and the remaining 70 requests can be tagged with V2. The 30 requests tagged with V1 will then be inferred using the V1 inference model, while the 70 requests tagged with V2 will be inferred using the V2 inference model. Traffic distribution based on the corresponding model version IDs allows for dynamic adaptation of the inference model to the Prompt template. This eliminates the need to modify the rollout process and allows for rolling updates, reducing operational costs. After tagging the user requests, the gateway sends the 100 inference requests to the Prompt instance according to routing rules.

[0063] The Prompt instance determines the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. It then concatenates the target Prompt template with the request data carried in the inference request to generate the target request and sends the target request to the first inference instance. The mapping relationship is sent by the registration center to the Prompt instance.

[0064] S202, according to the model version identifier in the target request, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0065] The proxy sidecar in the first inference instance can determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request. If the target inference instance is the first inference instance, the proxy sidecar in the first inference instance can forward the target request to the inference process in the first inference instance, and the inference process in the first inference instance determines the inference result based on the target request. If the target inference instance is not the first inference instance, the proxy sidecar in the first inference instance can use another inference instance corresponding to the model version identifier in the target request as the target inference instance, forward the target request to the proxy sidecar in the target inference instance, and the proxy sidecar in the target inference instance will forward the target request to the inference process in the target inference instance, and the inference process will determine the inference result based on the target request.

[0066] The Prompt instance sends a target request to the first reasoning instance, which can be Figure 1 For any of the inference instances shown in , if the model version identifier in the target request is V2, and the model version identifier of the first inference instance is V1, then the model version identifier of the first inference instance is inconsistent with the model version identifier in the target request. In this case, the first inference instance can determine the inference instance with the model version identifier of V2 based on the topology information sent by the received registration center, and forward the target request to the inference instance with the model version identifier of V2, and the inference instance with the model version identifier of V2 determines the inference result based on the target request. By forwarding the target request, the target Prompt template in the target request matches the inference model corresponding to the inference instance with the model version identifier of V2, and the target request can be processed by the inference model to improve the accuracy of the obtained inference result.

[0067] If the model version identifier in the target request is V2, and the model version identifier of the first inference instance is also V2, it means that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance, that is, the target Prompt template in the target request matches the inference model corresponding to the first inference instance. The target request can be processed through the inference model to improve the accuracy of the obtained inference result.

[0068] In this embodiment, during the inference service update process, the target request is obtained by receiving the request data carried in the target prompt template and the inference request, and then concatenating them. Based on the model version identifier in the target request, the target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service, thereby determining the inference result based on the target request using the target inference instance. This achieves a dynamically adaptable architecture for inference instances and prompt templates, ensuring a matching relationship between inference instances and prompt templates. This avoids situations where the inference model deployed on the inference instance has poor inference results due to incompatibility between the inference instance and the prompt template, thus ensuring good inference results during the inference service update process.

[0069] In an exemplary embodiment, Figure 4 As shown, Figure 4 This is a flow chart of a method for determining a target reasoning instance provided by an embodiment of the present application. The above-mentioned S202 includes the following steps S401 to S403:

[0070] S401 : Determine a model version identifier corresponding to a first inference instance based on topology information of multiple inference instances corresponding to an inference service; the topology information includes a model version identifier and an address identifier corresponding to each inference instance.

[0071] S402: Compare the model version identifier in the target request with the model version identifier corresponding to the first reasoning instance to obtain a comparison result.

[0072] S403: Determine, based on the comparison result, a target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service.

[0073] In this embodiment, by comparing the model version identifier in the target request with the model version identifier corresponding to the first reasoning instance to obtain a comparison result, the target reasoning instance corresponding to the model version identifier in the target request is determined from multiple reasoning instances corresponding to the reasoning service. The target reasoning instance that matches the target Prompt template in the target request can be determined, and the target reasoning instance and the target Prompt template are adapted to each other, ensuring that good reasoning effects can be obtained during the update of the reasoning service.

[0074] In an exemplary embodiment, the above S403 can be implemented as follows:

[0075] If the comparison result is that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance, the first inference instance is used as the target inference instance.

[0076] If the comparison result shows that the model version identifier in the target request is inconsistent with the model version identifier corresponding to the first inference instance, then determine the second inference instance from each inference instance according to the model version identifier corresponding to each inference instance in the topology information, and use the second inference instance as the target inference instance;

[0077] The second inference instance is an inference instance corresponding to the target model version identifier, and the target model version identifier is a model version identifier among the model version identifiers in the topology information that is consistent with the model version identifier in the target request.

[0078] It should be noted that the topology information may contain multiple model version identifiers that are consistent with the model version identifier in the target request. In this case, any inference instance corresponding to the model version identifier that is consistent with the model version identifier in the target request can be used as the second inference instance. Alternatively, based on load balancing, the inference instance with the lowest load can be selected from multiple inference instances as the second target inference instance, and the model version identifiers of these multiple inference instances are consistent with the model version identifier in the target request.

[0079] In this embodiment, a target reasoning instance that matches the target Prompt template in the target request can be determined, and the target reasoning instance and the target Prompt template are adapted to each other, thereby ensuring that good reasoning effects can be obtained during the reasoning service update process.

[0080] In an exemplary embodiment, Figure 5 As shown, Figure 5 This is the second flow chart of the method for determining the inference result provided in the embodiment of the present application, in which the method is applied to Figure 1 Taking the registration center in the example as an example, during the process of updating the inference service, the method includes the following steps S501 to S502:

[0081] S501, calculate the ratio of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state.

[0082] S502: Send the quantity ratio corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance.

[0083] Among them, the quantity ratio is used for the gateway to mark the multiple user requests received according to each quantity ratio using the model version identifier to obtain the corresponding inference request, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and to splice the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0084] like Figure 6 As shown, Figure 6 This is a schematic diagram of the registration center obtaining the mapping relationship and issuing the mapping relationship to the Prompt instance provided in the embodiment of the present application. Figure 6 , which shows that the registration center obtains the mapping relationship between each configured model version identifier and different Prompt templates, and sends the mapping relationship to Prompt instance 11, Prompt instance 12 and Prompt instance 13.

[0085] In this embodiment, during the update of the inference service, the registration center calculates the ratio of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier, sends the ratio of the number corresponding to each model version identifier to the gateway, and sends the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance, thereby realizing dynamic adaptation of the architecture of the inference instance and the Prompt template, ensuring the matching relationship between the inference instance and the prompt template, and avoiding the situation where the inference effect of the inference model deployed on the inference instance is poor due to incompatibility between the inference instance and the prompt template, thereby ensuring that good inference effect can be obtained during the update of the inference service.

[0086] In an exemplary embodiment, Figure 7 As shown, Figure 7 : is a flow chart of a method for determining an instance set provided in an embodiment of the present application, the method comprising the following steps S701 to S703:

[0087] S701, receive registration information sent by multiple inference instances corresponding to the inference service, and register the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and send the topology information to each inference instance; the registration information includes the model version identifier and address identifier corresponding to the inference instance.

[0088] S702: Call the health check interface provided by the inference service to perform a health check on the registered inference instance to obtain the status of the registered inference instance.

[0089] S703, if the status of the registered reasoning instance is healthy, and the registered reasoning instance does not exist in the instance set under the model version identifier corresponding to the registered reasoning instance, then the registered reasoning instance is added to the instance set under the corresponding model version identifier.

[0090] In an exemplary embodiment, Figure 8 As shown, Figure 8 It is a flow chart of the instance health check process provided by the embodiment of the present application. The registration center receives a registration request including registration information sent by the reasoning instance, saves the registration information, periodically checks the reasoning instance through the health check interface of the reasoning service, and judges the health status of the reasoning instance. If the reasoning instance is in a healthy state, it means that the reasoning instance is normal, then the reasoning instance participates in the calculation of the proportion of the number of instances under the model version identifier corresponding to the reasoning instance. If the reasoning instance is not included in the topology information, the registration information of the reasoning instance is added to the topology information. If the reasoning instance is abnormal or times out, you can try to access it again through the health check interface of the reasoning service to confirm whether the reasoning instance has returned to normal, avoid misjudgment due to a short-term fault, and ensure the accuracy of the judgment of the health status of the reasoning instance. If it has not returned to normal, it is determined that the reasoning instance is abnormal. If it has returned to normal, it is determined that the reasoning instance is normal. In the case of an abnormal reasoning instance, if the registration information of the reasoning instance exists in the topology information, the registration information of the reasoning instance is removed from the topology information.

[0091] In this embodiment, when the state of the registered inference instance is healthy and there is no registered inference instance in the instance set under the model version identifier corresponding to the registered inference instance, the registered inference instance is added to the instance set under the corresponding model version identifier, so that the healthy inference instance participates in the calculation of the number of inference instances in the instance set to obtain the quantity ratio, and then the gateway uses the model version identifier to mark the multiple user requests received according to each quantity ratio to obtain the corresponding inference request, and sends each inference request to the Prompt instance according to the routing rule. The Prompt instance determines the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and splices the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The first inference instance determines the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance. Implement an architecture that dynamically adapts inference instances and prompt templates to ensure a matching relationship between inference instances and prompt templates, thereby avoiding situations where the inference model deployed on the inference instance has poor inference results due to incompatibility between the inference instance and the prompt template, and ensuring good inference results during the inference service update process.

[0092] In an exemplary embodiment, Figure 9 As shown, Figure 9 This is the third flow chart of the method for determining the inference result provided in the embodiment of the present application, in which the method is applied to Figure 1 Taking the gateway in FIG. 1 as an example, during the process of updating the inference service, the method includes the following steps S901 to S902:

[0093] S901, receiving the quantity ratio corresponding to multiple model version identifiers sent by the registration center; the quantity ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state.

[0094] like Figure 10 As shown, Figure 10 This is a schematic diagram of the communication between the registration center and the gateway provided in the embodiment of the present application. The registration center can send the number ratio corresponding to the model version identifier to the gateway in real time.

[0095] S902 , based on the quantity ratio corresponding to each model version identifier, use the model version identifier to mark multiple user requests to obtain corresponding inference requests, and send the inference requests to the Prompt instance.

[0096] Among them, the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and to splice the target Prompt template and the request data carried in the inference request to obtain the target request, and send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0097] Since the gateway can distribute traffic according to the proportion of the number corresponding to the model version identifier, dynamic inference model and prompt template adaptation can be achieved. Therefore, there is no need to change the online process and a direct rolling update can be performed, reducing operation and maintenance costs.

[0098] In this embodiment, the gateway receives the number ratios corresponding to multiple model version identifiers sent by the registration center, and based on the number ratios corresponding to each model version identifier, uses the model version identifier to mark multiple user requests to obtain corresponding inference requests, and then sends the inference request to the Prompt instance. The Prompt instance determines the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and splices the target Prompt template with the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The first inference instance determines the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, and determines the inference result based on the target request through the target inference instance. This implements the architecture of dynamically adapting the inference instance and the Prompt template, ensuring the matching relationship between the inference instance and the prompt template, so as to avoid the situation where the inference model deployed on the inference instance has poor inference effect due to incompatibility between the inference instance and the prompt template, and ensures that good inference effect can be obtained during the inference service update process.

[0099] In an exemplary embodiment, Figure 11 As shown, Figure 11 This is the fourth flow chart of the method for determining the inference result provided in the embodiment of the present application, in which the method is applied to Figure 1 Taking the Prompt example in the example as an example, during the process of updating the inference service, the method includes the following steps S1101 to S1103:

[0100] S1101 : Determine a target Prompt template based on a model version identifier corresponding to an inference request and a mapping relationship between each model version identifier and different Prompt templates.

[0101] The inference request is the proportion of the number corresponding to each model version identifier of the gateway. The gateway uses the model version identifier to mark multiple user requests received. The proportion of the number is the proportion sent by the registration center received by the gateway. The proportion of the number is determined by the registration center based on the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set. The inference instances in the instance set are in a healthy state.

[0102] S1102: Concatenate the target Prompt template and the request data carried in the inference request to obtain a target request.

[0103] S1103, sending a target request to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0104] In this embodiment, the target Prompt template is determined based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, the target Prompt template and the request data carried in the inference request are spliced ​​to obtain the target request, and the target request is sent to the first inference instance to achieve dynamic adaptation of the architecture of the inference instance and the Prompt template, ensure the matching relationship between the inference instance and the prompt template, and avoid the situation where the inference effect of the inference model deployed on the inference instance is poor due to incompatibility between the inference instance and the prompt template, thereby ensuring that good inference effect can be obtained during the update of the inference service.

[0105] In an exemplary embodiment, Figure 12 As shown, Figure 12 This is a fifth flow chart of a method for determining an inference result provided in an embodiment of the present application, the method comprising:

[0106] S1201, the registration center receives registration information sent by multiple inference instances corresponding to the inference service, and registers the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and sends the topology information to each inference instance.

[0107] S1202: The registration center calls the health check interface provided by the inference service to perform a health check on the registered inference instance to obtain the status of the registered inference instance.

[0108] S1203, if the status of the registered reasoning instance is healthy, and the registered reasoning instance does not exist in the instance set under the model version identifier corresponding to the registered reasoning instance, the registration center adds the registered reasoning instance to the instance set under the corresponding model version identifier.

[0109] S1204: The registration center calculates the ratio of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier.

[0110] S1205: The registration center sends the quantity ratio corresponding to each model version identifier to the gateway, and sends the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance.

[0111] S1206: The gateway receives the quantity ratios corresponding to the multiple model version identifiers sent by the registration center.

[0112] S1207: The gateway uses the model version identifier to mark multiple user requests according to the quantity ratio corresponding to each model version identifier to obtain corresponding inference requests, and sends the inference requests to the Prompt instance.

[0113] S1208: The Prompt instance determines a target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates.

[0114] S1209: Concatenate the target Prompt template and the request data carried in the inference request to obtain the target request.

[0115] S1210: Send a target request to the first reasoning instance.

[0116] S1211, according to the model version identifier in the target request, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0117] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0118] Based on the same inventive concept, embodiments of the present application also provide an inference result determination device for implementing the aforementioned inference result determination method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the inference result determination device provided below can be found in the limitations of the inference result determination method described above and will not be repeated here.

[0119] In an exemplary embodiment, Figure 13 As shown, Figure 13 1300 is a structural block diagram of an apparatus for determining an inference result provided in an embodiment of the present application. The apparatus 1300 is provided in a first inference instance. During the inference service update process, the apparatus 1300 includes:

[0120] The first receiving module 1301 is used to receive a target request sent by a Prompt instance; the target request is obtained by splicing the target Prompt template and the request data carried in the inference request by the Prompt instance; the target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier; the quantity ratio is the ratio sent by the registration center received by the gateway; the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each instance set; and the inference instances in the instance set are in healthy state inference instances.

[0121] The first determination module 1302 is used to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0122] In one embodiment, the first determining module 1302 includes:

[0123] A first determining unit is configured to determine a model version identifier corresponding to a first inference instance based on topology information of multiple inference instances corresponding to the inference service; the topology information includes a model version identifier and an address identifier corresponding to each inference instance;

[0124] A second determining unit is configured to compare the model version identifier in the target request with the model version identifier corresponding to the first reasoning instance to obtain a comparison result;

[0125] The third determining unit is configured to determine, based on the comparison result, a target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service.

[0126] In one embodiment, the third determining unit is specifically configured to use the first inference instance as the target inference instance if the comparison result shows that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance.

[0127] If the comparison result shows that the model version identifier in the target request is inconsistent with the model version identifier corresponding to the first inference instance, then determine the second inference instance from each inference instance according to the model version identifier corresponding to each inference instance in the topology information, and use the second inference instance as the target inference instance;

[0128] The second inference instance is an inference instance corresponding to the target model version identifier, and the target model version identifier is a model version identifier among the model version identifiers in the topology information that is consistent with the model version identifier in the target request.

[0129] In an exemplary embodiment, a device for determining an inference result is provided. During an inference service update process, the device includes:

[0130] The calculation module is used to calculate the ratio of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state.

[0131] The first sending module is used to send the quantity ratio corresponding to each model version identifier to the gateway, and to send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance; the quantity ratio is used for the gateway to use the model version identifier to mark the multiple user requests received according to each quantity ratio to obtain the corresponding inference request, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and to splice the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0132] In an exemplary embodiment, the apparatus may further include:

[0133] A second receiving module is configured to receive registration information sent by multiple inference instances corresponding to the inference service, register the inference instances based on the registration information of each inference instance, generate topology information corresponding to the multiple inference instances, and send the topology information to each inference instance; the registration information includes a model version identifier and an address identifier corresponding to the inference instance;

[0134] The calling module calls the health check interface provided by the inference service to perform a health check on the registered inference instance to obtain the status of the registered inference instance;

[0135] The adding module is used to add the registered inference instance to the instance set under the corresponding model version identifier if the status of the registered inference instance is healthy and the registered inference instance does not exist in the instance set under the model version identifier corresponding to the registered inference instance.

[0136] In an exemplary embodiment, another apparatus for determining an inference result is provided. During an inference service update process, the apparatus includes:

[0137] A third receiving module is configured to receive a quantity ratio corresponding to multiple model version identifiers sent by a registration center; the quantity ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state;

[0138] The second sending module is used to mark multiple user requests with the model version identifier according to the quantity proportion corresponding to each model version identifier to obtain the corresponding inference request, and send the inference request to the Prompt instance; the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and to splice the target Prompt template and the request data carried in the inference request to obtain the target request, and send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0139] In an exemplary embodiment, another inference result determination device is provided. During an inference service update process, the device includes:

[0140] The second determination module is used to determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by marking the multiple user requests received by the gateway according to the number ratio corresponding to each model version identifier, and the number ratio is the ratio sent by the registration center received by the gateway, and the number ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0141] The splicing module is used to splice the target prompt template and the request data carried in the inference request to obtain the target request;

[0142] The third sending module is used to send a target request to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0143] Each module in the above-mentioned inference result determination device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0144] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for determining an inference result is implemented.

[0145] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0146] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the technical solution in any of the above method embodiments is implemented. The implementation principles and technical effects are similar and will not be repeated here.

[0147] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the technical solution in any of the above method embodiments is implemented. The implementation principles and technical effects are similar and will not be repeated here.

[0148] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the technical solution in any of the above method embodiments is implemented. The implementation principle and technical effects are similar and will not be repeated here.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0151] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for determining an inference result, characterized in that: The method is applied to a first reasoning instance, and during a reasoning service update process, the method includes: Receive a target request sent by a Prompt instance; the target request is obtained by the Prompt instance splicing the target Prompt template and the request data carried in the inference request; the target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple received user requests using the model version identifier according to the quantity ratio corresponding to each model version identifier; the quantity ratio is the ratio sent by the registration center received by the gateway; the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each instance set; the inference instances in the instance set are inference instances in a healthy state; According to the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

2. The method according to claim 1, characterized in that The determining, according to the model version identifier in the target request, from a plurality of inference instances corresponding to the inference service, a target inference instance corresponding to the model version identifier in the target request includes: Determine, based on topology information of multiple inference instances corresponding to the inference service, a model version identifier corresponding to the first inference instance; the topology information includes a model version identifier and an address identifier corresponding to each of the inference instances; Comparing the model version identifier in the target request with the model version identifier corresponding to the first reasoning instance to obtain a comparison result; According to the comparison result, a target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service.

3. The method according to claim 2, characterized in that Determining, based on the comparison result, a target inference instance corresponding to the model version identifier in the target request from a plurality of inference instances corresponding to the inference service, includes: If the comparison result is that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance, the first inference instance is used as the target inference instance; If the comparison result is that the model version identifier in the target request is inconsistent with the model version identifier corresponding to the first inference instance, determining a second inference instance from each of the inference instances according to the model version identifier corresponding to each of the inference instances in the topology information, and using the second inference instance as the target inference instance; The second inference instance is an inference instance corresponding to a target model version identifier, and the target model version identifier is a model version identifier in each of the model version identifiers in the topology information that is consistent with the model version identifier in the target request.

4. A method for determining an inference result, characterized in that: The method is applied to a registration center, and during the process of updating the inference service, the method includes: According to the number of inference instances in the instance set corresponding to each model version identifier, calculate the ratio of the number of instances corresponding to each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each of the instance sets, and the inference instances in the instance set are inference instances in a healthy state; Send the quantity ratio corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance; the quantity ratio is used for the gateway to mark the multiple user requests received according to each quantity ratio using the model version identifier to obtain the corresponding inference request, and send each inference request to the Prompt instance according to the routing rule, the inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship, and splice the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance, the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

5. The method according to claim 4, characterized in that The method further comprises: Receive registration information sent by multiple inference instances corresponding to the inference service, register the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and send the topology information to each inference instance; the registration information includes a model version identifier and an address identifier corresponding to the inference instance; Call the health check interface provided by the inference service to perform a health check on the registered inference instance to obtain the status of the registered inference instance; If the status of the registered reasoning instance is a healthy state, and the registered reasoning instance does not exist in the instance set under the model version identifier corresponding to the registered reasoning instance, then the registered reasoning instance is added to the instance set under the corresponding model version identifier.

6. A method for determining an inference result, characterized in that: The method is applied to a gateway, and during a process of inference service update, the method includes: Receive a number ratio corresponding to multiple model version identifiers sent by a registration center; the number ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state; According to the proportion of the number corresponding to each model version identifier, the model version identifier is used to mark multiple user requests to obtain corresponding inference requests, and the inference request is sent to the Prompt instance; the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and the target Prompt template and the request data carried in the inference request are spliced ​​to obtain a target request, and the target request is sent to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

7. A method for determining an inference result, characterized in that: The method is applied to the Prompt instance and includes: The target Prompt template is determined based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier, the quantity ratio is the ratio sent by the registration center received by the gateway, the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state; Concatenate the target prompt template and the request data carried in the inference request to obtain a target request; The target request is sent to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

8. A device for determining an inference result, characterized in that: The device is set in the first reasoning instance, and during the process of updating the reasoning service, the device includes: The first receiving module is used to receive a target request sent by a Prompt instance; the target request is obtained by splicing the target Prompt template and the request data carried in the inference request by the Prompt instance; the target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier; the quantity ratio is the ratio sent by the registration center received by the gateway; the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each instance set; and the inference instances in the instance set are inference instances in a healthy state; A determination module is used to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images