Inference result determination method and device, computer equipment and storage medium

Abstract

The invention relates to a reasoning result determination method and device, computer equipment and a storage medium. The method comprises the steps of receiving a target request obtained by splicing request data carried in a target Prompt template and a reasoning request in a reasoning service updating process, determining a target reasoning instance corresponding to a model version identifier in the target request from a plurality of reasoning instances corresponding to a reasoning service according to the model version identifier in the target request, and updating the target reasoning instance according to the determined target reasoning instance. And determining a reasoning result based on the target request through the target reasoning instance. According to the method and the device, the architecture of dynamically adapting the reasoning instance and the Prompt template is realized, the matching relationship between the reasoning instance and the Prompt template is ensured, the situation that the reasoning effect of a reasoning model deployed on the reasoning instance is poor due to incompatibility of the reasoning instance and the Prompt template is avoided, and a good reasoning effect can be obtained in the reasoning service updating process.

Description

Technical Field

[0001] The present application relates to the field of model reasoning technology, and in particular to a method, apparatus, computer equipment, and storage medium for determining reasoning results. Background Art

[0002] Large-model reasoning is currently the hottest technology, and a large number of products have begun integrating large-model reasoning capabilities to enhance their services. Prompt engineering is a critical component of the large-model reasoning process, directly impacting the final inference results. Because large-model training relies on large amounts of data, the newly trained inference model often performs poorly with the original prompt template, necessitating the generation of a new prompt template to adapt to the new inference model. However, during large-model inference service updates, the old prompt template and the new inference model may coexist, resulting in poor inference results during this period. Therefore, providing a prompt template that can match the inference model during inference service updates, thereby ensuring good inference results, has become a pressing technical issue in this field. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, device, computer equipment, and storage medium for determining inference results that can obtain a prompt template that matches the inference model during the inference service update process to address the above technical problems, thereby ensuring good inference results.

[0004] In a first aspect, the present application provides a method for determining an inference result, which is applied to a first inference instance. During an inference service update, the method includes:

[0005] Receive the target request sent by the Prompt instance; the target request is obtained by splicing the target Prompt template and the request data carried in the inference request by the Prompt instance. The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. The inference request is obtained by the gateway marking multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier. The quantity ratio is the ratio sent by the registration center received by the gateway. The quantity ratio is determined by the registration center based on the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set. The inference instances in the instance set are inference instances in a healthy state.

[0006] Based on the model version identifier in the target request, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0007] In one embodiment, determining the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request includes:

[0008] According to the topology information of multiple inference instances corresponding to the inference service, determine the model version identifier corresponding to the first inference instance; the topology information includes the model version identifier and address identifier corresponding to each inference instance;

[0009] Compare the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result;

[0010] According to the comparison result, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service.

[0011] In one embodiment, determining the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the comparison result includes: [[ID=!17]]

[0012] If the comparison result is that the model version identifier in the target request is the same as the model version identifier corresponding to the first inference instance, then use the first inference instance as the target inference instance.

[0013] If the comparison result is that the model version identifier in the target request is different from the model version identifier corresponding to the first inference instance, then according to the model version identifiers corresponding to each inference instance in the topology information, determine the second inference instance from each inference instance, and use the second inference instance as the target inference instance;

[0014] Wherein, the second inference instance is the inference instance corresponding to the target model version identifier, and the target model version identifier is the model version identifier that is the same as the model version identifier in the target request among the model version identifiers in the topology information.

[0015] In a second aspect, the present application also provides a method for determining an inference result. The method is applied to a registration center. During the update process of the inference service, the method includes:

[0016] According to the number of inference instances in the instance set corresponding to each model version identifier, calculate the proportion of the number of instances corresponding to each model version identifier to the total number of instances; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are in a healthy state;

[0017] Send the quantity proportion corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance; the quantity proportion is used for the gateway to mark multiple user requests received according to each quantity proportion by using the model version identifier to obtain corresponding inference requests, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship, and splice the target Prompt template and the request data carried in the inference request to obtain a target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0018] Thirdly, the present application further provides a method for determining an inference result. The method is applied to the gateway. During the update process of the inference service, the method includes:

[0019] Receive the quantity proportion corresponding to multiple model version identifiers sent by the registration center; the quantity proportion is determined by the registration center according to the ratio of the instance quantity of the inference instances in the instance set corresponding to each model version identifier to the total instance quantity of the inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state;

[0020] Mark multiple user requests with the model version identifier according to the quantity proportion corresponding to each model version identifier to obtain corresponding inference requests, and send the inference requests to the Prompt instance; the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and splice the target Prompt template and the request data carried in the inference request to obtain a target request, and send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0021] Fourthly, the present application further provides a method for determining an inference result. The method is applied to the Prompt instance. The method includes:

[0022] Determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple user requests received by using the model version identifier according to the quantity ratio corresponding to each model version identifier, the quantity ratio is the ratio sent by the registration center received by the gateway, the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0023] Concatenate the target Prompt template and the request data carried in the inference request to obtain a target request;

[0024] Send the target request to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0025] In a fifth aspect, the present application further provides an inference result determination device, which is arranged in the first inference instance. During the update process of the inference service, the device includes:

[0026] A first receiving module, configured to receive the target request sent by the Prompt instance; the target request is obtained by the Prompt instance concatenating the target Prompt template and the request data carried in the inference request, the target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, the inference request is obtained by the gateway marking multiple user requests received by using the model version identifier according to the quantity ratio corresponding to each model version identifier, the quantity ratio is the ratio sent by the registration center received by the gateway, the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0027] A determination module, configured to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0028] In a sixth aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of any of the above methods are implemented.

[0029] In a seventh aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0030] In the above method, apparatus, computer device, and storage medium for determining inference results, during the update process of the inference service, a target request is obtained by splicing the target Prompt template and the request data carried in the inference request. According to the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance. An architecture for dynamically adapting the inference instance to the Prompt template is implemented, ensuring the matching relationship between the inference instance and the prompt template, so as to avoid the situation where the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, and ensuring that a good inference effect can be obtained during the update process of the inference service. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0032] Figure 1 is an application environment diagram of the method for determining inference results provided by an embodiment of the present application;

[0033] Figure 2 is a schematic flowchart of the method for determining inference results provided by an embodiment of the present application;

[0034] Figure 3 is a schematic communication flowchart of Prompt instances, inference instances, and a registration center provided by an embodiment of the present application;

[0035] Figure 4 is a schematic flowchart of a method for determining a target inference instance provided by an embodiment of the present application;

[0036] Figure 5 is a schematic flowchart of the method for determining inference results provided by an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of the registration center obtaining a mapping relationship and sending the mapping relationship to a Prompt instance provided by an embodiment of the present application;

[0038] Figure 7It is a schematic flowchart of the example set determination method provided by an embodiment of the present application;

[0039] Figure 8 It is a schematic flowchart of the example health check process provided by an embodiment of the present application;

[0040] Figure 9 It is the third schematic flowchart of the inference result determination method provided by an embodiment of the present application;

[0041] Figure 10 It is a communication schematic diagram between the registration center and the gateway provided by an embodiment of the present application;

[0042] Figure 11 It is the fourth schematic flowchart of the inference result determination method provided by an embodiment of the present application;

[0043] Figure 12 It is the fifth schematic flowchart of the inference result determination method provided by an embodiment of the present application;

[0044] Figure 13 It is a structural block diagram of an inference result determination device provided by an embodiment of the present application;

[0045] Figure 14 It is the internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] Large model inference is the hottest technology at present. Currently, a large number of products have begun to incorporate the inference capabilities of large models to enhance their own businesses. During the large model inference process, Prompt engineering is an important part, directly affecting the final inference effect. Due to the large amount of data relied on during the large model training process, it often leads to poor performance of the newly trained inference model under the original prompt template, and new prompt templates need to be regenerated to adapt to the new inference model. However, during the update process of the large model inference service, there will be a situation where the original prompt template coexists with the new inference model, or the original prompt template, the original prompt template, the original inference model, and the new inference model coexist, resulting in relatively poor inference effects during this period. For example, the original prompt template used during the training of the original V1 version of the inference model was "Please generate an attractive recommendation copy based on the following product description: [product description]". The inference model was upgraded to V2. The new inference model of the V2 version added user behavior data (such as click-through rate, conversion rate) during training and optimized the sensitivity to promotional information. At this time, the best-matching prompt template for the new inference model should be adjusted to: "Please generate a high-conversion recommendation copy based on the following product description and promotional information: [product description] [promotional information]". However, during the update process of the large model inference service, due to the large volume of the model image, the service deployment adopts a batch update strategy (such as updating 50% of the servers first). During the alternation period of the old and new services, user requests will be randomly assigned to the V1 model or the V2 model. When a user request is routed to the new inference model, the input original prompt, generated using the original prompt template, does not contain the promotional information field, resulting in relatively poor inference effects obtained by the new inference model. Therefore, how to provide a method to obtain the prompt template matching the inference model during the update process of the inference service, so as to ensure good inference effects, has become a technical problem urgently to be solved in this field.

[0048] To solve the above technical problems, an embodiment of the present application provides a method for determining an inference result, which can be applied to an application environment as shown in Figure 1 including a gateway, a registry, a Prompt instance, and an inference instance. The inference service can correspond to multiple inference instances. Figure 1 Four inference instances, namely inference instance 11, inference instance 12, inference instance 13, and inference instance 14, are shown. An inference instance includes an inference process and a Proxy Sidecar. The inference instance is the carrier of the inference model, and the model version identifier is the version identifier of the inference model. The original inference model is deployed in some inference instances, and the new inference model is deployed in some inference instances.

[0049] Among them, the registration center receives the registration information of the inference instances corresponding to the inference service, performs health checks on the registered inference instances, generates topology information based on the registration information of the inference instances in a healthy state, and sends the topology information to each inference instance in a healthy state. The registration information may include the model version identifier and address information of the inference instance.

[0050] The registration center supports binding by model version identifier to the Prompt template, and issues the mapping relationship between the model version identifier and the Prompt template to the Prompt instance. Among them, the Prompt template may include the original Prompt template and the new Prompt template, and the model version identifier may include the model version identifier of the new inference model and the model version identifier of the original inference model that matches the original Prompt template. The original Prompt template in the mapping relationship corresponds to the model version identifier of the original inference model, and the new Prompt template corresponds to the model version identifier of the new inference model. The configuration center can also calculate the proportion of the number of inference instances corresponding to each model version identifier in real time according to the registration information of each inference instance corresponding to the inference service, and push the proportion to the gateway through a long connection.

[0051] The gateway receives the user request, combines the proportion of the number corresponding to the model version identifier calculated in real time by the registration center, and adds model version information to the user request based on the proportion of the number corresponding to the model version identifier, that is, uses the model version identifier to mark the inference instances in a healthy state. Then, the marked user request is forwarded to the Prompt instance according to the routing rules.

[0052] The Prompt instance manages the Prompt template: manages the Prompt template in combination with the model version management method. And determines the Prompt template for inference according to the marked user request and the mapping relationship, that is, determines the target Prompt template. The target Prompt template is concatenated with the marked user request to obtain a target request, and the target request is sent to the inference instance.

[0053] When the inference instance starts, the basic information of the inference instance is registered in the registration center through the inference service; the inference service can provide a health check interface for the registration center to check the health of the inference instance through heartbeats to determine whether the inference instance is in a healthy state.

[0054] The inference service can achieve automatic addressing, implement the Service Mesh proxy service, automatically determine the inference instance corresponding to the target request according to the model version identifier of the target request, and execute large model inference through the inference instance corresponding to the target request. By adding a Sidecar to the inference instance, the Proxy ability of the inference service is realized, and the registration and automatic addressing of the service are completed. It is simple and easy to use without modifying the inference service.

[0055] In an exemplary embodiment, as Figure 2 shown, Figure 2 is one of the flow diagrams of the inference result determination method provided by the embodiments of the present application. Taking the application of this method to Figure 1 the inference instance in as an example for illustration, the inference instance can be Figure 1 any one of the inference instances in. During the update of the inference service, the method includes the following steps S201 to step S202:

[0056] S201, receiving a target request sent by the Prompt instance; the target request is obtained by the Prompt instance by splicing the target Prompt template and the request data carried in the inference request.

[0057] The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. The inference request is marked by the gateway for multiple user requests received using the model version identifier according to the quantity ratio corresponding to each model version identifier. The quantity ratio is the ratio sent by the registration center received by the gateway. The quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are in a healthy state.

[0058] Figure 3 is the communication flow diagram of the Prompt instance, inference instance, and registration center provided by the embodiments of the present application. Referring to Figure 1 and Figure 3 , exemplarily, when the inference instance 11, inference instance 12, inference instance 13, and inference instance 14 are started, they send their registration information to the registration center, and the registration center saves the registration information to realize the registration of the inference instance. The registration information may include the model version identifier and address information. The address information may include Internet Protocol (IP) address, Media Access Control (MAC) address, etc. The registration center also stores topology information generated based on the registration information of the inference instance and sends the topology information to the inference instance.

[0059] Taking the start of inference instance 11 as an example, after the registration of inference instance 11, the registration center can call the health check interface provided by the inference service to perform a health check on inference instance 11 to obtain the status of inference instance 11. If the status of inference instance 11 is a healthy state and inference instance 11 does not exist in the instance set under the model version identifier corresponding to inference instance 11, then inference instance 11 is added to the instance set under the corresponding model version identifier, that is, inference instance 11 participates in the calculation of the number of instances of the inference instances under the corresponding model version identifier. And, assuming that before the start of inference instance 11, the topology information in the registration center includes the registration information of inference instance 11, inference instance 12, inference instance 13, and inference instance 14, that is, the registration information of inference instance 11 exists in this topology information. In this case, there is no need to update the topology information based on the registration information of inference instance 11. If the registration information of inference instance 11 does not exist in the topology information, then it is necessary to update the topology information based on the registration information of inference instance 11, that is, if the status of inference instance 11 is a healthy state and the registration information of inference instance 11 does not exist in the topology information, then it is necessary to update the topology information based on the registration information of inference instance 11.

[0060] If the status of inference instance 11 is an unhealthy state and the registration information of inference instance 11 exists in the topology information, then the registration information of inference instance 11 in the topology information is removed to obtain the new topology information.

[0061] The registration center calculates the ratio of the number of instances corresponding to each model version identifier to the total number of instances based on the number of instances of the inference instances in the instance set under each model version identifier, and sends the ratio corresponding to each model version identifier to the gateway.

[0062] The gateway marks the multiple user requests received using the model version identifier according to the ratio corresponding to each model version identifier to obtain the inference requests corresponding to the user requests. For example, the model version identifiers include V1 and V2, the ratio corresponding to V1 is 3 / 10, and the ratio corresponding to V2 is 7 / 10. If there are 100 user requests, then 30 requests can be marked with V1 respectively, and the remaining 70 requests can be marked with V2 respectively. The 30 requests marked with V1 are subsequently inferred through the inference model of version V1 to obtain the inference results, and the 70 requests marked with V2 are subsequently inferred through the inference model of version V2 to obtain the inference results. It realizes traffic distribution according to the ratio of the number of instances corresponding to the model version identifier, realizes the dynamic adaptation of the inference model and the Prompt template. Therefore, there is no need to modify the online process, and it can be directly updated incrementally, reducing the operation and maintenance cost. After the gateway marks the user requests, it can send the obtained 100 inference requests to the Prompt instance according to the routing rules.

[0063] The Prompt instance determines the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, and splices the target Prompt template and the request data carried in the inference request to obtain the target request, and sends the target request to the first inference instance. Among them, the mapping relationship is sent by the registration center to the Prompt instance.

[0064] S202. According to the model version identifier in the target request, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0065] The proxy sidecar in the first inference instance can determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request. If the target inference instance is the first inference instance, the proxy sidecar in the first inference instance can forward the target request to the inference process in the first inference instance, and the inference process in the first inference instance determines the inference result based on the target request. If the target inference instance is not the first inference instance, the proxy sidecar in the first inference instance can use another inference instance corresponding to the model version identifier in the target request as the target inference instance, forward the target request to the proxy sidecar in the target inference instance, and the proxy sidecar in the target inference instance forwards the target request to the inference process in the target inference instance, and the inference process determines the inference result based on the target request.

[0066] The Prompt instance sends the target request to the first inference instance, and the first inference instance can be Figure 1 any of the inference instances shown in. If the model version identifier in the target request is V2 and the model version identifier of the first inference instance is V1, then the model version identifier of the first inference instance is inconsistent with the model version identifier in the target request. In this case, the first inference instance can determine the inference instance with the model version identifier of V2 according to the topology information sent by the received registration center, and forward the target request to the inference instance with the model version identifier of V2, and the inference instance with the model version identifier of V2 determines the inference result based on the target request. By forwarding the target request, the target Prompt template in the target request is matched with the inference model corresponding to the inference instance with the model version identifier of V2, and the target request can be processed by this inference model to improve the accuracy of the obtained inference result.

[0067] If the model version identifier in the target request is V2 and the model version identifier of the first inference instance is also V2, it means that the model version identifier in the target request is consistent with the model version identifier corresponding to the first inference instance, that is, the target Prompt template in the target request matches the inference model corresponding to the first inference instance, and the target request can be processed by the inference model to improve the accuracy of the obtained inference result.

[0068] In this embodiment, during the update of the inference service, the target request obtained by splicing the target Prompt template and the request data carried in the inference request is received, and according to the model version identifier in the target request, the target inference instance corresponding to the model version identifier in the target request is determined from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance. An architecture for dynamically adapting the inference instance and the Prompt template is implemented to ensure the matching relationship between the inference instance and the prompt template, so as to avoid the situation that the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, and ensure that a good inference effect can be obtained during the update of the inference service.

[0069] In an exemplary embodiment, as Figure 4 shown, Figure 4 FIG. is a schematic flowchart of a method for determining a target inference instance provided by an embodiment of the present application. The above S202 includes the following steps S401 to S403:

[0070] S401, according to the topology information of multiple inference instances corresponding to the inference service, determine the model version identifier corresponding to the first inference instance; the topology information includes the model version identifier and the address identifier corresponding to each inference instance.

[0071] S402, compare the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result.

[0072] S403, according to the comparison result, determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service.

[0073] In this embodiment, by comparing the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result, and determining the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, the target inference instance that matches the target Prompt template in the target request can be determined, realizing the adaptation of the target inference instance and the target Prompt template, and ensuring that a good inference effect can be obtained during the update of the inference service.

[0074] In an exemplary embodiment, the above S403 can be implemented in the following manner:

[0075] If the comparison result is that the model version identifier in the target request is the same as the model version identifier corresponding to the first inference instance, then the first inference instance is used as the target inference instance.

[0076] If the comparison result is that the model version identifier in the target request is different from the model version identifier corresponding to the first inference instance, then according to the model version identifiers corresponding to the inference instances in the topology information, a second inference instance is determined from each inference instance, and the second inference instance is used as the target inference instance;

[0077] Among them, the second inference instance is the inference instance corresponding to the target model version identifier, and the target model version identifier is the model version identifier that is the same as the model version identifier in the target request among the model version identifiers in the topology information.

[0078] It should be noted that there may be multiple model version identifiers in the topology information that are the same as the model version identifier in the target request. In this case, any inference instance corresponding to a model version identifier that is the same as the model version identifier in the target request can be used as the second inference instance. Or, based on load balancing, the inference instance with a smaller load is selected from multiple inference instances as the second target inference instance, and the model version identifiers of the multiple inference instances are the same as the model version identifier in the target request.

[0079] In this embodiment, the target inference instance matching the target Prompt template in the target request can be determined, realizing the adaptation of the target inference instance to the target Prompt template, and ensuring good inference effects during the update process of the inference service.

[0080] In an exemplary embodiment, as Figure 5 shown, Figure 5 is the second flowchart of the inference result determination method provided by the embodiment of the present application. Taking the application of this method to the Figure 1 registration center as an example, during the update process of the inference service, this method includes the following steps S501 to step S502:

[0081] S501, calculate the proportion of the number of instances corresponding to each model version identifier to the total number of instances according to the number of instances of the inference instances in the instance set corresponding to each model version identifier; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are in a healthy state.

[0082] S502, send the proportion of the number corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance.

[0083] Among them, the quantity ratio is used for the gateway to mark multiple received user requests according to each quantity ratio by using the model version identifier to obtain corresponding inference requests, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier and mapping relationship corresponding to the inference request, and splice the target Prompt template and the request data carried in the inference request to obtain a target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result through the target inference instance based on the target request.

[0084] As Figure 6 shown, Figure 6 is a schematic diagram of the registration center obtaining the mapping relationship and sending the mapping relationship to the Prompt instance provided by the embodiment of the present application. Exemplarily, Figure 6 shows that the registration center obtains the mapping relationship between each configured model version identifier and different Prompt templates, and sends the mapping relationship to Prompt instance 11, Prompt instance 12, and Prompt instance 13.

[0085] In this embodiment, during the update process of the inference service, the registration center calculates the quantity ratio of the instance quantity of the inference instances in the instance set corresponding to each model version identifier to the total instance quantity, sends the quantity ratio corresponding to each model version identifier to the gateway, and sends the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance, realizing the architecture of dynamically adapting the inference instance and the Prompt template, ensuring the matching relationship between the inference instance and the prompt template, so as to avoid the situation that the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, and ensuring that a good inference effect can be obtained during the update process of the inference service.

[0086] In an exemplary embodiment, as Figure 7 shown, Figure 7 is a schematic flowchart of the instance set determination method provided by the embodiment of the present application. The method includes the following steps S701 to step S703:

[0087] S701 receives the registration information sent by multiple inference instances corresponding to the inference service, registers the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and sends the topology information to each inference instance; the registration information includes the model version identifier and the address identifier corresponding to the inference instance.

[0088] S702 calls the health check interface provided by the inference service to perform a health check on the registered inference instances to obtain the status of the registered inference instances.

[0089] S703, if the status of the registered inference instance is a healthy status and the registered inference instance does not exist in the instance set under the model version identifier corresponding to the registered inference instance, adds the registered inference instance to the instance set under the corresponding model version identifier.

[0090] In an exemplary embodiment, as Figure 8 shown, Figure 8 is a schematic flowchart of the instance health check process provided by an embodiment of the present application. The registration center receives a registration request including registration information sent by an inference instance, saves the registration information, periodically checks the inference instance through the health check interface of the inference service, and judges the health status of the inference instance. If the inference instance is in a healthy status, it means that the inference instance is normal. Then, the inference instance participates in the calculation of the proportion of the number of instances under the model version identifier corresponding to the inference instance. When the topology information does not include the inference instance, the registration information of the inference instance is added to the topology information. If the inference instance is abnormal or times out, it can try to access through the health check interface of the inference service again to confirm whether the inference instance has returned to normal, avoiding misjudgment due to a short-term failure and ensuring the accuracy of the judgment of the health status of the inference instance. If it does not return to normal, it is determined that the inference instance is abnormal. If it returns to normal, it is determined that the inference instance is normal. In the case where the inference instance is abnormal, if the registration information of the inference instance exists in the topology information, the registration information of the inference instance is removed from the topology information.

[0091] In this embodiment, when the status of the registered inference instance is the healthy status and there is no registered inference instance in the instance set corresponding to the model version identifier of the registered inference instance, the registered inference instance is added to the instance set corresponding to the model version identifier, so as to enable the inference instance in the healthy status to participate in the calculation of the number of inference instances in the instance set to obtain the quantity ratio. Furthermore, the gateway marks the multiple user requests received by using the model version identifier according to each quantity ratio to obtain the corresponding inference requests, and sends each inference request to the Prompt instance according to the routing rule. The Prompt instance determines the target Prompt template based on the model version identifier and the mapping relationship corresponding to the inference request, and splices the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The first inference instance determines the target inference instance corresponding to the model version identifier in the target request from the multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance. An architecture for dynamically adapting the inference instance and the Prompt template is implemented to ensure the matching relationship between the inference instance and the prompt template, so as to avoid the situation that the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, and ensure that a good inference effect can be obtained during the update process of the inference service.

[0092] In an exemplary embodiment, as Figure 9 shown, Figure 9 FIG. 3 is a third flowchart of the inference result determination method provided by the embodiment of the present application. Taking the gateway in Figure 1 as an example for description, during the update process of the inference service, the method includes the following steps S901 to step S902:

[0093]

[0093] S901, receiving the quantity ratios corresponding to multiple model version identifiers sent by the registration center; the quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in the healthy status.

[0094] As Figure 10 shown, Figure 10 FIG. 4 is a communication schematic diagram between the registration center and the gateway provided by the embodiment of the present application. The registration center can send the quantity ratio corresponding to the model version identifier to the gateway in real time.

[0095] S902, marking the multiple user requests by using the model version identifier according to the quantity ratios corresponding to each model version identifier to obtain the corresponding inference requests, and sending the inference requests to the Prompt instance.

[0096] Among them, the inference request is used for the Prompt instance to determine the target Prompt template corresponding to the inference request based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates, splice the target Prompt template and the request data carried in the inference request to obtain a target request, and send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0097] Since the gateway can distribute traffic according to the proportion of the number corresponding to the model version identifier, realizing the dynamic adaptation of the inference model and the Prompt template, there is no need to modify the online process, and it can be directly updated incrementally, reducing the operation and maintenance cost.

[0098] In this embodiment, the gateway receives the proportion of the number corresponding to multiple model version identifiers sent by the registration center, marks multiple user requests with the model version identifier according to the proportion of the number corresponding to each model version identifier to obtain corresponding inference requests, and sends the inference requests to the Prompt instance. The Prompt instance determines the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship, and splices the target Prompt template and the request data carried in the inference request to obtain a target request, so as to send the target request to the first inference instance. The first inference instance determines the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance. An architecture for dynamically adapting the inference instance and the Prompt template is implemented, ensuring the matching relationship between the inference instance and the prompt template, so as to avoid the situation that the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, and ensuring that a good inference effect can be obtained during the update process of the inference service.

[0099] In an exemplary embodiment, as Figure 11 shown, Figure 11 is the fourth flowchart of the inference result determination method provided by the embodiment of the present application. Taking the method applied to the Prompt instance in Figure 1 as an example, during the update process of the inference service, the method includes the following steps S1101 to step S1103:

[0100] S1101, determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates.

[0101] The inference request is obtained by the gateway marking multiple user requests received according to the quantity proportion corresponding to each model version identifier. The quantity proportion is the proportion sent by the registration center received by the gateway. The quantity proportion is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set. The inference instances in the instance set are inference instances in a healthy state.

[0102] S1102, splice the target Prompt template and the request data carried in the inference request to obtain a target request.

[0103] S1103, send the target request to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

[0104] In this embodiment, the target Prompt template is determined based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. The target Prompt template and the request data carried in the inference request are spliced to obtain a target request, and the target request is sent to the first inference instance, so as to implement an architecture for dynamically adapting the inference instance and the Prompt template, ensure the matching relationship between the inference instance and the prompt template, and avoid the situation that the inference effect of the inference model deployed on the inference instance is poor due to the incompatibility between the inference instance and the prompt template, ensuring that a good inference effect can be obtained during the update of the inference service.

[0105] In an exemplary embodiment, as Figure 12 shown, Figure 12 is the fifth flowchart of the inference result determination method provided by the embodiment of the present application. The method includes:

[0106] S1201, the registration center receives the registration information sent by multiple inference instances corresponding to the inference service, and registers the inference instances based on the registration information of each inference instance to generate topology information corresponding to multiple inference instances, and sends the topology information to each inference instance.

[0107] S1202, the registration center calls the health check interface provided by the inference service to perform a health check on the registered inference instances to obtain the status of the registered inference instances.

[0108] S1203. If the status of the registered inference instance is healthy and there is no registered inference instance in the instance set corresponding to the model version identifier of the registered inference instance, the registration center adds the registered inference instance to the instance set corresponding to the model version identifier.

[0109] S1204. The registration center calculates the proportion of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier.

[0110] S1205. The registration center sends the proportion corresponding to each model version identifier to the gateway and sends the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance.

[0111] S1206. The gateway receives the proportion corresponding to multiple model version identifiers sent by the registration center.

[0112] S1207. The gateway marks multiple user requests with the model version identifier according to the proportion corresponding to each model version identifier to obtain the corresponding inference requests, and sends the inference requests to the Prompt instance.

[0113] S1208. The Prompt instance determines the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates.

[0114] S1209. Concatenate the target Prompt template and the request data carried in the inference request to obtain the target request.

[0115] S1210. Send the target request to the first inference instance.

[0116] S1211. Determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service according to the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0117] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0118] Based on the same inventive concept, an embodiment of the present application also provides a reasoning result determination device for implementing the reasoning result determination method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the reasoning result determination device provided below can refer to the limitations on the reasoning result determination method in the above text, and will not be repeated here.

[0119] In an exemplary embodiment, as Figure 13 shown, Figure 13 is a structural block diagram of a reasoning result determination device provided by an embodiment of the present application. This device 1300 is set in the first reasoning instance. During the update of the reasoning service, this device 1300 includes:

[0120] A first receiving module 1301, configured to receive a target request sent by a Prompt instance; the target request is obtained by the Prompt instance splicing the target Prompt template and the request data carried in the reasoning request. The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the reasoning request and the mapping relationship between each model version identifier and different Prompt templates. The reasoning request is marked by the gateway for multiple user requests received by using the model version identifier according to the quantity proportion corresponding to each model version identifier. The quantity proportion is the proportion sent by the registration center received by the gateway. The quantity proportion is determined by the registration center according to the ratio of the number of reasoning instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of reasoning instances in each instance set, and the reasoning instances in the instance set are in a healthy state.

[0121] A first determination module 1302, configured to determine, according to the model version identifier in the target request, a target reasoning instance corresponding to the model version identifier in the target request from multiple reasoning instances corresponding to the reasoning service, so as to determine a reasoning result based on the target request through the target reasoning instance.

[0122] In one embodiment, the first determination module 1302 includes:

[0123] A first determination unit, configured to determine a model version identifier corresponding to a first inference instance according to topology information of a plurality of inference instances corresponding to an inference service; the topology information includes a model version identifier and an address identifier corresponding to each inference instance;

[0124] A second determination unit, configured to compare the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result;

[0125] A third determination unit, configured to determine a target inference instance corresponding to the model version identifier in the target request from a plurality of inference instances corresponding to the inference service according to the comparison result.

[0126] In one embodiment, the third determination unit is specifically configured to, if the comparison result is that the model version identifier in the target request is the same as the model version identifier corresponding to the first inference instance, use the first inference instance as the target inference instance.

[0127] If the comparison result is that the model version identifier in the target request is different from the model version identifier corresponding to the first inference instance, then according to the model version identifiers corresponding to the inference instances in the topology information, determine a second inference instance from each inference instance, and use the second inference instance as the target inference instance;

[0128] wherein, the second inference instance is an inference instance corresponding to the target model version identifier, and the target model version identifier is the model version identifier that is the same as the model version identifier in the target request among the model version identifiers in the topology information.

[0129] In an exemplary embodiment, an inference result determination device is provided. During the update process of the inference service, the device includes:

[0130] A calculation module, configured to calculate the proportion of the number of instances corresponding to each model version identifier to the total number of instances according to the number of inference instances in the instance set corresponding to each model version identifier; the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are in a healthy state.

[0131] The first sending module is used to send the quantity proportion corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance; the quantity proportion is used for the gateway to mark the received multiple user requests according to each quantity proportion by using the model version identifier to obtain the corresponding inference requests, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier and the mapping relationship corresponding to the inference request, and splice the target Prompt template and the request data carried in the inference request to obtain the target request, so as to send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service based on the model version identifier in the target request, so as to determine the inference result based on the target request through the target inference instance.

[0132] In an exemplary embodiment, the device may further include:

[0133] The second receiving module is used to receive the registration information sent by multiple inference instances corresponding to the inference service, and register the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and send the topology information to each inference instance; the registration information includes the model version identifier and the address identifier corresponding to the inference instance.

[0134] The calling module calls the health check interface provided by the inference service to perform a health check on the registered inference instances to obtain the status of the registered inference instances.

[0135] The joining module is used to add the registered inference instance to the instance set corresponding to the corresponding model version identifier if the status of the registered inference instance is a healthy state and the registered inference instance does not exist in the instance set corresponding to the model version identifier.

[0136] In an exemplary embodiment, another inference result determination device is provided. During the update process of the inference service, the device includes:

[0137] The third receiving module is used to receive the quantity proportion corresponding to multiple model version identifiers sent by the registration center; the quantity proportion is determined by the registration center according to the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are in a healthy state.

[0138] A second sending module, configured to mark multiple user requests with model version identifiers according to the quantity proportion corresponding to each model version identifier to obtain corresponding inference requests, and send the inference requests to a Prompt instance; the inference requests are used for the Prompt instance to determine a target Prompt template corresponding to the inference requests based on the model version identifier corresponding to the inference requests and the mapping relationship between each model version identifier and different Prompt templates, and splice the target Prompt template and the request data carried in the inference requests to obtain a target request, and send the target request to a first inference instance, and the target request is used for the first inference instance to determine a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

[0139] In an exemplary embodiment, another inference result determination device is provided. During the update process of the inference service, the device includes:

[0140] A second determination module, configured to determine a target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates; the inference request is obtained by the gateway marking multiple received user requests with model version identifiers according to the quantity proportion corresponding to each model version identifier, the quantity proportion is the proportion sent by the registration center received by the gateway, the quantity proportion is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state;

[0141] A splicing module, configured to splice the target Prompt template and the request data carried in the inference request to obtain a target request;

[0142] A third sending module, configured to send the target request to the first inference instance; the target request is used for the first inference instance to determine a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

[0143] Each module in the above inference result determination device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of the processor, or be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0144] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in Figure 14 . The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for determining an inference result.

[0145] Those skilled in the art can understand that Figure 14 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0146] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the technical solution in any of the above method embodiments. The implementation principle and technical effect are similar and will not be described in detail here.

[0147] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the technical solution in any of the above method embodiments. The implementation principle and technical effect are similar and will not be described in detail here.

[0148] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the technical solution in any of the above method embodiments. The implementation principle and technical effect are similar and will not be described in detail here.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0150] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0151] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0152] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for determining an inference result, characterized in that The method is applied to a first inference instance. During the update of the inference service, the method includes: Receiving a target request sent by a Prompt instance; the target request is obtained by the Prompt instance by splicing a target Prompt template and request data carried in an inference request. The target Prompt template is determined by the Prompt instance based on the model version identifier corresponding to the inference request and the mapping relationship between each model version identifier and different Prompt templates. The inference request is marked by the gateway for multiple received user requests using the model version identifier according to the quantity ratio corresponding to each model version identifier. The quantity ratio is the ratio sent by the registration center received by the gateway. The quantity ratio is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances. The total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state; Determining, according to the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

2. The method according to claim 1, wherein The determining, according to the model version identifier in the target request, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service includes: Determining the model version identifier corresponding to the first inference instance according to the topology information of multiple inference instances corresponding to the inference service; the topology information includes the model version identifier and address identifier corresponding to each inference instance; Comparing the model version identifier in the target request with the model version identifier corresponding to the first inference instance to obtain a comparison result; Determining, according to the comparison result, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service.

3. The method according to claim 2, wherein The determining, according to the comparison result, a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service includes: If the comparison result is that the model version identifier in the target request is the same as the model version identifier corresponding to the first inference instance, then using the first inference instance as the target inference instance; If the comparison result is that the model version identifier in the target request is different from the model version identifier corresponding to the first inference instance, then determining a second inference instance from each inference instance according to the model version identifier corresponding to each inference instance in the topology information, and using the second inference instance as the target inference instance; Wherein, the second inference instance is an inference instance corresponding to a target model version identifier, and the target model version identifier is the model version identifier that is the same as the model version identifier in the target request among each model version identifier in the topology information.

4. A method for determining an inference result, characterized in that The method is applied to a registration center. During the update of the inference service, the method includes: Calculate the proportion of the number of instances corresponding to each model version identifier to the total number of instances based on the number of inference instances in the instance set corresponding to each model version identifier; the total number of instances is the total number of inference instances in each of the instance sets, and the inference instances in the instance set are inference instances in a healthy state. Send the proportion corresponding to each model version identifier to the gateway, and send the mapping relationship between each model version identifier and different Prompt templates to the Prompt instance; the proportion is used for the gateway to mark multiple user requests received based on each proportion using the model version identifier to obtain corresponding inference requests, and send each inference request to the Prompt instance according to the routing rule. The inference request is used for the Prompt instance to determine the target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationship, and splice the target Prompt template and the request data carried in the inference request to obtain a target request, and send the target request to the first inference instance. The target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine the inference result based on the target request through the target inference instance.

5. The method according to claim 4, wherein The method further includes: Receive the registration information sent by multiple inference instances corresponding to the inference service, and register the inference instances based on the registration information of each inference instance to generate topology information corresponding to the multiple inference instances, and send the topology information to each inference instance; the registration information includes the model version identifier and address identifier corresponding to the inference instance. Call the health check interface provided by the inference service to perform a health check on the registered inference instances to obtain the status of the registered inference instances. If the status of the registered inference instance is a healthy state and the registered inference instance does not exist in the instance set corresponding to the model version identifier of the registered inference instance, add the registered inference instance to the instance set corresponding to the corresponding model version identifier.

6. A method for determining an inference result, characterized in that The method is applied to the gateway. During the update of the inference service, the method includes: Receive the proportion corresponding to multiple model version identifiers sent by the registration center; the proportion is determined by the registration center according to the ratio of the number of inference instances in the instance set corresponding to each model version identifier to the total number of inference instances in each instance set, and the inference instances in the instance set are instances in a healthy state. According to the quantity proportion corresponding to each model version identifier, use the model version identifier to mark multiple user requests to obtain corresponding inference requests, and send the inference requests to the Prompt instance; the inference requests are used for the Prompt instance to determine the target Prompt template corresponding to the inference requests based on the model version identifier corresponding to the inference requests and the mapping relationships between the model version identifiers and different Prompt templates, and splice the target Prompt template and the request data carried in the inference requests to obtain a target request, and send the target request to the first inference instance, and the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

7. A method for determining an inference result, characterized in that, The method is applied to a Prompt instance, and the method includes: Determine a target Prompt template based on the model version identifier corresponding to the inference request and the mapping relationships between the model version identifiers and different Prompt templates; the inference request is obtained by the gateway marking multiple received user requests using the model version identifier according to the quantity proportion corresponding to each model version identifier, the quantity proportion is the proportion sent by the registration center received by the gateway, the quantity proportion is determined by the registration center according to the ratio of the number of inference instances in the instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are healthy inference instances; Splice the target Prompt template and the request data carried in the inference request to obtain a target request; Send the target request to the first inference instance; the target request is used for the first inference instance to determine the target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to the inference service, so as to determine an inference result based on the target request through the target inference instance.

8. An inference result determination device, characterized in that, The device is arranged in the first inference instance, and during the update process of the inference service, the device includes: A first receiving module, configured to receive a target request sent by a Prompt instance; the target request is obtained by the Prompt instance by splicing a target Prompt template and request data carried in an inference request, the target Prompt template is determined by the Prompt instance based on a model version identifier corresponding to the inference request and a mapping relationship between each model version identifier and different Prompt templates, the inference request is obtained by the gateway marking multiple received user requests by using the model version identifier according to a quantity proportion corresponding to each model version identifier, the quantity proportion is the proportion sent by a registration center received by the gateway, the quantity proportion is determined by the registration center according to a ratio of the number of inference instances in an instance set under each model version identifier to the total number of instances, the total number of instances is the total number of inference instances in each instance set, and the inference instances in the instance set are inference instances in a healthy state; A determination module, configured to determine a target inference instance corresponding to the model version identifier in the target request from multiple inference instances corresponding to an inference service, so as to determine an inference result based on the target request by the target inference instance.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images