Multi-source large model service standardized management and control system and method

Abstract

The invention belongs to the technical field of large model management and control, and particularly discloses a multi-source large model service standardization management and control system and method, and the system comprises a standardization layer which is used for carrying out the standardization processing of an original message sequence, and obtaining a standard message sequence which is arranged according to a standard sequence; and the model adaptation layer is used for converting the standard message sequence into request bodies corresponding to different types of large models as input, converting response bodies output by the different types of large models into standard response bodies and feeding back the standard response bodies to a user. The invention provides a unified management and control technology during calling of a multi-source large model service, and format standardization and exception elimination of a user request message flow are completed through a standardization layer; standardization of a large model service interface is completed through the model adaptation layer, and unification of a multi-source large model calling request format, usage statistics and exception handling is ensured.

Description

Technical Field

[0001] The present application belongs to the field of large model management and control technology, and more specifically, relates to a multi-source large model service standardization management and control system and method. Background Art

[0002] Large model providers provide trained large machine learning models (such as GPT, BERT, etc.) to users in the form of services. Large model services have the advantages of high efficiency, ease of use, and scalability. However, when users call large models, there are the following defects:

[0003] On the one hand, since each model provider uses different technical architectures and design solutions when developing large models, the service request formats they provide are different. This diversity brings inconvenience to users, and requires specific adaptation and calling of the services of each model provider.

[0004] On the other hand, in addition to the differences in service request formats, the calling interfaces of various model providers are also not unified. This includes interface parameters, return values, error handling, etc. Users need to spend a lot of time and energy to familiarize themselves with and master the calling interfaces of different model providers, which undoubtedly increases the cost of use.

[0005] On the other hand, there is still a lack of unified standards and specifications for large model calls. This leads to the lack of unified technical requirements and evaluation standards for various model providers when providing services, making it difficult to ensure the quality and stability of services. At the same time, it also brings difficulties to users in choosing and using.

[0006] Finally, due to the inconsistency of service request formats and calling interfaces, large models from various model providers face challenges in integration and interoperability, which limits the collaborative work and data sharing between different models and reduces the application efficiency and value of large models.

[0007] In summary, the current large model calls have problems such as diversity, inconsistency, and lack of standards, which restrict the development of large model services. Summary of the invention

[0008] In view of the defects existing in the related technologies, the embodiments of the present application provide a multi-source large model service standardized management and control system and method, aiming to solve the problem of lack of standards for calling multi-source large model services.

[0009] In a first aspect, an embodiment of the present application provides a multi-source large model service standardization management and control system, including:

[0010] The standardization layer is used to standardize the original message sequence and obtain a standard message sequence arranged in a standard order;

[0011] The model adaptation layer is used to convert the standard message sequence into the request body corresponding to different types of large models as input, and convert the response body output by different types of large models into a standard response body and then feed it back to the user.

[0012] In some embodiments, obtaining a standard message sequence arranged in a standard order includes:

[0013] Determine the message role in the original message sequence, the message role is a system message, a user message or an assistant message;

[0014] Merge, reorder, and filter messages based on their roles to obtain a standard message sequence.

[0015] In some embodiments, message merging, reordering, and filtering are performed based on message roles, including:

[0016] Check whether there is a system prompt in the original message sequence. If not, add the system prompt to the sequence head to obtain the message sequence containing the system prompt.

[0017] Detect whether there are multiple system prompts. If so, merge the multiple system prompts into one, and obtain a message sequence containing one system prompt.

[0018] Detect the order of different role messages in a message sequence containing a system prompt. If they are not arranged in a standard order, re-arrange them and perform at least one of the following operations to obtain a standard message sequence:

[0019] Merge consecutive Assistant messages;

[0020] Merge consecutive user messages;

[0021] Filter out assistant messages at the end of the sequence;

[0022] The standard sequence is: starting with a system prompt, followed by a combination of one or more assistant messages and user messages.

[0023] In some embodiments, the system also includes a business enabling layer for replacing, splicing or optimizing the system prompts in the header of the standard message sequence into customized system prompts according to business requirements.

[0024] In some embodiments, the model adaptation layer includes a model adapter and a standard input and output interface. The model adapter includes a request adapter and a response adapter. The request adapter is used to convert a standard message sequence into a request body corresponding to different types of large models as input. The response adapter is used to convert the response body output by different types of large models into a standard response body and then feed it back to the user. The standard message sequence initiates a large model call request through the standard input and output interface, and the standard response body is fed back to the user through the standard input and output interface.

[0025] In some embodiments, the model adapter further comprises:

[0026] The exception adapter is used to convert the exception response body fed back by the response adapter into a standard exception response body and then feed it back to the user.

[0027] In some embodiments, the model adapter further comprises:

[0028] The resource consumption adapter is used to perform unified billing measurement on the large model resources consumed by the standard message sequence request and feed back the unified billing measurement results to the user.

[0029] In some embodiments, the system further comprises:

[0030] The resource management and control layer is used to manage large model resources and user resources.

[0031] In some embodiments, large model resources and user resources are managed and controlled, including:

[0032] Register large model resources provided by different model providers into the resource pool;

[0033] The large model resources are coordinated from the resource pool based on the request flow of the large model resources uploaded by the model adaptation layer and the remaining status of the large model resources in the resource pool, and the large model resources consumed by the request are deducted.

[0034] In some embodiments, large model resources and user resources are managed and controlled, including:

[0035] Verify the model availability based on the standard exception response body uploaded by the model adaptation layer;

[0036] Based on the unified billing measurement results uploaded by the model adaptation layer, the user resource balance is managed and controlled. In the second aspect, the embodiment of the present application also provides a standardized management method for multi-source large model services, including:

[0037] Standardize the original message sequence to obtain a standard message sequence arranged in a standard order;

[0038] The standard message sequence is converted into a request body corresponding to different types of large models as input, and the response bodies output by different types of large models are converted into a standard response body and then fed back to the user.

[0039] The multi-source big model service standardized management and control system and method provided in the embodiment of the present application provide a unified management and control technology for calling multi-source big model services, and completes the format standardization and exception elimination of the user request message flow through the standardization layer; completes the customized design of the system prompts through the business empowerment layer, and realizes business empowerment; completes the standardization of the big model service interface through the model adaptation layer, ensuring the unification of the multi-source big model call request format, usage statistics and exception handling; and realizes the unified management and control of big model resources and user resources through the resource management and control layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a schematic diagram of the architecture of the multi-source large model service standardization management and control system provided in the embodiment of the present application;

[0042] Figure 2 It is a schematic diagram of the process of standardization processing performed by the standardization layer provided in the embodiment of the present application;

[0043] Figure 3 It is a schematic diagram of the architecture of the model adaptation layer provided in an embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of the process of registering AI resources at the resource management and control layer provided in an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of the process of allocating user resources by the resource management and control layer provided in an embodiment of the present application;

[0046] Figure 6 It is a flow chart of the standardized management and control method of multi-source large model services provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Figure 1 Schematic diagram of the architecture of the multi-source large model service standardization management and control system provided in the embodiment of the present application, such as Figure 1 As shown, the system at least includes: a standardization layer and a model adaptation layer.

[0049] The standardization layer is used to standardize the original message sequence and obtain a standard message sequence arranged in a standard order. The model adaptation layer is used to convert the standard message sequence into a request body corresponding to different types of large models as input, and convert the response body output by different types of large models into a standard response body and then feed it back to the user.

[0050] The standardization layer ensures the stability of the overall management and control system, and acts as a message filter and format standardizer.

[0051] In some embodiments, the standardization layer performs standardization processing on the original message sequence to obtain a standard message sequence arranged in a standard order, specifically including:

[0052] Determine the message role in the original message sequence, the message role is a system prompt, a user message or an assistant message;

[0053] Merge, reorder, and filter messages based on their roles to obtain a standard message sequence.

[0054] Specifically, during the big model call process, to communicate with the big model, one or more messages need to be provided. Each message must be associated with a role in the system, user, or assistant. The big model can understand how to behave and the subject that initiated the call based on the role of the message.

[0055] System messages help create context or scope for a conversation by assigning specific behavior to a larger model. Passing system messages is optional but helps set model behavior for the conversation within the system.

[0056] User messages refer to messages coming from the client, that is, messages or prompts initiated by users, which are generally model call requests initiated by users.

[0057] Assistant messages represent entities that respond to end-user messages and are used to set the model's previous response in the current request to maintain conversation coherence.

[0058] Figure 2 is a schematic diagram of a process of performing standardization processing at the standardization layer provided in an embodiment of the present application, such as Figure 2 As shown, messages are merged, reordered, and filtered according to their roles, including:

[0059] Check whether there is a system prompt in the original message sequence. If not, add the system prompt to the sequence head to obtain the message sequence containing the system prompt; if it exists, directly obtain the message sequence containing the system prompt.

[0060] In natural language processing, a prompt is an input text or question used to guide or prompt the model to generate effects. Its purpose is to clearly indicate the type of output required by the model and the expected answer method. By providing a clear and specific prompt, a large model can be guided to generate answers related to a specific business. System prompts are strategies set by large model providers to improve user experience. Generally, there is only one in a session, and it will have a lasting impact on the model's response throughout the session. It is a type of system message.

[0061] Furthermore, it is detected whether there are multiple system prompts in the message sequence containing the system prompt. If so, the multiple system prompts are merged into one to obtain a message sequence containing only one system prompt; if not, the message sequence containing only one system prompt is directly obtained.

[0062] Furthermore, the arrangement order of the messages of different roles in the message sequence containing only one system prompt is detected. If they are arranged in a standard order, the message sequence with the correct order is directly obtained; if they are not arranged in a standard order, at least one of the following operations is performed after re-arranging:

[0063] Merge consecutive Assistant messages;

[0064] Merge consecutive user messages;

[0065] Filter out assistant messages at the end of the sequence.

[0066] Then, a standard message sequence arranged in a standard order is obtained.

[0067] Specifically, the standard sequence is: starting with a system prompt, followed by a combination of one or more assistant messages and user messages. If A, B, and C are used to identify system prompts, assistant messages, and user messages, respectively, the standard message sequence is ABC...BC.

[0068] When the messages of different roles in a message sequence containing only one system prompt are not arranged in the standard order, they are reordered. In addition, on the one hand, it is detected whether there are continuously repeated assistant messages or user messages, that is, whether the message sequence contains message fragments arranged like BC...CB or CB...BC. If so, the continuously repeated assistant messages or user messages are merged.

[0069] On the other hand, it is detected whether the end of the sequence is a user message. If it is not a user message, that is, there is a non-user message (mainly assistant message), the assistant message at the end of the sequence is filtered out. Finally, a standard message sequence arranged in a standard order is obtained.

[0070] The standardization layer can detect problems such as message role confusion and message role duplication in user messages, reorder and merge user messages, and filter out non-user messages at the end of the sequence, thereby achieving format standardization and exception elimination of user request message flows, eliminating exceptions in advance for subsequent processes.

[0071] Optionally, the system also includes a business enabling layer for replacing, splicing or optimizing the system prompts at the header of the standard message sequence output by the standardization layer into customized system prompts according to business requirements.

[0072] The business enablement layer provides scalability for the management and control system and plays a role in business enablement. Figure 1 As shown in the figure, the business enablement layer can flexibly combine, replace, and optimize system prompts in requests according to business needs. By customizing system prompts, subsequent large models can understand the intent of the request more accurately, and can enhance model capabilities (for example, by defining the function format in the system prompt, the model service that originally did not have the function call function has the basic function call function), which greatly improves the availability of the model service and provides a flexible calling method for businesses such as intelligent agents.

[0073] Take combination / splicing as an example:

[0074] Initial system prompt: You are a translation tool that can translate any language into Chinese.

[0075] Combined system prompt: You are a translation tool that can translate any language into Chinese; when translating, only provide the translation, and no other response content is required.

[0076] By splicing the initial system prompts into customized system prompts, the response experience of large models can be improved, with only the translation replied without any other superfluous content.

[0077] Take replacement as an example:

[0078] Initial system prompt: You are my AI assistant.

[0079] The replaced system prompt: You are a translation tool that can translate any language into Chinese; when translating, only provide the translation, and no other response content is required.

[0080] By replacing the initial system prompts with customized system prompts, the system prompts can be made specific according to user needs, allowing the big model to provide more accurate service responses to users.

[0081] The model adaptation layer provides flexibility for the management and control system and completes the standardization of large model service interfaces.

[0082] like Figure 1 As shown, the model adaptation layer includes a model adapter, which can integrate multiple large model services of different types from different vendors (e.g. Figure 1 The inputs and outputs of the big models (such as Zhipu, Baidu, and OpenAI) shown in the figure are uniformly converted into standard defined inputs and outputs. Specifically, on the one hand, the standard message sequence based on the standardized processing of the user's call request is converted into the request body corresponding to different types of big models as the input of each big model. On the other hand, the response body output by each big model is converted into a standard response body and then fed back to the user, thereby supporting flexible access to large language model services of multiple manufacturers and types, and ensuring the unification of call request formats, usage statistics, and exception handling of multiple model providers.

[0083] Figure 3 is a schematic diagram of the architecture of the model adaptation layer provided in the embodiment of the present application, such as Figure 3 As shown, the model adapter includes at least a request adapter, which is used to convert a standard message sequence into a request body corresponding to different types of large models as input, and a response adapter which is used to convert the response body output by different types of large models into a standard response body and then feed it back to the user.

[0084] In addition, the model adaptation layer also defines a set of standard input and output interfaces. Users who want to request to call the big model service must initiate the request through this interface. Similarly, the management and control system will also return the results to the user in a standard format. Specifically, the standard message sequence must initiate the big model call request through the standard input and output interface, and the standard response must be fed back to the user through the standard input and output interface, thereby further ensuring the uniformity of the call request format of different model providers.

[0085] Optionally, the model adapter also includes an exception adapter. When an exception occurs in the model call, the response adapter will send the exception response body to the exception adapter, which will convert it into a standard response body and then feed it back to the user and the resource management layer of the system. The model call exception can be an exception in the model call process, such as a call failure, or a model output exception, such as an obvious abnormal output result, a failure in the output result format conversion, etc.

[0086] Optionally, the model adapter also includes a resource consumption adapter, which is used to perform unified billing measurement on the large model resources consumed by the standard message sequence request, and feed back the unified billing measurement result to the user. As an intermediary between the user and the large model service, the model adapter performs unified billing measurement for large model services with different charging standards, implements a unified billing strategy, and improves user experience.

[0087] The essence of each adapter is a conversion function, among which the conversion functions corresponding to the request adapter and the response adapter are the core. The conversion function corresponding to the request adapter is used to convert the format of the input request: convert the standard request body initiated by the user into the image request body of each model provider; the conversion function corresponding to the response adapter is used to convert the format of the output response: convert each model provider (for example Figure 1 The response body returned by the examples of Zhipu, Baidu, OpenAI, etc. is converted into a standard response body and returned to the user. This ensures that no matter which model provider the user requests, the user's input and output formats are unified.

[0088] The resource management and control layer provides the management and control system with the ability to manage large model resources and user resources.

[0089] On the one hand, the large model resources provided by different model providers are registered in the resource pool. On the other hand, when the user initiates a request, the large model resources are coordinated from the resource pool based on the request flow of the large model resources uploaded by the model adaptation layer and the remaining large model resources in the resource pool, and the large model resources consumed by the request are deducted.

[0090] In the latter process, model availability verification is involved. When a large model calls an exception, the exception adapter not only feeds back the standard exception response body to the user, but also uploads it to the resource management layer. The resource management layer needs to verify the model availability based on the standard exception response body. When it comes to user resource balance query, the resource consumption adapter not only feeds back the unified billing measurement results to the user, but also uploads it to the resource management layer. The resource management layer needs to query the user's resource balance. When the user's balance resources are sufficient, the called resources are deducted. When the user's balance resources are insufficient, feedback is given to the user to remind the user to perform recharge and other operations.

[0091] Figure 4 : is a schematic diagram of the process of registering AI resources at the resource management and control layer provided in the embodiment of the present application, such as Figure 4 As shown in the figure, taking the registration of Baidu cloud service resources as an example, the administrator registers Baidu AI resources to the AI ​​resource pool through the resource management layer.

[0092] The resource management and control layer designs AI resource entities, stores request Uniform Resource Locator (URL), API key, large model resource balance information, etc. in the database, and implements AI resource registration and renewal interfaces for administrators to manage AI resource pools from the console front end.

[0093] Figure 5 is a schematic diagram of a process of allocating user resources by the resource management and control layer provided in an embodiment of the present application, such as Figure 5As shown in the figure, taking the provision of Baidu Cloud services to users as an example, the resource pools of each user are maintained. When a user initiates a request, the resource management layer coordinates the Baidu Cloud resources in the AI ​​resource pool to initiate the request based on the traffic and surplus of Baidu Cloud AI resources, and deducts the Baidu Cloud resources consumed by the request from the resource pool. Among them, the search traffic of resource calls is measured by queries per second (QPS).

[0094] Figure 6 : is a flow chart of a standardized management and control method for multi-source large model services provided in an embodiment of the present application, such as Figure 6 As shown, the method comprises at least the following steps:

[0095] S601. Standardize the original message sequence to obtain a standard message sequence arranged in a standard order;

[0096] S602. Convert the standard message sequence into request bodies corresponding to different types of large models as input, and convert the response bodies output by different types of large models into standard response bodies and feed them back to the user.

[0097] In some embodiments, S601 specifically includes:

[0098] Determine the message role in the original message sequence, the message role is a system message, a user message or an assistant message;

[0099] Merge, reorder, and filter messages based on their roles to obtain a standard message sequence.

[0100] In some embodiments, message merging, reordering, and filtering are performed according to message roles, specifically including:

[0101] Check whether there is a system prompt in the original message sequence. If not, add the system prompt to the sequence head to obtain the message sequence containing the system prompt.

[0102] Detect whether there are multiple system prompts. If so, merge the multiple system prompts into one, and obtain a message sequence containing one system prompt.

[0103] Detect the order of different role messages in a message sequence containing a system prompt. If they are not arranged in a standard order, re-arrange them and perform at least one of the following operations to obtain a standard message sequence:

[0104] Merge consecutive Assistant messages;

[0105] Merge consecutive user messages;

[0106] Filter out assistant messages at the end of the sequence;

[0107] The standard sequence is: starting with a system prompt, followed by a combination of one or more assistant messages and user messages.

[0108] In some embodiments, the method further comprises:

[0109] Replace, splice or optimize the system prompts in the header of the standard message sequence into customized system prompts based on business needs.

[0110] In some embodiments, the method further comprises:

[0111] Convert the exception response body into a standard exception response body and then feed it back to the user;

[0112] The large model resources consumed by the standard message sequence request are uniformly billed and measured, and the uniform billing and measurement results are fed back to the user.

[0113] In some embodiments, the method further comprises:

[0114] Manage and control large model resources and user resources.

[0115] In some embodiments, large model resources and user resources are managed and controlled, including:

[0116] Register large model resources provided by different model providers into the resource pool;

[0117] The large model resources are coordinated from the resource pool based on the request flow of the large model resources uploaded by the model adaptation layer and the remaining status of the large model resources in the resource pool, and the large model resources consumed by the request are deducted.

[0118] In some embodiments, large model resources and user resources are managed and controlled, including:

[0119] Verify the model availability based on the standard abnormal response body;

[0120] Based on the unified billing measurement results, user resource balances are managed and controlled.

[0121] It should be understood that the above method is executed through the above system, and the corresponding process steps in the method, their implementation principles and technical effects are similar to those described in the above system. The implementation process of the method can refer to the corresponding modules in the above system, which will not be repeated here.

[0122] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multi-source large model service standardization management and control system, characterized in that: include: The standardization layer is used to standardize the original message sequence and obtain a standard message sequence arranged in a standard order; The model adaptation layer is used to convert the standard message sequence into a request body corresponding to different types of large models as input, and convert the response body output by different types of large models into a standard response body and then feed it back to the user.

2. The multi-source large model service standardization management and control system according to claim 1 is characterized in that: The step of obtaining a standard message sequence arranged in a standard order includes: Determining the message role in the original message sequence, the message role being a system message, a user message or an assistant message; Messages are merged, reordered, and filtered according to message roles to obtain the standard message sequence.

3. The multi-source large model service standardization management and control system according to claim 2 is characterized in that: The message merging, reordering and filtering according to the message roles include: Detect whether there is a system prompt in the original message sequence, and if not, add the system prompt to the sequence head to obtain a message sequence containing the system prompt; Detect whether there are multiple system prompts. If so, merge the multiple system prompts into one, and obtain a message sequence containing one system prompt. Detect the arrangement order of different role messages in the message sequence containing a system prompt, and if they are not arranged in the standard order, re-arrange them and perform at least one of the following operations to obtain the standard message sequence: Merge consecutive Assistant messages; Merge consecutive user messages; Filter out assistant messages at the end of the sequence; The standard sequence is: starting with a system prompt, followed by a combination of one or more assistant messages and user messages.

4. The multi-source large model service standardization management and control system according to claim 3 is characterized in that: The system also includes a business enabling layer, which is used to replace, splice or optimize the system prompts in the header of the standard message sequence into customized system prompts according to business needs.

5. The multi-source large model service standardization management and control system according to claim 1 is characterized in that: The model adaptation layer includes a model adapter and a standard input and output interface. The model adapter includes a request adapter and a response adapter. The request adapter is used to convert the standard message sequence into a request body corresponding to different types of large models as input, and the response adapter is used to convert the response body output by different types of large models into a standard response body and then feed it back to the user. The standard message sequence initiates a large model call request through the standard input and output interface, and the standard response body is fed back to the user through the standard input and output interface.

6. The multi-source large model service standardization management and control system according to claim 5 is characterized in that: The model adapter also includes: An exception adapter, used for converting the exception response body fed back by the response adapter into a standard exception response body and then feeding it back to the user; The resource consumption adapter is used to perform unified billing measurement on the large model resources consumed by the standard message sequence request and feed back the unified billing measurement result to the user.

7. The multi-source large model service standardization management and control system according to claim 5 is characterized in that: The system further comprises: The resource management and control layer is used to manage large model resources and user resources.

8. The multi-source large model service standardization management and control system according to claim 7 is characterized in that: The management and control of large model resources and user resources includes: Register large model resources provided by different model providers into the resource pool; The large model resources are coordinated from the resource pool based on the request flow of the large model resources uploaded by the model adaptation layer and the remaining status of the large model resources in the resource pool, and the large model resources consumed by the request are deducted.

9. The multi-source large model service standardization management and control system according to claim 7 is characterized in that: The management and control of large model resources and user resources includes: Verifying the model availability based on the standard exception response body uploaded by the model adaptation layer; Based on the unified billing measurement results uploaded by the model adaptation layer, the user resource balance is managed and controlled.

10. A method for standardizing and controlling multi-source large model services, characterized in that: Applied to the multi-source large model service standardization management and control system according to any one of claims 1 to 9, the method comprises: Standardize the original message sequence to obtain a standard message sequence arranged in a standard order; The standard message sequence is converted into a request body corresponding to different types of large models as input, and the response bodies output by different types of large models are converted into a standard response body and then fed back to the user.

Images