Large language model management system

Abstract

The invention discloses a management system for a large language model, and the system comprises a configuration module which is used for responding to a user triggering operation, obtaining configuration information based on the user triggering operation, and transmitting the configuration information to at least one cluster; wherein the configuration information comprises deployment information and model information of the large language model; the cluster is used for receiving the configuration information and performing configuration processing on the configuration information based on the contained engine service to obtain a corresponding large language model; the log query module is used for receiving a query request so as to query the call record information in the at least one cluster based on the query request. According to the system, through management operation on multiple clusters, the flexibility and adaptability of multi-scene reasoning service application of a large language model are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a management system for large language models. Background Art

[0002] Since automotive finance business involves multiple scenarios and data sources, to ensure the security of data finance data, inference services often need to be collaboratively managed across at least one cluster and environment.

[0003] However, the current collaborative management technical means have great complexity in communication and resource scheduling among multiple clusters, making it difficult to achieve efficient resource utilization and guarantee of inference performance. At the same time, the isolation and deployment strategies of multiple environments lack a unified management plan, which easily leads to resource conflicts between environments and unreliable inference results. Summary of the Invention

[0004] The present invention provides a management system for large language models, which realizes the flexibility and adaptability of multi-scenario inference service applications of large language models through management operations on multiple clusters.

[0005] According to one aspect of the present invention, there is provided a management system for large language models, the management system for large language models includes a configuration module, at least one cluster, and a log query module: the configuration module is communicatively connected to at least one cluster through a data operation service, and the at least one cluster is communicatively connected to the log query module through a network;

[0006] The configuration module is configured to, in response to a user trigger operation, obtain configuration information based on the user trigger operation, and send the configuration information to at least one cluster; wherein, the configuration information includes deployment information and model information of a large language model;

[0007] The cluster is configured to receive the configuration information, and obtain a corresponding large language model after performing configuration processing on the configuration information based on the included engine service;

[0008] The log query module is configured to receive a query request, and query call record information in the at least one cluster based on the query request.

[0009] Optionally, the configuration module includes a model repository unit and an application management unit;

[0010] The model repository unit is configured to provide large language models of multiple functional types in response to a user trigger operation;

[0011] The application management unit is configured to select corresponding deployment information in response to a user trigger operation.

[0012] Optionally, the cluster includes a target environment library;

[0013] The configuration module is used to deploy the large language model to the at least one cluster and the corresponding target environment library.

[0014] Optionally, the target environment library includes a production environment library and / or a test environment library.

[0015] Optionally, the configuration module is used to publish the large language model to the test environment library for testing and verification, and deploy the large language model that passes the test and verification to the production environment for production use.

[0016] Optionally, the cluster includes an engine service and a model inference service;

[0017] The engine service unit is used to load the application information of the large language model published to the target environment library into the model inference service unit at a set time interval;

[0018] The model inference service is used to perform a registration operation on the service information of the large language model to obtain the registration information of the large language model.

[0019] Optionally, the cluster includes an application scheduling service unit;

[0020] The engine service unit is used to load the application information of the large language model published to the target environment library into the application scheduling service unit at a set time interval;

[0021] The application scheduling service unit parses the application configuration information of the large language model to call the model inference service unit according to the application configuration information and the registration information.

[0022] Optionally, the model inference service unit is further used to provide an interface for the large language model application inference service to the outside, so that users can call the large language model through the interface for use.

[0023] Optionally, the cluster includes a log push unit;

[0024] The log push unit is used to push the execution records of the model inference service unit and the application scheduling service unit to the message queue.

[0025] Optionally, the cluster further includes a log parsing unit;

[0026] The log parsing unit is used to parse the execution records in the message queue and save them to the log library of the cluster for the log query module to view.

[0027] In the technical solution of the embodiment of the present invention, a configuration module is used to respond to a user trigger operation, obtain configuration information based on the user trigger operation, and send the configuration information to at least one cluster; wherein, the configuration information includes the deployment information and model information of a large language model; the cluster is used to receive the configuration information and obtain the corresponding large language model after performing configuration processing on the configuration information based on the included engine service; the log query module is used to receive a query request to query the call record information in the at least one cluster based on the query request. Through the management operation of multiple clusters, the present system realizes the flexibility and adaptability of the multi-scenario inference service application of the large language model.

[0028] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 FIG. 1 is a schematic structural diagram of a management system for a large language model according to Embodiment 1 of the present invention;

[0031] Figure 2 FIG. 2 is a structural example diagram of a management system for a large language model according to Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that the terms "target", "original", etc. in the description, claims and the above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 is a schematic structural diagram of a management system for a large language model provided according to Embodiment 1 of the present invention, as Figure 1 shown. The management system of the large language model includes a configuration module 110, at least one cluster 120, and a log query module 130; the configuration module is communicatively connected to at least one cluster through a data operation service, and the at least one cluster is communicatively connected to the log query module through a network; wherein, the data operation service may refer to a communication mechanism or service based on a set, which is responsible for processing data transmission and interaction between the large language model and each cluster. In this embodiment, the data operation service may refer to implementing communication between the configuration module and at least one cluster through a set communication interface. Exemplarily, the data operation service may be an op service. The configuration module in this embodiment may be presented to the user in a cloud page configuration manner, so that the user can select corresponding configuration content through a trigger operation. For example, information such as the deployment location of the large language model, the specific objects to be deployed, and the storage location can be selected and determined based on the configuration information. The large language model in this embodiment may be a large language model for the automotive finance risk control scenario.

[0036] The configuration module 110 is configured to, in response to a user trigger operation, obtain configuration information based on the user trigger operation, and send the configuration information to at least one cluster.

[0037] Among them, the configuration information includes deployment information and model information of the large language model; the deployment information may include information such as deployment objects and specific deployment locations. The deployment objects may be objects such as deployment specified environments and clusters. The deployment location may be the location where the large language model finally needs to be deployed. The model information may refer to the location information where the large language model is stored. Among them, in this embodiment, at least one cluster may include Cluster A, Cluster B, Cluster C... Cluster N, etc., and the specific quantity is not limited.

[0038] In this embodiment, the configuration module can trigger operations in response to the deployment information of the large language model triggered by the user, as well as content such as model information. Based on these trigger content information, the large language model is configured and sent to at least one cluster based on the deployment object. In this embodiment, the management system of the large language model adopts a unified multi-cluster and multi-environment management method. Specifically, the environment library and the cluster are deployed in a hybrid manner. There is an environment library in the cluster, but it does not need to have a complete environment library. By combining the isolation of multiple clusters and the deployment strategy of multiple environments, the problem of resource conflicts between environments is solved. It can be understood that at least one of the clusters in this embodiment can include one or more specified environment libraries, so as to select the corresponding cluster and the specified environment library it contains when deploying the large language model. Among them, the environment library can include, but is not limited to, a test environment library and a production environment library.

[0039] In this embodiment, when deploying the large language model based on the configuration module, the environment library can be selected first, and then the deployment cluster can be selected to determine the cluster object and environment object where the large language model is finally deployed. In this embodiment, the page configuration service during deployment and the network isolation of each cluster can be solved through the data operation service, and the problem of reverse display of the call logs of each isolated cluster can be solved by collecting the logs of each cluster separately. For the large model in the financial risk control scenario, in order to ensure the reliability of the inference results as much as possible, it is necessary to first release and deploy the test environment for verification, and then deploy the same large model service application to the production environment for use.

[0040] The cluster 120 is used to receive the configuration information, and after performing configuration processing on the configuration information based on the included engine service, the corresponding large language model is obtained.

[0041] Among them, the engine service can be a software service for processing tasks related to the large language model. The engine service included in the cluster in this embodiment usually can include multiple functional modules for processing different aspects of the large language model. Specifically, the engine service can include a training module, an inference module, an optimization module, etc., and each module is responsible for a specific task.

[0042] In this embodiment, after each cluster receives the configuration information corresponding to the large language model sent by the configuration module, according to the different environment libraries included in each cluster, the relevant application information of the large language model is first processed through the environment library, and then other services of the engine service are processed, so that the configured large language model can be obtained.

[0043] The log query module 130 is used to receive a query request to query the call record information in the at least one cluster based on the query request.

[0044] Among them, the query request can be a request sent by the user to query the usage situation within each cluster. In this embodiment, each cluster has its own log push function. Each time a function within a different cluster is called, a call record will be generated accordingly, and then it will be pushed to the call details through the log for the log query module to query. Among them, the configuration module includes call details for displaying the call logs within different clusters on the cloud page. In this embodiment, the call record information and other usage information corresponding to at least one cluster can be queried through the log query request sent by the user.

[0045] In this embodiment, optionally, the configuration module includes a model repository unit and an application management unit; the model repository unit is used to provide large language models of multiple function types in response to user trigger operations; the application management unit is used to select corresponding deployment information in response to user trigger operations.

[0046] The configuration module of the large language model management system in this embodiment includes a model warehouse management unit and an application management unit. The model warehouse management unit can include large language models of various different function types, and is used to upload and save the large language model through the cloud page in response to user trigger operations; the application management module is used to select a cluster and an environment in response to user trigger operations to deploy the large model application to the corresponding cluster and the specified environment.

[0047] In this embodiment, through such a setting, information such as the model application information and the model file of the large language model can be configured through the cloud page, so as to flexibly determine the model configuration that meets the user's needs.

[0048] In this embodiment, optionally, the cluster includes a target environment library; the configuration module deploys the large language model to the at least one cluster and the corresponding target environment library.

[0049] Among them, the target environment library includes a production environment library and / or a test environment library. The test environment library can be used to perform application test operations on the large language model. The production environment library can be used to produce large language model service applications. It can be understood that different clusters can include different target environment libraries. Exemplarily, cluster A can include a production environment library; cluster B can include a test environment library; cluster C can include a production environment library and a test environment library. In this embodiment, the target environment library included can be determined according to information such as the specific positioning settings of the cluster.

[0050] In this embodiment, the large language model can be deployed to at least one corresponding cluster and the target environment library corresponding to the cluster through the configuration module. Through such a setting in this embodiment, multiple clusters and multiple environments can be managed to provide users with different types of deployment settings to meet user needs.

[0051] In this embodiment, optionally, the configuration module publishes the large language model to the test environment library for test verification and deploys the large language model that passes the test verification to the production environment for production use.

[0052] In this embodiment, the configuration module can send the large language model to the corresponding test environment library to test and verify the application function of the large language model, and publish the large language model that passes the test verification to the production environment, so as to apply the large language model service to production use through the production environment library.

[0053] Through such a setting in this embodiment, it is necessary to first publish and deploy the test environment for verification, and then deploy the same large model service application to the production environment for use, thereby ensuring the reliability of the inference results of the large language model in the financial risk control scenario.

[0054] In this embodiment, optionally, the cluster includes an engine service and a model inference service; the engine service unit is used to load the application information of the large language model published to the target environment library into the model inference service unit at a set time interval; the model inference service is used to register the service information of the large language model to obtain the registration information of the large language model.

[0055] Among them, the set time interval can be a preset time interval. Exemplarily, the set time interval can be 2 minutes or 3 minutes, and in this embodiment, it can be set according to actual needs. The model inference service can include multiple large language models such as large model A, large model B, and large model C.

[0056] Each cluster in this embodiment can include an engine service unit, which is used to load the application information of the large model published to the target environment library through the cloud page of the configuration module into the model inference service at a set time interval, so that the model inference service unit registers the obtained service information of the large language model in the Consul registration center to obtain the registration information of the large language model.

[0057] Through such a setting in this embodiment, the service information of the large language model can be registered to facilitate the implementation of the application inference service of the large language model.

[0058] In this embodiment, optionally, the cluster includes an application scheduling service unit; the engine service unit is configured to load the large language model application information published to the target environment library into the application scheduling service unit at a set time interval; the application scheduling service unit parses the application configuration information of the large language model to call the model inference service unit according to the application configuration information and the registration information.

[0059] The engine service unit in this embodiment is configured to load the large model application information published to the target environment library through the cloud page of the configuration module into the application scheduling service at a set time interval. The application scheduling service in this embodiment can read and parse and verify the various parameters of the large language model application information, so as to ensure the accuracy of the large language model information, call the model inference service unit that meets the application requirements according to the application configuration information and the registration information, and provide the inference service of the large language model externally.

[0060] Specifically, in this embodiment, the application scheduling service unit retrieves entries in the registration information library that match the parsed application configuration information; the registration information library usually stores the detailed information of available models, including the unique identifier of the model, the deployment location, the required resources, etc. Then, according to the model name or identifier in the configuration information, the corresponding model entry will be found in the registration information library. Thereby verifying the availability of the model, including checking whether the model is in an online state and whether there are sufficient resources to support the inference service, etc.

[0061] In this embodiment, through such a setting, the application scheduling service and the model inference service included in the engine service in the cluster can parse the model application configuration information and the registration information, and call the model inference service unit that meets the usage requirements, improving the adaptability of the application inference function of the large language model.

[0062] In this embodiment, optionally, the model inference service unit is further configured to provide an interface for the large language model application inference service externally, so that users can call the large language model through the interface for use.

[0063] The interface of the large language model application inference service in this embodiment can be an interface or a callable address. The model inference service unit in this embodiment also provides an interface for the large language model application inference externally, so as to facilitate users to call the large language model based on the interface for use of the inference function. Specifically, when a user uses the large language model, a corresponding question request is sent through the interface of the large language model application inference service, and after being processed by the application inference service of the large language model, a corresponding processing result is returned. In this embodiment, through such a setting, after deploying a large language model that meets the application requirements, a corresponding external interface can be set to facilitate meeting the usage requirements of users.

[0064] In this embodiment, optionally, the cluster includes a log pushing unit; the log pushing unit is configured to push the execution records of the model inference service unit and the application scheduling service unit into a message queue.

[0065] Among them, the execution record can be considered as the execution usage record of the model inference service unit and the application scheduling service unit. In this embodiment, each cluster can include a corresponding log pushing function. When calling the model inference service function and the application service unit corresponding to the large language model, the corresponding execution usage record information will be generated and pushed into the message queue of the log.

[0066] In this embodiment, through such a setting, the call records of the service can be automatically pushed into the log parsing unit for parsing through the log pushing function, improving the reliability of log records in the cluster.

[0067] In this embodiment, optionally, the cluster further includes a log parsing unit; the log parsing unit is configured to parse the execution records in the message queue and save them to the log library of the cluster for the log query module to view.

[0068] In this embodiment, each cluster can include a corresponding log parsing unit function. In this embodiment, the log parsing unit can parse and save the log messages in the message queue RabbitMQ to the cluster log library for the log query module to view.

[0069] In this embodiment, the log parsing unit can parse the log information in each cluster and store it in the corresponding log storage library to facilitate querying the corresponding service call records in response to a log query request.

[0070] The technical solution of the embodiment of the present invention includes a configuration module for responding to a user trigger operation, obtaining configuration information based on the user trigger operation, and sending the configuration information to at least one cluster; wherein, the configuration information includes the deployment information and model information of the large language model; the cluster is configured to receive the configuration information and obtain the large language model after performing configuration processing on the configuration information based on the included engine services; the log query module is configured to receive a query request to query the call record information in the at least one cluster based on the query request. This system realizes the flexibility and adaptability of the multi-scenario inference service application of the large language model through the management operation of multiple clusters.

[0071] Embodiment Two

[0072] Figure 2 is a structural example diagram of a management system for a large language model provided according to Embodiment Two of the present invention. AsFigure 2 As shown, the overall design architecture of the system in the embodiment of the present invention is mainly divided into two parts: the cloud configuration and log viewing module, clusters A and B, and the engine service modules included in each cluster. As Figure 2 shown, the test environment library and the production environment library are included in cluster A; the production environment library is included in cluster B.

[0073] Specifically, the cloud configuration and log viewing module in this embodiment may include: a model repository, application management, and call details. The model repository is used to upload and save large language models from trigger operations on the cloud page; the application management module is used to select clusters and environments from trigger operations on the cloud page and deploy the large model application to the specified environment and cluster; the call details module is used to display the call logs in different clusters on the cloud page. The engine service module within the cluster includes: an engine service part, a log push and storage part. The engine service part periodically loads the large model application information published to the specified environment library into the application scheduling service and the model inference service. The model inference service registers the service information in the Consul registry. The application scheduling service parses the large model application configuration information, calls the model inference service according to the application configuration and registration information, and provides the large model application inference service externally. The log push and storage part pushes the execution records of the model inference service and the application scheduling service to RabbitMQ. The log parsing part parses and saves the log messages in RabbitMQ to the cluster log library for the log query module to view. Among them, the model inference may include large model A, large model B, and large model C.

[0074] In this embodiment, both the configuration model application information and the large model model file can be configured through the cloud page; among them, the model repository manages the relationships of all models on the entire page, which may include information such as the name, type, and location of the models. By deploying the large model application configuration information to the specified environment and cluster; the engine service part within the cluster loads the large model application configuration and the large model file, and the large model application becomes effective within the cluster and environment, and can provide the large model application service externally; the records of external calls to the large model application are pushed and collected into the cluster through the log push and storage part, and the details of the cluster log records are viewed in the call details module on the cloud.

[0075] In the embodiments of the present invention, when deploying a large language model, the environment is first selected, and then the deployment cluster is selected to determine the final deployment cluster and environment of the large model. The page configuration service during deployment and the network isolation of each cluster are solved through the data operation service. The problem of reverse display of the call logs of each isolated cluster is solved by separately collecting the logs of each cluster. In this embodiment, the large model application configured through the cloud page first publishes the large model in the financial risk control scenario to the test library, which is then loaded by the large model inference engine service. After passing the test verification, it is published to the production library to apply the large model service to production, ensuring the reliability of the inference service. In this embodiment, the cloud page service and the cluster are associated through the data operation service, and the relevant configurations of the cloud page large model application are synchronized to the release library, solving the network isolation problem between the cloud configuration page and the inference engine cluster. In this embodiment, the environment library and the cluster are mixedly deployed. There is an environment library in the cluster, but it does not need to have a complete environment library. By combining the isolation of multiple clusters with the multi-environment deployment strategy, the resource conflict problem between environments is solved, and a unified management method for multiple clusters and multiple environments is realized.

[0076] It should be understood that various forms of processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0077] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A management system for a large language model, characterized in that: The management system of the large language model includes a configuration module, at least one cluster and a log query module: the configuration module is connected to the at least one cluster through a data operation service, and the at least one cluster is connected to the log query module through a network communication; The configuration module is used to respond to a user trigger operation, obtain configuration information based on the user trigger operation, and send the configuration information to at least one cluster; wherein the configuration information includes deployment information and model information of the large language model; The cluster is used to receive the configuration information, and obtain a corresponding large language model after performing configuration processing on the configuration information based on the included engine service; The log query module is used to receive a query request to query the call record information in the at least one cluster based on the query request.

2. The system according to claim 1, characterized in that The configuration module includes a model warehouse unit and an application management unit; The model warehouse unit is used to provide a large language model of multiple functional types in response to a user trigger operation; The application management unit is used to select corresponding deployment information in response to a user trigger operation.

3. The system according to claim 1, characterized in that The cluster includes a target environment library; The configuration module is used to deploy the large language model to the at least one cluster and the corresponding target environment library.

4. The system according to claim 3, characterized in that The target environment library includes a production environment library and / or a test environment library.

5. The system according to claim 4, characterized in that The configuration module is used to publish the large language model to the test environment library for test verification, and deploy the large language model that passes the test verification to the production environment for production use.

6. The system according to claim 1, characterized in that The cluster includes an engine service and a model reasoning service; The engine service unit is used to load the large language model application information published to the target environment library into the model reasoning service unit according to the set duration; The model inference service is used to register the service information of the large language model to obtain the registration information of the large language model.

7. The system according to claim 6, characterized in that The cluster includes an application scheduling service unit; The engine service unit is used to load the large language model application information published to the target environment library into the application scheduling service unit according to the set duration; The application scheduling service unit parses the application configuration information of the large language model to call the model reasoning service unit according to the application configuration information and the registration information.

8. The system according to claim 6, characterized in that The model inference service unit is also used to provide an interface for large language model application inference service to the outside, so that the user can call the large language model for use through the interface.

9. The system according to claim 6, characterized in that The cluster includes a log pushing unit; The log pushing unit is used to push the execution records of the model reasoning service unit and the application scheduling service unit to the message queue.

10. The system according to claim 9, characterized in that The cluster also includes a log parsing unit; The log parsing unit is used to parse the execution records in the message queue and save them in the log library of the cluster for viewing by the log query module.

Images