Artificial intelligence component invocation method, device, medium, and computer program product

Abstract

The application discloses an AI component calling method, device, medium and computer program product, relates to the technical field of container images, and comprises the following steps: when detecting that a first calling request sent by a large model application passes through a gateway, determining a target AI component container instance corresponding to the first calling request, the AI component container instance being a container instance obtained by containerizing and encapsulating an AI component that can be independently deployed, and comprising a model file and a protocol adapter; converting the first calling request into a second calling request according to a preset protocol format through the protocol adapter in the target AI component container instance; calling a model corresponding to the model file to process a task according to the second calling request, and obtaining a processing result; and encapsulating the processing result into a response message according to the preset protocol format through the protocol adapter, and sending the response message to the large model application through the gateway. The application improves the calling efficiency of the large model application in calling each AI component.

Description

[0001] AI component invocation methods, devices, media, and computer program products Technical Field

[0002] This application relates to the specific field of container image technology, and in particular to AI component invocation methods, devices, media, and computer program products. Background Technology

[0003] In current AI (Artificial Intelligence) systems, various AI components (such as Natural Language Processing (NLP) services, Computer Vision (CV) services, and speech recognition services) are typically deployed using a monolithic architecture. When external systems (such as large-scale model applications) need to call these AI components, various shortcomings exist, leading to low calling efficiency. For example, due to the heterogeneity of AI component protocols, large-scale model applications need to develop different protocol interfaces to call different AI components, further contributing to low efficiency. Therefore, improving the calling efficiency of large-scale model applications when calling various AI components has become an urgent problem to be solved.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an AI component invocation method, device, medium, and computer program product, aiming to solve the technical problem of how to improve the invocation efficiency of large model applications invoking various AI components.

[0006] To achieve the above objectives, this application proposes an AI component invocation method, which includes:

[0007] When the first call request sent by the large model application through the gateway is detected, the AI ​​component container instance corresponding to the first call request is determined as the target AI component container instance. The AI ​​component container instance is a container instance obtained by containerizing and encapsulating the AI ​​component that can be deployed independently. The AI ​​component container instance includes model files and protocol adapters.

[0008] The first call request is converted into a second call request by the protocol adapter in the target AI component container instance according to the preset protocol format.

[0009] Based on the second invocation request, the model corresponding to the model file in the target AI component container instance is invoked to perform task processing and obtain the processing result;

[0010] The processing result is encapsulated into a response message in a preset protocol format by a protocol adapter in the target AI component container instance, and is sent to the large model application through the gateway.

[0011] Optionally, the AI component calling method further includes:

[0012] The plurality of monolithic architectures containing AI capability components are decoupled to obtain a plurality of AI components that can be independently deployed.

[0013] Determine the dependency file corresponding to each AI component, which contains at least one dependency, wherein the dependency includes at least one of the model file of the AI component, the inference engine library, the operating system package, the protocol adapter and the health monitoring script.

[0014] For each AI component, the AI component and the dependency file of the AI component are containerized and encapsulated to obtain an AI component container instance.

[0015] Optionally, the step of containerizing and encapsulating the AI component and the dependency file of the AI component to obtain the AI component container instance includes:

[0016] Encapsulate the AI component and at least one dependency in the dependency file into a preset container image to obtain an AI component container image.

[0017] Determine the declaration file corresponding to the AI component container image, and deploy the AI component container image in the preset container orchestration platform with the declaration file.

[0018] Determine the AI component container instance to be registered for service and metadata publishing in the container orchestration platform according to the deployed declaration file, wherein the AI component container instance is registered for service and metadata publishing in the container orchestration platform, so that the large model application can call the AI component container instance for task processing through the gateway.

[0019] Optionally, the preset container image includes two different first and second base images, and the step of encapsulating the AI component and the dependency file of the AI component into the preset container image to obtain the AI component container image includes:

[0020] In the compilation environment stage, copy the AI component and at least one dependency in the dependency file to the first base image to obtain a temporary build image, wherein the preset Python library and operating system package are installed in the first base image, the inference engine in the inference engine library is downloaded and compiled to obtain a compiled inference engine binary file, and the model file is preheated according to the inference engine corresponding to the compiled inference engine binary file to obtain a preheating mark file.

[0021] In the running environment stage, if the preheating state corresponding to the preheating mark file is preheating completion, the model file, the protocol adapter, the Python library and the compiled inference engine binary file in the temporary constructed image are copied to the second base image to obtain the AI component container image.

[0022] Optionally, the step of determining the declaration file corresponding to the AI component container image comprises:

[0023] According to the preset resource configuration rule, the computing resource required by the AI component container instance is determined, wherein the resource configuration rule comprises the scaling rule determined according to the real load of the computing resource and the preset load threshold, the computing resource support rule of at least two AI component container instances, and the node affinity and stain tolerance rule of the computing resource;

[0024] According to the computing resource, the declaration file is determined, wherein the declaration file comprises the specified AI component container image, the number of copies and the computing resource, and the number of copies is the number of instances of the AI component container instance.

[0025] Optionally, the AI component calling method further comprises:

[0026] When the target AI component container instance is called by the large model application to process a task, the service state corresponding to the target AI component container instance is monitored in real time through a preset container probe;

[0027] When the service state is monitored to be abnormal, the task processing operation of the target AI component container instance is stopped, and a new target AI component container instance is reconstructed to continue the task processing according to the new target AI component container instance.

[0028] Optionally, the AI component calling method further comprises:

[0029] The AI component container instance corresponding to the first calling request receives the first calling request sent by the gateway according to the preset intelligent routing decision logic, wherein the preset intelligent routing decision logic comprises that the gateway filters a plurality of available first AI component container instances in the preset registration center according to the first calling request sent by the large model application, and filters the first AI component container instance with the best performance from the plurality of first AI component container instances as the AI component container instance corresponding to the first calling request according to the real-time monitoring data of at least one first AI component container instance and the preset service quality requirement, and sends the first calling request to the AI component container instance corresponding to the first calling request.

[0030] In addition, in order to achieve the above-mentioned purpose, the application further provides an AI component calling device, which comprises:

[0031] The determining module is configured to determine that the AI component container instance corresponding to the first calling request is a target AI component container instance when it is detected that the large model application sends the first calling request through the gateway, wherein the AI component container instance is a container instance obtained by containerizing and encapsulating an AI component that can be independently deployed, and the AI component container instance includes a model file and a protocol adapter;

[0032] The conversion module is configured to perform format conversion on the first calling request according to a preset protocol format through the protocol adapter in the target AI component container instance to obtain a second calling request.

[0033] The processing module is configured to perform task processing on a model corresponding to the model file in the target AI component container instance according to the second calling request to obtain a processing result.

[0034] The sending module is configured to encapsulate the processing result into a response message according to the preset protocol format through the protocol adapter in the target AI component container instance, and send the response message to the large model application through the gateway.

[0035] In addition, to achieve the above-mentioned purpose, the present application further provides an AI component calling device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the AI component calling method as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application further provides a medium, which is a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the AI component calling method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by the processor to implement the steps of the AI component calling method as described above.

[0038] In the embodiments of the present application, since the AI component container instance is a container instance obtained by containerizing and encapsulating the separately deployable AI component, and at least contains the model file and the protocol adapter, the high coupling degree of the monolithic architecture system can be avoided, and the phenomenon of poor service continuity caused by the need for overall maintenance when upgrading or replacing the AI component can be avoided. When the first invocation request sent by the large model application through the gateway is detected, the target AI component container instance corresponding to the first invocation request is determined, and the protocol adapter in the target AI component container instance is used to convert the first invocation request into a second invocation request according to the preset protocol format, and then the model corresponding to the model file in the target AI component container instance is invoked to process the task, and the processing result is obtained, and then the protocol adapter is used to encapsulate the processing result into a response message according to the preset protocol format, and the response message is sent to the large model application through the gateway. Therefore, the phenomenon of low calling efficiency caused by the protocol heterogeneity of the AI component and the need for developing different protocol interfaces for calling different AI components by the large model application can be avoided, and the calling efficiency of the large model application for calling each AI component can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0041] Figure 1 A flowchart provided by the first embodiment of the AI component calling method of the present application;

[0042] Figure 2 A flowchart provided by the second embodiment of the AI component calling method of the present application;

[0043] Figure 3 A flowchart of the AI component calling method of the present application;

[0044] Figure 4 A module diagram of the AI component calling device according to the embodiments of the present application;

[0045] Figure 5 A device structure diagram of the hardware running environment involved in the AI component calling method in the embodiments of the present application.

[0046] The objectives, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0048] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0049] In the current AI system, each AI component (such as natural language processing NLP service, computer vision CV service, speech recognition service, etc.) is usually deployed using a monolithic architecture, which has the following defects:

[0050] High system coupling: AI component upgrade or replacement requires overall downtime maintenance, poor service continuity;

[0051] Resource allocation is rigid: in high concurrency scenarios, high-frequency AI components cannot be independently expanded, resulting in inefficient AI component grabbing computing resources (such as GPU);

[0052] Protocol heterogeneity: Different AI components require custom interface development for large model calls, and there are problems such as non-uniform cross-component communication protocols and low development efficiency;

[0053] Lack of elastic capability: Traditional deployment methods cannot dynamically scale instances according to load, and fault recovery relies on manual intervention.

[0054] The above defects seriously affect the availability and resource utilization of the AI system, resulting in low call efficiency of the large model application calling the AI component.

[0055] In the present embodiment, AI component decoupling deployment is achieved by constructing a microservice architecture based on a unified protocol, which can be a microservice architecture based on MCP (Model Calling Protocol, model calling protocol). MCP is a standard protocol for communication between models and tools. It allows models to dynamically call external tools (such as APIs (Application Programming Interface, application programming interface), databases, etc.) according to task requirements, thereby enhancing model capabilities.

[0056] In the present embodiment, when a large model application calls an AI component through a microservice architecture based on a unified protocol, the call efficiency can be improved, and the following advantages can be achieved:

[0057] Protocol layer unified scheduling: define a standardized MCP communication protocol (including fields such as service identification, input and output format, Qos (quality of service) quality requirements, and timeout mechanism) to enable the large model to call heterogeneous components (i.e., AI components) through a single protocol;

[0058] Containerized packaging: encapsulate each AI component (such as OCR (Optical Character Recognition), speech recognition) as an independent container instance (i.e., AI component container instance), which includes an inference engine and a protocol adapter (such as an MCP protocol adapter) for model inference;

[0059] Dual-level scheduling mechanism: dynamically route requests by the gateway (such as the MCP gateway) (select the optimal AI component container instance according to Qos requirements and real-time load), and link the container orchestration platform to realize resource elastic supply;

[0060] Health monitoring closed loop: real-time detection of AI component service status through container probes, and instance reconstruction or traffic switching triggered by abnormal fields.

[0061] Therefore, the microservice architecture based on the unified protocol in the embodiment can realize independent operation and on-demand scaling of AI components through protocol decoupling and service isolation.

[0062] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, or an electronic device, an AI component calling device, etc. capable of realizing the above functions. The following takes the AI component calling device as an example to illustrate the embodiment and the following embodiments.

[0063] Based on this, the AI component calling method provided in the embodiment of the present application is provided, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the AI component calling method of the present application is shown.

[0064] In the embodiment, the AI component calling method includes steps S10-S40.

[0065] Step S10, when detecting that the large model application sends a first calling request through the gateway, determining that the AI component container instance corresponding to the first calling request is a target AI component container instance;

[0066] It should be noted that the AI component container instance is a container instance obtained by containerizing and packaging the AI component that can be independently deployed, and the AI component container instance includes a model file and a protocol adapter;

[0067] Optionally, the model file can be a file embodying the function of the model, such as a file containing the code of the model, and the model corresponding to the model file can be a model embodying the function of the AI component, such as a speech recognition model, a text recognition model, etc.

[0068] Optionally, the protocol adapter can be an MCP protocol adapter, and each AI component container instance has a protocol adapter, such as an MCP protocol adapter.

[0069] Optionally, the terminal where the large model application is located, the gateway, and the terminal where the microservice architecture based on the unified protocol are connected in turn. The terminal where the microservice architecture based on the unified protocol can include a plurality of AI component container instance pools, and can be provided with a container orchestration platform, an AI component (such as an AI component with a natural language processing NLP service function, an AI component with a computer vision CV service function, and an AI component with a speech recognition service), etc.

[0070] Optionally, the large model application can be an application in a terminal (such as a server) that can use a large model. The large model can be a natural language large model or other types of large models, which are not limited here.

[0071] Optionally, the external system such as the large model application can call the capabilities of the AI component through the gateway, and can perform request analysis of the large model application in the gateway when sending a request to the gateway, determine the required service capability type, and then select the available instance pool containing a plurality of AI component container instances from the registration center according to the service capability type, and can combine real-time monitoring data (such as the current load, historical response delay, error rate, GPU (Graphics Processing Unit, GPU) utilization, and geographic location of each AI component container instance, etc.) and Qos requirements, etc. The optimal AI component container instance is selected as the target AI component container instance, and the request of the large model application is forwarded to the target AI component container instance as the first call request.

[0072] Optionally, the AI component calling method further comprises: receiving, by the AI component container instance corresponding to the first call request, the first call request sent by the gateway according to the preset intelligent routing decision logic.

[0073] It should be noted that the preset intelligent routing decision logic includes that the gateway screens a plurality of available first AI component container instances from the preset registration center according to the first invocation request sent by the large model application, and screens the first AI component container instance with the best performance from the plurality of first AI component container instances as the AI component container instance corresponding to the first invocation request according to the real-time monitoring data of at least one first AI component container instance and the preset service quality requirement, and sends the first invocation request to the first AI component container instance corresponding to the first invocation request.

[0074] Optionally, the first AI component container instance can be an AI component container instance that has completed service registration and metadata publishing in the preset registration center through the container orchestration platform.

[0075] Optionally, the real-time monitoring data can include current load, historical response delay, error rate, GPU utilization, geographical location, etc. of the AI component container instance.

[0076] Optionally, the service quality requirement can be a Qos requirement, such as low delay, high precision, and specific version, etc.

[0077] Optionally, the first AI component container instance with the best performance can be an AI component container instance with the lowest delay, an AI component container instance with the smallest number of connections, etc.

[0078] Optionally, the large model application sends the first invocation request to the gateway, and the gateway can screen the AI component container instance corresponding to the first invocation request from the registration center according to the preset intelligent routing decision logic, that is, the gateway screens a plurality of available first AI component container instances from the preset registration center according to the first invocation request sent by the large model application, and screens the first AI component container instance with the best performance from the plurality of first AI component container instances as the AI component container instance corresponding to the first invocation request according to the real-time monitoring data of at least one first AI component container instance and the preset service quality requirement, and forwards the first invocation request to the AI component container instance corresponding to the first invocation request. The AI component container instance corresponding to the first invocation request receives the first invocation request sent by the gateway and performs subsequent task processing.

[0079] In this embodiment, the gateway first determines a plurality of available first AI component container instances in the registration center, then filters the first AI component container instance with the best performance according to real-time monitoring data and preset service quality requirements as the AI component container instance corresponding to the first invocation request, and forwards the first invocation request to the AI component container instance corresponding to the first invocation request. The AI component container instance corresponding to the first invocation request receives the first invocation request sent by the gateway and performs subsequent task processing, thereby ensuring that the large model application invokes an effective AI component container instance to efficiently complete the corresponding task.

[0080] In step S20, the protocol adapter in the target AI component container instance performs format conversion on the first invocation request according to the preset protocol format to obtain a second invocation request.

[0081] Optionally, the preset protocol format can be a format consistent with the protocol adapter, such as the protocol format of the MCP protocol.

[0082] Optionally, after the target AI component container instance receives the first invocation request, the protocol adapter in the target AI component container instance can perform format conversion on the first invocation request, such as converting it into a standardized MCP communication protocol format invocation request, and taking it as the second invocation request.

[0083] In step S30, the model corresponding to the model file in the target AI component container instance is invoked according to the second invocation request to perform task processing, and a processing result is obtained.

[0084] Optionally, in the target AI component container instance, the model corresponding to the encapsulated model file can be invoked according to the second invocation request, so as to perform task processing according to the invoked model, such as image recognition, voice translation, etc., thereby obtaining the processing result of the model. For example, if image recognition is required, the image to be recognized included in the second invocation request is input to the model, and the recognition result for the image to be recognized is output as the processing result.

[0085] In step S40, the protocol adapter in the target AI component container instance encapsulates the processing result into a response message according to the preset protocol format, and sends it to the large model application through the gateway.

[0086] Optionally, when the protocol adapter is an MCP protocol adapter, the MCP protocol adapter can be used to convert the processing result of the model into a standard MCP response format response message, send it to the gateway, and send it to the large model application through the gateway.

[0087] In addition, to assist in understanding the process of invoking the AI component by the large model application in this embodiment, the following examples are provided.

[0088] For example, an AI capability call initiated by an external system (such as a large model application) will be expressed as a MCP request in a standard format, which is first sent to the gateway. After receiving the request, the gateway will perform key intelligent routing decision logic, including: request parsing, instance matching and selection, protocol conversion and invocation, result adaptation and return, and finally response.

[0089] Request parsing: The gateway parses the service_id (explicitly required service capability type) and Qos requirements (such as low latency, high precision, and specific version, etc.) in the request.

[0090] Instance matching and selection: According to the service_id, the gateway can filter out the available instance pool (including multiple available AI component container instances) that meets the conditions from the registration center. Then, combined with real-time monitoring data (such as the current load, historical response delay, error rate, GPU utilization, and geographical location of each AI component container instance, etc.) and Qos requirements, the gateway calculates and selects the optimal AI component container instance for request forwarding through optimized routing algorithms (such as lowest latency first, weighted round robin, minimum connection number, etc.), i.e. sending the first invocation request to the target AI component container instance through the gateway.

[0091] Protocol conversion and invocation: After the gateway sends the first invocation request to the target AI component container instance, the protocol adapter in the target AI component container instance, such as the MCP protocol adapter, processes the first invocation request and invokes the encapsulated model for task processing.

[0092] Result adaptation and return: After the model processing is completed, the processing result is generated, which is converted into a response message in the standard MCP response format by the protocol adapter, and then sent to the gateway.

[0093] Final response: The gateway sends the response message in the standard MCP response format to the large model application.

[0094] This complete invocation process ensures that the invoker (i.e. the large model application) only needs to follow one protocol to transparently access all different backend services.

[0095] In the embodiment, since the AI component container instance is a container instance obtained after containerization encapsulation processing is performed on an AI component that can be separately deployed, and at least contains a model file and a protocol adapter, the high coupling degree of a monolithic architecture system can be avoided, the phenomenon that the AI component needs to be upgraded or replaced and the whole system needs to be shut down for maintenance, and the service continuity is poor, and when it is detected that the first invocation request sent by the large model application passes through the gateway, the target AI component container instance corresponding to the first invocation request is determined, and the protocol adapter in the target AI component container instance is used to convert the first invocation request into a second invocation request according to a preset protocol format, and then the model corresponding to the model file in the target AI component container instance is invoked to process a task, and a processing result is obtained, and then the protocol adapter is used to encapsulate the processing result into a response message according to the preset protocol format, and the response message is sent to the large model application through the gateway, so that the phenomenon that the large model application needs to develop different protocol interfaces to call different AI components due to the protocol heterogeneity of the AI components can be avoided, and the calling efficiency of the large model application to call each AI component can be improved.

[0096] The second embodiment,

[0097] Based on the first embodiment, the second embodiment is proposed, and the first embodiment is referred to Figure 2 In the embodiment, the AI component calling method further includes steps S100-S300.

[0098] Step S100, decoupling a plurality of monolithic architectures containing AI capability components to obtain a plurality of AI components that can be separately deployed;

[0099] Optionally, component micro-service reconstruction and protocol adapter development can be performed to build a micro-service architecture based on a unified protocol.

[0100] Optionally, each AI capability component (such as a natural language processing (NLP) service, a computer vision (CV) service, and a speech recognition service) in the original monolithic architecture can be decoupled and split into independent, separately deployable and maintainable functional units, that is, independent, separately deployable and maintainable AI components.

[0101] Optionally, a protocol adapter dedicated to each AI component, such as an MCP protocol adapter, can be developed and deployed, and the protocol adapter can be integrated into the AI component.

[0102] Optionally, the protocol adapter can realize bidirectional conversion, at the input end of the AI component, receiving and parsing the standard MCP request from the gateway, extracting the payload (such as image data, text content), and converting it into an input format understandable by the internal API of the AI component; at the output end of the AI component, receiving the processing result (such as the recognized object, the translated text) of the internal AI component, and encapsulating it into standardized response information according to the MCP protocol specification, so that the heterogeneous components expose a unified communication protocol to the outside.

[0103] Step S200, determining the dependency file corresponding to each AI component, which contains at least one dependency;

[0104] It should be noted that the dependency includes at least one of the model file of the AI component, the inference engine library, the operating system package, the protocol adapter, and the health monitoring script.

[0105] Optionally, the model file can include a model weight file, a model body code, a model identifier, etc. The model can be a model that implements the basic function of the AI component, such as a speech recognition model if the basic function of the AI component is speech recognition.

[0106] Optionally, the inference engine library includes at least one inference engine, such as a TensorRT engine.

[0107] Optionally, the operating system package can include various operating systems.

[0108] Optionally, the protocol adapter can be a unified protocol adapter, such as an MCP protocol adapter.

[0109] Optionally, the health monitoring script can be used to control the container probe to monitor the service state of the AI component container, and automatically trigger AI component container instance reconstruction or traffic switching when an exception occurs.

[0110] Optionally, the dependency file can include multiple dependencies, including but not limited to a model file, an inference engine or an inference engine library, a necessary operating system package, an MCP protocol adapter (which can be embodied by MCP protocol adapter code), and a health monitoring script, etc.

[0111] Step S300, for each AI component, containerizing and encapsulating the AI component and the dependency file of the AI component to obtain an AI component container instance.

[0112] Optionally, after the component micro-service reconstruction and protocol adapter development, container image construction and environment encapsulation can be performed to build a micro-service architecture based on a unified protocol.

[0113] Optionally, each independent AI component and each dependency in the dependency file can be containerized and encapsulated to obtain an AI component container instance.

[0114] Optionally, the AI component and the dependency can be encapsulated into a Docker container image, and a Dockerfile (a description file for building a Docker image) can be written to define the build steps, including selecting a base image (such as an NVIDIA PyTorch image containing a specific CUDA version), copying code and model files, installing runtime dependencies, exposing service ports, configuring health check probes (such as an HTTP GET / healthz endpoint), and setting commands to run at startup, such as starting the MCP protocol adapter service, and finally building an AI component container image that is highly portable and can be deployed in any compatible Docker / Kubernetes environment with the same runtime consistency, thereby solving the coupling and deployment rigidity problems of monolithic applications.

[0115] Optionally, a container orchestration management platform (such as Kubernetes (K8s)) can be used to cluster and deploy the AI component container image, and to register and publish metadata, so that the gateway can discover and call the AI component container instance.

[0116] In this embodiment, by decoupling a plurality of monolithic architectures, a plurality of AI components are obtained, dependency files corresponding to each AI component are determined, and for each AI component, the AI component and the dependency file are containerized and encapsulated to obtain an AI component container instance, thereby ensuring the effectiveness of the obtained AI component container instance.

[0117] A third embodiment,

[0118] Based on the first or second embodiment, a third embodiment is proposed, in which step S300, the step of containerizing and encapsulating the AI component and the dependency file of the AI component to obtain an AI component container instance, includes steps a10-a30.

[0119] Step a10, encapsulating the AI component and at least one dependency in the dependency file into a preset container image to obtain an AI component container image;

[0120] Optionally, each dependency in the dependency file of the AI component can be determined, and the AI component and at least one dependency can be encapsulated into a preset container image to obtain an AI component container image.

[0121] Optionally, each independent AI component can correspond to an AI component container image.

[0122] Optionally, after obtaining the AI component container image, the AI component container image can be stored in a preset image warehouse for subsequent invocation by a container orchestration platform.

[0123] Optionally, the preset container image includes two different first and second base images.

[0124] Optionally, the first base image can be a base image containing a complete compilation tool chain, used for installing compilation-time dependencies, downloading and compiling source code of a high-performance inference engine (such as TensorRT or ONNX Runtime), and installing Python dependency packages.

[0125] Optionally, the second base image can be a minimal base image, such as nvidia / cuda:11.8.0-runtime-ubuntu22.04.

[0126] Optionally, in step a10, the AI component and the dependency file of the AI component are encapsulated into a preset container image to obtain an AI component container image, including steps a11-a12.

[0127] In step a11, in the compilation environment stage, at least one dependency in the AI component and the dependency file is copied to the first base image to obtain a temporary build image.

[0128] It should be noted that the preset Python library and operating system package are installed in the first base image, the inference engine in the inference engine library is downloaded and compiled to obtain a compiled inference engine binary file, and the model file is preheated according to the inference engine corresponding to the compiled inference engine binary file to obtain a preheating mark file.

[0129] Optionally, the temporary build image can include the first base image with the copied AI component, dependency, preheating mark file, Python, complete compilation tool chain, and the like.

[0130] Optionally, when building the AI component container image, a multi-stage building method can be used to optimize the image size and security.

[0131] Optionally, the multi-stage building method can include a compilation environment stage (builder stage) and a running environment stage (runtime stage).

[0132] Optionally, in the compiling environment stage, a base image containing a complete compiling tool chain can be selected as the first base image, and in the first base image, an operating system package and a Python library are installed, and a reasoning engine is downloaded and compiled in the reasoning engine library. The AI component and at least one dependency (such as a protocol adapter, a model file, etc.) of the AI component are copied into the first base image.

[0133] Optionally, in the first base image, the downloaded reasoning engine can be compiled according to the complete compiling tool chain and the Python library, and the model file can be loaded into the reasoning engine for model warm-up processing. That is, in the compiling environment stage, the model warm-up and caching mechanism can be implemented, and in the building instruction of the Dockerfile, a special RUN command can be added, which can execute a model warm-up script. The model warm-up script can download the required model weight file from the model warehouse (such as Hugging Face, S3 storage bucket) in the building stage, and perform model file simulation reasoning.

[0134] Optionally, the model corresponding to the model file can be simulated to complete the model warm-up processing (such as GPU memory allocation, CUDA kernel compilation, etc.), and a warm-up mark file is generated to determine whether the image building operation in the running environment stage needs to be continued according to the warm-up mark file.

[0135] In step a12, in the running environment stage, if the warm-up state corresponding to the warm-up mark file is warm-up completed, the model file, the protocol adapter, the Python library and the compiled reasoning engine binary file in the temporary building image are copied to the second base image to obtain the AI component container image.

[0136] Optionally, in the running environment stage, the warm-up state corresponding to the warm-up mark file can be judged, and if the warm-up state is warm-up completed, the model file, the protocol adapter, the Python library and the compiled reasoning engine binary file in the temporary building image in the compiling environment stage can be copied into the very simple second base image to obtain the running stage image, which is used as the AI component container image.

[0137] In the embodiment, the AI component and the at least one dependency are encapsulated into a preset container image, and a multi-stage building manner is used for processing when the container is encapsulated, that is, in the compilation environment stage, the AI component and the dependency are copied to a first base image to obtain a temporary building image, in the running environment stage, the model file, the protocol adapter, the Python library and the compiled inference engine binary file in the temporary building image are copied to a second base image to obtain the AI component container image, thereby reducing the size of the AI component container image and ensuring the effectiveness of the obtained AI component container image.

[0138] In step a20, a declaration file corresponding to the AI component container image is determined, and the declaration file deployment of the AI component container image is performed in the preset container orchestration platform.

[0139] Optionally, the declaration file of the AI component container corresponding to the AI component container image can be edited and deployed in the container orchestration platform.

[0140] Optionally, in step a20, the declaration file corresponding to the AI component container image is determined, including steps a21-a22.

[0141] In step a21, the computing resource required by the AI component container instance is determined according to a preset resource configuration rule, wherein the resource configuration rule includes an expansion and contraction rule determined according to a real load of the computing resource and a preset load threshold, a computing resource support rule of at least two AI component container instances, and a node affinity and stain tolerance rule of the computing resource.

[0142] Optionally, the computing resource can be a GPU resource or a CPU resource, and the following is an example of the GPU resource.

[0143] A GPU allocation strategy can be configured for the GPU resource to meet the computing resource requirement of the AI component container instance when performing task processing.

[0144] Optionally, GPU resource declaration and request can be performed: in the resources.limiits (resource limit) and requests (resource request) of the AI component container, nvidia.com / gpu:1 is explicitly declared, and the scheduler of the container orchestration platform (such as Kubernetes) is forced to allocate a complete and exclusive GPU device for the Pod (the smallest scheduling unit). Optionally, the Pod can include the AI component container instance.

[0145] Optionally, the GPU resource allocation can be performed according to the resource configuration rule to determine the computing resource, such as the GPU resource, required by the at least one AI component container instance.

[0146] Optionally, node affinity of computing resource and taint tolerance rule: through nodeAffinity, the Pod is forced to be scheduled only on the node with a specific model of GPU (such as NVIDIA-A100) indicated in the label. At the same time, the taint (Taints) of the GPU node is set, such as kubectl taint nodes gpu-node-1 nvidia.com / gpu=true:NoSchedule. The Pod can tolerate the taint through tolerations, ensuring that the expensive GPU node is only used by the Pod that needs GPU, and achieving logical isolation of physical resources.

[0147] Optionally, computing resource support rule of at least two AI component container instances, such as multi-instance GPU (MIG) support: for a GPU (such as NVIDIA-A100) supporting MIG, a MIG partition instead of the entire GPU can be requested in the resource declaration (a declaration item included in the declaration file) (such as nvidia.com / mig-1g.5gb: 1), so as to split a single physical GPU into multiple independent micro-service instances (i.e. AI component container instances), achieving finer-grained resource sharing and higher utilization.

[0148] Optionally, the scaling rule (HorizontalPod Autoscaler, HPA) determined according to the actual load of the computing resource and the preset load threshold can make scaling decisions according to the set CPU and / or GPU utilization target threshold (TargetUtilization). For example, HPA can be configured to increase the number of replicas when the average GPU utilization of the AI component container is continuously more than 70%, and to decrease the number of replicas when it is less than 30%. K8s ensures that the AI component container instance is scheduled and run on the Node node according to the strategy, achieving elastic supply and isolation of resources.

[0149] Optionally, the scaling rule determined according to the actual load of the computing resource and the preset load threshold can include elastic scaling based on GPU actual load: configure HorizontalPodAutoscaler (HPA) to no longer rely on indirect indicators such as CPU or memory, but directly associate the custom metric of GPU actual utilization (nvidia_gpu_utilization) collected by the Prometheus monitoring system. When the average GPU utilization of all instances (i.e. AI component container instances) continuously exceeds the preset threshold (such as 70%), HPA will automatically increase the number of replicas, and vice versa, achieving precise and dynamic response to the demand for computing resources.

[0150] Step a22, determine the declaration file according to the computing resources, wherein the declaration file includes the specified AI component container image, the number of copies and the computing resources, and the number of copies is the number of instances of the AI component container instance.

[0151] Optionally, each declaration item can be written in the container orchestration management platform according to the available computing resources (such as GPU resources, CPU resources, etc.), to form a declaration file.

[0152] Optionally, the declaration item can include the specified AI component container image, the number of instances of the AI component container instance, the computing resources of the AI component container configuration, etc.

[0153] Optionally, the AI component container image stored in the image repository and its version number can be specified in the declaration file, and the number of copies of the AI component container image and the required computing resources can be defined through the replicas field.

[0154] Optionally, the declaration file can be submitted to the K8s cluster, and the K8s schedules the AI component container instance according to the node resources and labels, and allocates the required computing resources (such as GPU) of the scheduled AI component container instance.

[0155] Optionally, the AI component container image specified in the declaration file will be pulled from the image repository by the container orchestration platform (such as K8s), and will be run on the nodes of the K8s cluster. The K8s will automatically create or destroy the AI component container instance corresponding to the specified AI component container image according to the replicas field, to ensure that there is always a specified number of AI component container instances running.

[0156] In this embodiment, the computing resources are determined according to the resource configuration rules, and the declaration file is determined according to the computing resources, thereby ensuring the validity of the determined declaration file.

[0157] Step a30, determine the AI component container instance to be registered for service and published for metadata in the container orchestration platform according to the deployed declaration file, wherein the AI component container instance is registered for service and published for metadata in the container orchestration platform, so that the large model application can call the AI component container instance for task processing through the gateway.

[0158] Optionally, a service object can be created according to a declaration file deployed in a container orchestration platform to perform a service registration and metadata publishing process. Optionally, a group of pods (such as AI component container instances) can be exposed as accessible service objects. When the AI component container instance is successfully started in K8s, the internal startup script automatically performs a service registration process, and the AI component container instance reports its key registration information by calling a range registration API endpoint provided by the MCP gateway. Optionally, the key registration information can include the running address (IP address or service discovery domain name) and port number of the AI component container instance, the core capability identifier (Service ID) it implements (such as 'nlp-translation-en2zh'), the service version number, the maximum processing capability supported (such as the maximum query rate per second Max QPS), the current health status (Healthy / Unhealthy), and other metadata that may be needed (such as supported input parameter options, model version). The MCP gateway receives and persists these registration information, and builds a global view of the entire microservice cluster capability, providing basic data support for subsequent intelligent routing decisions. That is, at this time, the large model application can call each AI component container instance in the K8s cluster through the gateway to perform task processing, such as voice translation, image recognition, etc.

[0159] Optionally, after the component microservice reconstruction and protocol adapter development, as well as the container image construction and environment encapsulation, the cluster deployment and resource configuration, and the service registration and metadata publishing can be performed to build a microservice architecture based on a unified protocol.

[0160] Optionally, when performing cluster deployment and resource configuration, the Kubernetes (K8s) container orchestration platform can be used to manage the deployed and encapsulated AI component containers, and a container orchestration platform deployment declaration (Deployment) file can be written to specify the container image (such as a specified AI component container image), the number of replicas (Replicas), and the required resources (i.e., computing resources such as GPU resources, CPU resources).

[0161] Optionally, when performing service registration and metadata publishing, the declaration file can be used to determine the service object (such as a group of AI component container instances), and the service registration API endpoint provided by the gateway can be used to perform service registration and metadata publishing.

[0162] In the embodiment, the AI component container image is obtained by encapsulating the AI component and at least one dependency into a preset container image, the declaration file of the AI component container image is deployed, and the service registration and metadata are published, so that the large model application can normally call the AI component container instance through the gateway to perform task processing, thereby ensuring the calling efficiency of the large model application in calling each AI component.

[0163] A fourth embodiment,

[0164] Based on the first embodiment, the second embodiment or the third embodiment, a fourth embodiment is provided. In the embodiment, the AI component calling method further includes steps b10-b20.

[0165] In step b10, when the large model application calls the target AI component container instance for task processing, the service state corresponding to the target AI component container instance is monitored in real time through a preset container probe.

[0166] In step b20, when the service state is monitored to be abnormal, the task processing operation of the target AI component container instance is stopped, and a new target AI component container instance is reconstructed to continue the task processing according to the new target AI component container instance.

[0167] Optionally, at any link where the large model application starts to call the target AI component container instance for task processing until the task processing is completed and a feedback response message is received, the service state of the AI component container of the target AI component container instance can be detected through the container probe set in advance to determine the health state.

[0168] When the service state is monitored to be abnormal, the corresponding task processing operation can be stopped, a new target AI component container instance is reconstructed, and the task processing is continued according to the new target AI component container instance. Flow switching and other operations, such as reselecting the AI component container instance, can also be performed.

[0169] For example, as shown in Figure 3 The AI component can be sequentially subjected to component micro-service reconstruction, container image construction, cluster deployment and service registration. When the external large model application requests to dispatch and execute any AI component container instance of the AI component, operation and maintenance monitoring loop is performed to update the health state, such as real-time monitoring of the service state through the container probe to determine the health state of the AI component, and the expansion and contraction of the computing resources, such as GPU, are fed back.

[0170] In the embodiment, when the large model application invokes the target AI component container instance for task processing, the service state corresponding to the target AI component container instance is monitored through the container probe, and when it is determined that the service state is abnormal, the task processing operation of the target AI component container instance is stopped, and a new target AI component container instance is reconstructed to continue task processing according to the new target AI component container instance, thereby ensuring the effective performance of task processing.

[0171] In addition, in order to achieve the above-mentioned purpose, referring to Figure 4 The application also provides an AI component calling device, which comprises:

[0172] The determining module A10 is configured to determine that the AI component container instance corresponding to the first calling request sent by the large model application through the gateway is a target AI component container instance when it is detected that the first calling request is sent by the large model application through the gateway, wherein the AI component container instance is a container instance obtained by containerizing and encapsulating an AI component that can be independently deployed, and the AI component container instance comprises a model file and a protocol adapter;

[0173] The conversion module A20 is configured to convert the first calling request into a second calling request according to a preset protocol format through the protocol adapter in the target AI component container instance.

[0174] The processing module A30 is configured to call a model corresponding to the model file in the target AI component container instance to perform task processing according to the second calling request, and obtain a processing result.

[0175] The sending module A40 is configured to encapsulate the processing result into a response message according to the preset protocol format through the protocol adapter in the target AI component container instance, and send the response message to the large model application through the gateway.

[0176] The AI component calling device provided by the application adopts the AI component calling method in the above-mentioned embodiments, and can improve the calling efficiency of the large model application in calling each AI component. Compared with the prior art, the AI component calling device provided by the application has the same beneficial effects as the AI component calling method provided by the above-mentioned embodiments, and the other technical features in the AI component calling device are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0177] The application provides an AI component calling device, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the AI component calling method in the above-mentioned embodiment one.

[0178] The following will be described with reference toFigure 5 FIG. 1 illustrates a structural diagram of an AI component calling device suitable for implementing embodiments of the present application. The AI component calling device in embodiments of the present application can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Portable Application Description), a PMP (Portable Media Player), a car terminal (e.g., a car navigation terminal), and the like, as well as a stationary terminal such as a digital TV, a desktop computer, and the like. The device illustrated in the figure is merely an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0179] As illustrated, the AI component calling device can include a processing device 1001 (e.g., a central processor, a graphic processor, etc.) that can perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for device operation are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An I / O (Input / Output) interface 1006 is also connected to the bus. In general, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output device 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, and the like; the storage device 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 1009. The communication device 1009 can allow the AI component calling device to communicate with other devices wirelessly or by wire to exchange data. Although the AI component calling device having various systems is illustrated in the figure, it should be understood that all the illustrated systems are not required to be implemented or provided. More or less systems can be alternatively implemented or provided.

[0180] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0181] The AI component calling device provided by the present application adopts the AI component calling method in the above-mentioned embodiments, which can improve the calling efficiency of the large model application in calling each AI component. Compared with the prior art, the AI component calling device provided by the present application has the same beneficial effects as the AI component calling method provided by the above-mentioned embodiments, and other technical features in the AI component calling device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0182] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0183] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0184] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the AI component calling method in the above-mentioned embodiments.

[0185] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), and the like, or any suitable combination of the above.

[0186] The above computer readable storage medium can be contained in the AI component calling device, or can exist separately without being assembled into the AI component calling device.

[0187] The above computer readable storage medium carries one or more programs, which, when executed by the AI component calling device, enable the AI component calling device to perform the step flow of the above AI component calling method.

[0188] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through an Internet service provider to connect through the Internet).

[0189] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0190] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0191] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the AI component calling method described above, and can improve the calling efficiency of the large model application in calling each AI component. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the AI component calling method provided by the above embodiments, which will not be repeated here.

[0192] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the AI component calling method as described above.

[0193] The computer program product provided by the present application can improve the calling efficiency of the large model application in calling each AI component. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the AI component calling method provided by the above embodiments, which will not be repeated here.

[0194] The above is only some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for invoking an AI component, characterized in that, The method includes: When a first call request sent by a large model application through a gateway is detected, the AI ​​component container instance corresponding to the first call request is determined as the target AI component container instance. The AI ​​component container instance is a container instance obtained by containerizing and encapsulating a separately deployable AI component. The AI ​​component container instance includes model files and a protocol adapter. The AI ​​component is obtained by decoupling a monolithic AI capability component. Containerizing and encapsulating the AI ​​component includes: determining the AI ​​component container image corresponding to the AI ​​component; deploying a declaration file corresponding to the AI ​​component container image in a preset container orchestration platform; determining the AI ​​component container instance to be registered for service and published in the container orchestration platform based on the deployed declaration file; and performing service registration and metadata publication on the AI ​​component container instance to be registered for service and published in the container orchestration platform. The first call request is converted into a second call request by the protocol adapter in the target AI component container instance according to a preset protocol format. Based on the second invocation request, the model corresponding to the model file in the target AI component container instance is invoked to perform task processing and obtain the processing result; The processing result is encapsulated into a response message by the protocol adapter in the target AI component container instance according to the preset protocol format, and then sent to the large model application through the gateway.

2. The AI ​​component invocation method as described in claim 1, characterized in that, The AI ​​component invocation method also includes: Decouple multiple monolithic architectures containing AI capability components to obtain multiple AI components that can be deployed independently; Determine the dependency file for each AI component, which contains at least one dependency, wherein the dependency includes at least one of the following: model file, inference engine library, operating system package, protocol adapter, and health monitoring script of the AI ​​component; For each AI component, the AI ​​component and its dependency files are containerized and encapsulated to obtain an AI component container instance.

3. The AI ​​component invocation method as described in claim 2, characterized in that, The step of containerizing and encapsulating the AI ​​component and its dependency files to obtain an AI component container instance includes: The AI ​​component and at least one dependency in the dependency file are encapsulated into a preset container image to obtain the AI ​​component container image; Determine the declaration file corresponding to the AI ​​component container image, and deploy the declaration file of the AI ​​component container image in the preset container orchestration platform; Based on the deployed declaration file, the AI ​​component container instance to be registered for services and published for metadata in the container orchestration platform is determined. The AI ​​component container instance is registered for services and published for metadata in the container orchestration platform so that large model applications can call the AI ​​component container instance for task processing through the gateway.

4. The AI ​​component invocation method as described in claim 3, characterized in that, The preset container image includes two different first base images and second base images. The step of encapsulating the AI ​​component and its dependency files into the preset container image to obtain the AI ​​component container image includes: During the compilation environment stage, the AI ​​component and at least one dependency from the dependency file are copied to the first base image to obtain a temporary build image. In the first base image, a preset Python library and operating system package are installed, the inference engine in the inference engine library is downloaded and compiled to obtain a compiled inference engine binary file, and the model file is preheated according to the inference engine corresponding to the compiled inference engine binary file to obtain a preheating marker file. During the runtime environment phase, if the preheating status corresponding to the preheating marker file is preheating complete, the model file, protocol adapter, Python library, and compiled inference engine binary file in the temporary build image are copied to the second base image to obtain the AI ​​component container image.

5. The AI ​​component invocation method as described in claim 3, characterized in that, The step of determining the declaration file corresponding to the AI ​​component container image includes: The computing resources required by the AI ​​component container instance are determined according to the preset resource configuration rules. The resource configuration rules include scaling rules determined based on the actual load of the computing resources and preset load thresholds, computing resource support rules for at least two AI component container instances, and node affinity and taint tolerance rules for the computing resources. The declaration file is determined based on the computing resources, wherein the declaration file includes a specified AI component container image, a number of replicas, and computing resources, and the number of replicas is the number of instances of the AI ​​component container instance.

6. The AI ​​component invocation method as described in any one of claims 1-5, characterized in that, The AI ​​component invocation method also includes: When the large model application calls the target AI component container instance to process tasks, the service status corresponding to the target AI component container instance is monitored in real time through a preset container probe; When an abnormal service status is detected, the task processing operation of the target AI component container instance is stopped, and a new target AI component container instance is reconstructed to continue task processing based on the new target AI component container instance.

7. The AI ​​component invocation method as described in any one of claims 1-5, characterized in that, The AI ​​component invocation method also includes: The first call request is received by the AI ​​component container instance corresponding to the first call request from the gateway according to a preset intelligent routing decision logic. The preset intelligent routing decision logic includes the gateway filtering multiple available first AI component container instances in a preset registry center based on the first call request sent by the large model application, and selecting the first AI component container instance with the best performance from the multiple first AI component container instances as the AI ​​component container instance corresponding to the first call request based on real-time monitoring data for at least one first AI component container instance and preset service quality requirements, and sending the first call request to the AI ​​component container instance corresponding to the first call request.

8. An AI component calling device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the AI ​​component invoking method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the AI ​​component invocation method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the AI ​​component invocation method as described in any one of claims 1 to 7.

Images