RAG-oriented embedded service elastic deployment method

Abstract

The application discloses a RAG-oriented embedded service elastic deployment method, and belongs to the technical field of computer resource configuration. The method firstly acquires an embedded model suitable for a RAG scene and constructs an embedded service, tests different resource configuration parameters in batches through automatic resource analysis, and screens optimal resource configuration meeting service quality requirements; secondly, integrates RPS monitoring and a dynamic batch processing mechanism, combines a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy; then triggers a hybrid expansion and contraction mechanism according to inference request load to form cooperative scheduling; finally, distributes inference requests through a polling mechanism and continuously optimizes service performance. Through the cooperative action of automatic resource analysis, GPU resource fragmentation optimization and a hybrid expansion and contraction strategy, the application can significantly improve GPU resource utilization and the performance of the embedded service, and effectively supports high-concurrency and low-latency inference requirements in the RAG scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer resource configuration, in particular to an embedding service elastic deployment method for RAG. BACKGROUND

[0002] With the rapid development of artificial intelligence large models and retrieval-augmented generation (RAG) technology, embedding services have become the core functional components of RAG systems. Their core role is to convert large-scale unstructured data into vector representations, providing a foundation for downstream semantic retrieval, intelligent question answering, and other tasks. The performance and scalability of embedding services directly affect the overall effectiveness and user experience of RAG systems.

[0003] The embedding models relied upon by embedding services have limited GPU resource requirements during runtime and do not require exclusive use of entire GPU devices. In existing embedding service deployments, static GPU resource configurations are not dynamically adapted to hardware environments, model characteristics, and business needs, resulting in a large amount of GPU resources being idle and low utilization rates. In multi-instance deployment scenarios, existing solutions lack intelligent scheduling strategies, and the random allocation of instances can easily cause GPU resource fragmentation. Some devices are overloaded due to instance clustering, while others are idle for long periods of time, further exacerbating resource waste. When sudden load requests arrive, the newly added instances need to go through model loading, cache initialization, and other cold start processes, making it difficult to quickly handle sudden traffic and effectively respond to load changes, which cannot meet the high concurrency and low latency service requirements in RAG scenarios.

[0004] The existing embedding service deployment scheme has low GPU resource utilization, severe resource fragmentation, and delayed load response, among other issues, necessitating an embedding service elastic deployment method for RAG. SUMMARY

[0005] The present application aims to provide an embedding service elastic deployment method for RAG that can significantly improve GPU resource utilization and the performance of embedding services through automated resource analysis, GPU resource fragmentation optimization, and the coordinated use of hybrid scaling strategies, effectively supporting high concurrency and low latency inference requirements in RAG scenarios.

[0006] Technical solution: The embedding service elastic deployment method for RAG of the present application includes the following steps:

[0007] Obtain embedding models suitable for RAG scenarios and construct embedding services that include model loading and inference execution to process inference requests;

[0008] Perform automated resource analysis on embedded services, batch test different resource configuration parameters, calculate performance indicators, and select the parameter combination that meets service quality requirements and maximizes performance indicators as the optimal resource configuration.

[0009] Inject the optimal resource configuration into the embedded service, monitor changes in service load, execute dynamic batch processing threads based on dual trigger conditions, provide vectorized interfaces to the outside world, compile and generate GPU control shared libraries, configure the system's shared library preload list, and build the embedded service image accordingly.

[0010] Unified allocation of port resources, combined with GPU resource fragmentation optimization algorithm and three-level candidate GPU scheduling strategy to intelligently allocate GPU resources, and deploy embedded service instances based on container technology;

[0011] Hybrid scaling is performed based on the embedded inference request load, dynamically adjusting the number of embedded service instances to achieve horizontal scaling.

[0012] When a sudden load occurs, the GPU resource allocation of existing ready instances is adjusted synchronously in conjunction with the vertical scaling strategy to form a collaborative scheduling;

[0013] Inference requests are distributed through a polling scheduling mechanism, and data on the operation and resource consumption of embedded service instances are collected in real time. The system's GPU resource utilization and embedded service performance are continuously optimized based on a hybrid scaling strategy, and the optimization results are used for subsequent GPU resource allocation and scaling decisions.

[0014] Furthermore, the steps of automating resource analysis for embedded services, batch testing different resource configuration parameters, calculating performance indicators, and selecting the parameter combination that meets service quality requirements and maximizes performance indicators as the optimal resource configuration include:

[0015] The computational efficiency index, used to evaluate the rationality of resource allocation, is formulated as follows:

[0016] ,

[0017] In the formula, This represents the total number of inference texts that the embedded service can successfully process per unit of time. The percentage of streaming multiprocessors;

[0018] Initialize parameters, including setting the inference latency threshold, streaming multiprocessor ratio, and batch size.

[0019] Perform inference tests on the embedded service, and record parameters such as streaming multiprocessor ratio, batch size, inference time, and throughput.

[0020] The resource analysis adopts a parameter search strategy with inference delay threshold constraint, adjusts the stream multiprocessor occupancy and batch size parameters, and takes the efficiency index as the target to select the configuration parameters with the maximum efficiency index under the condition of meeting the inference time The configuration parameters with the maximum efficiency index under the condition of meeting the inference delay threshold are selected as the optimal resource configuration.

[0021] Further, while monitoring the service load changes, the steps of executing dynamic batch processing threads according to the double triggering conditions include:

[0022] A monitoring thread is created, which is periodically running, and the inference frequency is counted and the number of requests per second is calculated in each period;

[0023] A dynamic batch processing thread is created, and two conditions are used to determine when to start processing: one is to reach the injected batch size parameter, and the other is that the request waiting time in the queue exceeds the waiting threshold;

[0024] The dynamic batch processing thread checks the two conditions through a loop, and when any condition is met, the current batch of requests is extracted from the global queue, batch inference is performed, and the inference result is returned through the independent result queue bound to each request.

[0025] Further, the port resources are uniformly allocated, the GPU resource fragmentation optimization algorithm and the three-level candidate GPU scheduling strategy are combined to intelligently allocate GPU resources, and the steps of deploying embedded service instances based on container technology include:

[0026] The GPU resource fragmentation score is calculated, and the formula is:

[0027] ,

[0028] In the formula, is the stream multiprocessor fragmentation weight coefficient, β is the video memory fragmentation weight coefficient, , and ; SMU is the allocated SM request amount, is the SM number requested by the instance, is the total SM number of the GPU, and SM represents the stream multiprocessor; is the currently allocated video memory capacity, is the video memory capacity requested by the instance, is the total video memory capacity of the GPU;

[0029] A port resource pool containing a preset continuous range is constructed, a mutex lock is used to protect the port allocation operation, and a port is automatically allocated when the embedded service instance is created, and the port is automatically recycled when the instance is deleted;

[0030] A three-level candidate GPU scheduling strategy is adopted: first, find the co-located GPU that has deployed the same service instance as the first candidate set; if the co-located GPU does not exist or the resources are insufficient, then select the active GPU that carries other services as the second candidate set; if the active GPU does not exist or the resources are insufficient, then select the idle GPU that does not carry any instance as the third candidate set;

[0031] For the GPUs in each candidate set, first check whether the resource requirements are met, if met, then perform GPU resource fragmentation scoring, select the GPU in the candidate set that meets the current instance resource requirements and has the lowest fragmentation score to deploy the instance, and inject the SMP parameters in the optimal resource configuration into the container environment variable.

[0032] Further, according to the embedded inference request load, perform hybrid scaling, dynamically adjust the number of embedded service instances, and realize the steps of horizontal scaling, which include:

[0033] The scaling threshold is calculated, and the formula is:

[0034] ,

[0035] The scaling threshold is calculated, and the formula is:

[0036] ,

[0037] In the formula, is the number of existing instances of the embedded service that have been successfully deployed and are in normal operation, is the maximum number of inference requests that a single instance of the embedded service can handle in a unit of time;

[0038] Real-time monitoring of the embedded service inference request load, load threshold checking, when the load reaches the scaling threshold, performing horizontal scaling to increase the number of embedded service instances; when the load reaches the scaling threshold, performing horizontal scaling to reduce the number of embedded service instances; when the load is normal, keeping the current service instance unchanged;

[0039] Through the Docker container management mechanism, dynamically create or destroy embedded service instances.

[0040] Further, when the load reaches the scaling threshold, the steps of performing horizontal scaling include:

[0041] Create an embedded service instance through a container creation command, first pass through a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy to select a GPU that meets the current instance resource requirements and has the lowest fragmentation score for deployment, simultaneously call the created embedded service image, inject the SMP parameters in the optimal resource configuration into the container environment variable, and obtain an idle port from the port pool, after the instance health check passes and is registered to the service management center, complete the scaling.

[0042] Further, the step of performing horizontal scaling down when the load reaches the scaling down threshold comprises:

[0043] Select the last instance from the embedded service instance managed by the service management center, first remove the traffic distribution from the request distribution layer, and then destroy the container through the container deletion command after the remaining request processing is completed, release the port and GPU resources, and unregister the instance message from the service management center, realize the scaling down.

[0044] Further, the step of synchronously cooperating with the vertical scaling strategy to adjust the GPU resource allocation of the existing ready instance when the load bursts comprises:

[0045] When the burst load request arrives, the system triggers horizontal scaling and starts a new instance at the same time, and the vertical scaling strategy performs related operations on the existing ready instance, preloads the GPU control shared library in the container through the user mode library preloading mechanism, and the shared library hijacks the related call of the unified computing device architecture kernel through the hook technology.

[0046] Advantages: compared with the prior art, the present application has the following advantages:

[0047] 1. The present application adopts automatic resource analysis, collects performance indicators such as inference time and throughput by batch testing different flow multi-processor proportion and batch size parameter combinations, and automatically selects the parameter combination that meets the service quality requirements and has the maximum efficiency indicator as the optimal resource configuration, which significantly improves the accuracy and efficiency of resource configuration, realizes fine deployment of embedded services under the condition of meeting performance and resource requirements, and improves the GPU resource utilization rate;

[0048] 2. The present application adopts a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy, preferentially selects a GPU with low fragmentation degree for resource allocation by calculating the resource fragmentation score, and applies for resources from the target GPU node according to the GPU resource fragmentation score algorithm when a new embedded service instance is added, thereby reducing the waste of GPU resources;

[0049] 3. The present application adopts a hybrid scaling strategy, which combines horizontal scaling (adjusting the number of service instances) and vertical scaling (adjusting the GPU resource allocation of a single instance), compared with the traditional single scaling method, the cooperative scheduling of horizontal scaling and vertical scaling strategy effectively guarantees the rapid response of embedded services under burst load, and solves the problem of service response lag under the traditional strategy. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 It is a flowchart of a kind of RAG-oriented embedded service elastic deployment method;

[0051] Figure 2A flowchart for automated resource analysis of embedded services;

[0052] Figure 3 A flowchart for GPU resource fragmentation optimization algorithm and three-level candidate GPU scheduling strategy;

[0053] Figure 4 A flowchart for performing hybrid scaling according to embedded service inference request load;

[0054] Figure 5 Efficiency indicators of embedded services for embedded model construction under the condition of meeting service quality requirements The curve diagram changes with the parameters SM ratio, batch size. DETAILED DESCRIPTION

[0055] The embodiments of the present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, and not to limit the embodiments of the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the embodiments of the present application are shown in the drawings, not all structures.

[0056] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as target system architecture, techniques, in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed description of well-known systems, devices, circuits and methods is omitted to avoid unnecessary details that hinder the description of the present application.

[0057] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, whole, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0058] It should also be understood that the term "and / or" used in the specification and appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0059] In addition, in the description of the present application and the appended claims, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0061] The flowchart of the elastic deployment method for embedded services for RAG described in this embodiment is as follows. Figure 1 As shown, it includes the following steps:

[0062] Step 1: Obtain an embedded model suitable for the RAG scenario and build an embedded service that includes model loading and inference execution to handle inference requests.

[0063] Obtain an embedding model suitable for the RAG scenario, such as a BGE series text embedding model like BGE-large-zh-v1.5 or BGE-m3. For a given embedding model, load the embedding model, model configuration, and tokenizer using the Transformers library, and load the embedding model onto the GPU device. Build an embedding service to execute inference requests, converting text data into vectors.

[0064] Step 2: Perform automated resource analysis on the embedded service, batch test different resource configuration parameters, calculate performance indicators, and select the parameter combination that meets the service quality requirements and maximizes the performance indicators as the optimal resource configuration.

[0065] During the inference execution phase, a gradient-free computation mode is used to perform vectorized inference. A preset pooling strategy is used to select the last hidden state of the first tag as the sentence embedding. At the same time, the percentage of streaming multiprocessors (SM), batch size, inference time and throughput are recorded to support performance analysis.

[0066] Furthermore, the steps of automating resource analysis for embedded services, batch testing different resource configuration parameters, calculating performance indicators, and selecting the parameter combination that meets service quality requirements and maximizes performance indicators as the optimal resource configuration include:

[0067] The computational efficiency index, used to evaluate the rationality of resource allocation, is formulated as follows:

[0068] ,

[0069] In the formula, This represents the total number of inference texts that the embedded service can successfully process per unit of time. For stream multiprocessor occupancy ratio;

[0070] Initialize parameters, including setting inference delay threshold, stream multiprocessor occupancy ratio parameter and batch size parameter;

[0071] Perform inference test of embedded service, record stream multiprocessor occupancy ratio parameter, batch size parameter, inference time and throughput;

[0072] Resource analysis adopts parameter search strategy constrained by inference delay threshold, adjusts stream multiprocessor occupancy and batch size parameter, and takes efficiency index as target, selects configuration parameter with maximum efficiency index under the condition of meeting inference time delay threshold as optimal resource configuration, wherein the configuration parameter includes stream multiprocessor occupancy ratio and batch size.

[0073] In one example, as shown in Figure 2 Initialize parameters, set inference delay threshold, SM occupancy ratio parameter (SMP), batch size parameter (batch_size), and specify SMP and batch_size range, SMP test range is 10%-100%, and batch_size test range is 1-64. Perform initial search, fix batch_size as 1, gradually increase SMP by step of 10%, perform inference test of embedded service and record inference time and other data, if inference time delay threshold meets service quality requirement, find first SMP meeting service quality requirement, and enter dynamic optimization cycle phase. In dynamic optimization cycle phase, perform the following:

[0074] Perform inference test of embedded service based on current SMP and batch_size, record SMP, batch_size, inference time, throughput at this time and calculate ;

[0075] When service quality requirement is met, update optimal configuration, gradually increase batch_size by step of 2 times of current value, continue to perform inference test; when service quality requirement is violated, update optimal configuration, gradually increase SM occupancy ratio by step of 10%, continue to perform inference test;

[0076] If batch_size and SMP reach upper limit or current is 20% lower than last time , terminate cycle and output optimal configuration.

[0077] Through this automatic resource analysis, the system can automatically find the optimal resource configuration combination under the premise of ensuring the quality of service, maximize the throughput under the delay constraint, and finally output the optimal SMP and batch_size configuration that meets the quality of service requirements.

[0078] Due to the model size and computational demand characteristics of the embedded service, it usually does not need to occupy the complete GPU resources. Compared with the static GPU deployment of the embedded service, this step provides the optimal resource configuration parameters for the subsequent deployment of the embedded service, enabling the fine-grained deployment of the embedded service under the performance and resource demand, and improving the GPU resource utilization.

[0079] Step 3: Inject the optimal resource configuration into the embedded service, and simultaneously monitor the service load changes in real time. Based on the double triggering conditions, execute the dynamic batch processing thread, provide the vectorization interface to the outside, compile the GPU control shared library, configure the shared library preload list of the system, and build the embedded service image accordingly.

[0080] Inject the batch size parameter in the optimal resource configuration into the embedded service.

[0081] Create a monitoring thread to implement the requests per second (RPS) monitoring function. The thread runs continuously at a configurable monitoring period, and in each period, it counts the number of inferences and calculates the requests per second. This monitoring mechanism uses a non-blocking design and includes shared data access through a mutex to ensure the real-time and consistency of the monitoring data.

[0082] Implement the dynamic batch processing mechanism through double triggering conditions. Use a global request queue to store the embedded inference requests to be processed. Create a dynamic batch processing thread that determines when to start processing based on two conditions: one is that the size of the injected batch parameter is reached, and the other is that the waiting time of the requests in the queue exceeds the waiting threshold dynamically generated by the RPS monitoring thread. This waiting threshold is calculated in real time based on the current RPS and the historical average inference time. The dynamic batch processing thread checks the above two conditions through a loop, and when either condition is met, it extracts the current batch of requests from the global queue, executes batch inference, and returns the inference results through the independent result queue bound to each request, achieving efficient processing and accurate response of the requests.

[0083] Implement the vectorization interface based on the HTTP protocol, receive JSON format text input, support single text or text list, add the text to the batch processing system through the global request queue, create an independent result queue for each text, and return the corresponding embedded vector result after batch processing is completed.

[0084] The compilation generates a core shared library containing GPU usage monitoring and limiting logic, and then generates multiple library file copies with different names through a copy operation. The paths of these library files are written into a system shared library preloading configuration file, and through the preloading mechanism of the dynamic link library, it is ensured that the control library is preferentially loaded when the embedded service starts, so as to realize independent regulation and control of the GPU resources of each instance.

[0085] Based on the embedded service code, its dependent runtime package, GPU control shared library and image construction script, an embedded service image is constructed, and the embedded service image is called based on the container deployment to facilitate the standardization and automation of instance creation, which can not only ensure that all embedded service instances are started based on unified configuration, but also quickly connect the dynamic scaling process of the container management system, while reducing the adaptation cost of cross-environment deployment.

[0086] By using a multi-thread architecture, parallel running of RPS monitoring, dynamic batch processing and web service is realized, and data synchronization is realized through inter-thread shared memory and mutex mechanism -- the web service thread is responsible for receiving external requests and writing into a global queue, the dynamic batch processing thread independently executes batch reasoning logic, the RPS monitoring thread collects performance data in real time and dynamically adjusts the waiting threshold, and the three cooperate to ensure the high-concurrency processing capability of the embedded service, fully meeting the real-time requirements of the RAG system for the embedded service.

[0087] Step 4, uniformly allocate port resources, intelligently allocate GPU resources in combination with a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy, and deploy the embedded service instance based on container technology.

[0088] Further, the step of uniformly allocating port resources, intelligently allocating GPU resources in combination with a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy, and deploying the embedded service instance based on container technology includes:

[0089] The calculation of the GPU resource fragmentation score is as follows:

[0090] ,

[0091] In the formula, is the stream multiprocessor fragmentation weight coefficient, β is the video memory fragmentation weight coefficient, , and ; SMU is the currently allocated SM request amount, is the SM amount requested by the instance, is the total SM amount of the GPU, and SM represents a stream multiprocessor; is the currently allocated video memory capacity, is the video memory capacity requested by the instance, is the total video memory capacity of the GPU;

[0092] Construct a port resource pool containing a preset continuous range (such as a port size range of 15000-20000), use mutex locks to protect port allocation operations, automatically allocate ports when service instances are created, and automatically reclaim ports when instances are deleted, so as to achieve centralized management and efficient allocation of port resources and solve the port conflict problem in multi-instance concurrent deployment scenarios.

[0093] like Figure 3 As shown, a three-level candidate GPU scheduling strategy is adopted: first, co-located GPUs that have deployed the same service instance are searched as the first candidate set; if the co-located GPUs do not exist or have insufficient resources, active GPUs that carry other services are selected as the second candidate set; if active GPUs do not exist or have insufficient resources, idle GPUs that have never carried any instance are selected as the third candidate set.

[0094] For each GPU in the candidate set, first check whether the resource requirements are met. If they are met, perform GPU resource fragmentation scoring, select the GPU in the candidate set that meets the current instance's resource requirements and has the lowest fragmentation score to deploy the instance, and inject the SMP parameters from the optimal resource configuration into the container environment variables.

[0095] In one example, embedded services are typically computationally intensive tasks in large-scale batch processing scenarios. In computationally intensive tasks, the system tends to optimize SM fragmentation during resource allocation. Therefore, in the GPU resource fragmentation scoring formula, let... ,like Set to 0.6, Set it to 0.4.

[0096] Traditional embedded service deployments often employ a single instance to exclusively utilize a GPU or randomly select an idle GPU, neglecting resource fragmentation. This results in underutilization of SM (Software Provider Interface) or GPU memory resources due to fragmented allocation, and resource waste occurs when multiple instances of the same service are deployed in a dispersed manner. By using a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy to intelligently allocate GPU resources, when deploying a new embedded service instance, the system first selects a shared GPU already deployed for the same service, and then chooses the optimal allocation scheme based on fragmentation scoring. This significantly improves GPU resource utilization and reduces GPU resource waste.

[0097] Step 5: Perform hybrid scaling based on the embedded inference request load, dynamically adjust the number of embedded service instances, and achieve horizontal scaling.

[0098] Furthermore, the steps for performing hybrid scaling based on embedded inference request load, dynamically adjusting the number of embedded service instances, and achieving horizontal scaling include:

[0099] The expansion threshold is calculated using the following formula:

[0100] ,

[0101] The shrinkage threshold is calculated by the formula:

[0102] ,

[0103] wherein, is the number of existing instances of the embedded service that are currently successfully deployed and in normal operation, is the maximum number of inference requests that a single instance of the embedded service can handle in a unit of time;

[0104] Real-time monitoring of the inference request load of the embedded service is performed, and load threshold checking is performed. When the load reaches the expansion threshold, horizontal expansion is performed to increase the number of instances of the embedded service. When the load reaches the shrinkage threshold, horizontal shrinkage is performed to reduce the number of instances of the embedded service. When the load is normal, the current number of service instances is maintained;

[0105] The embedded service instances are dynamically created or destroyed through the Docker container management mechanism.

[0106] Further, the step of performing horizontal expansion when the load reaches the expansion threshold includes:

[0107] An embedded service instance is created through a container creation command. Through a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy, a GPU that meets the current instance resource requirements and has the lowest fragmentation score is selected for deployment. At the same time, an embedded service image that has been created is called, the SMP parameters in the optimal resource configuration are injected into the container environment variables, an idle port is obtained from a port pool, and after the instance passes the health check and is registered to the service management center, the expansion is completed.

[0108] Further, the step of performing horizontal shrinkage when the load reaches the shrinkage threshold includes:

[0109] The last instance is selected from the embedded service instances managed by the service management center. First, the traffic distribution is removed from the request distribution layer. After the remaining requests are processed, the container is destroyed through a container deletion command, the port and GPU resources are released, and the instance message is unregistered from the service management center, thereby achieving shrinkage.

[0110] Step 6, when the load bursts, the vertical expansion and shrinkage strategy is synchronized to adjust the GPU resource allocation of the existing ready instances, forming a coordinated scheduling.

[0111] When horizontal expansion is triggered, if a burst load request arrives, in order to cope with the performance fluctuations caused by the cold start of the new instance, the vertical expansion and shrinkage is synchronously executed to adjust the GPU resource allocation of the existing instances, quickly improve the processing capacity of the existing instances, and realize the coordinated scheduling of horizontal and vertical expansion and shrinkage.

[0112] Further, the step of adjusting the GPU resource allocation of the existing ready instance in coordination with the load burst includes:

[0113] Deploy a CUDA kernel manager as the central node of resource scheduling through Docker. Before performing the hybrid scaling operation, the startup state of the kernel manager needs to be verified to ensure that it can normally receive scheduling instructions and complete communication with the shared library of each embedded service instance.

[0114] When a burst load request arrives, the system triggers horizontal scaling and starts a new instance, and at the same time, the vertical scaling strategy performs related operations on the existing ready instance. The GPU control shared library is preloaded in the container through the user mode library preloading mechanism, and the shared library hijacks the CUDA kernel startup related call through the hook technology.

[0115] Specifically, when the embedded service instance executes an inference request and generates a CUDA kernel function call, the shared library will implement timing scheduling and frequency control on the kernel startup according to the real-time CUDA kernel manager issued by the fixed period of kernel execution frequency quota. When the quota is sufficient, the kernel is allowed to be submitted immediately, and when the quota is insufficient, time slice waiting is triggered. Without changing the static division of stream multiprocessors or memory, the GPU computing intensity of a single instance is dynamically constrained in a time slice limited manner to achieve fine-grained resource allocation.

[0116] As shown in Figure 4 When a burst load request arrives, the system triggers horizontal scaling and starts a new instance, and at the same time, the vertical control performs temporary quota adjustment on the existing ready instance: first, it is evaluated whether the existing quota meets the immediate demand, and if not, the CUDA kernel manager increases the kernel execution frequency quota of the instance, and the CUDA hijacked client (i.e., the embedded service instance process loaded with the GPU control shared library) reduces the time slice waiting and increases the kernel submission rate to quickly raise the immediate processing capacity accordingly; after the new instance passes the health check and is connected to the load, the temporary quota is smoothly returned to normal according to the strategy.

[0117] Traditional embedded service deployment often uses a single scaling strategy, which only adjusts the number of instances horizontally to cope with load changes, and the startup of a new instance needs to go through the cold start process of model loading, cache initialization, etc. When a burst load request arrives, the existing ready instance cannot temporarily increase the processing capacity due to the fixed resource quota, resulting in a sharp increase in service response delay. By using the hybrid scaling in this example, the horizontal and vertical scaling strategies are cooperatively scheduled, effectively ensuring the fast response of the embedded service under burst load, and solving the problem of service response lag under the traditional strategy.

[0118] Step 7, distribute inference requests through a polling scheduling mechanism, collect embedded service instance running and resource consumption data in real time, continuously optimize system GPU resource utilization and embedded service performance according to the hybrid scaling strategy, and use the optimization results for subsequent GPU resource allocation and scaling decisions.

[0119] After the deployment of the embedded service, a polling index is maintained for each embedded service. When an inference request is received, the index is incremented and modulo is used to achieve a circular allocation strategy. Through this mechanism, load balancing of requests is achieved, avoiding the increase of delay caused by request overload of a single instance, and ensuring the balance of resource utilization of each instance. When distributing requests, only service instances in a ready state are selected for request distribution to ensure that requests can be processed normally.

[0120] The number of requests for each service instance is counted in real time through Flask routing and recorded in the service state. Based on the hybrid scaling strategy, the system supports scaling decisions by maintaining 40 historical data points. When scaling operations are triggered, the corresponding service's historical records are cleared and data collection starts again to ensure the timeliness of the decision. Through this dynamic adjustment mechanism, the system can maximize GPU resource utilization while ensuring the performance of embedded services, and the optimization results are fed back to subsequent GPU resource allocation and scaling decisions.

[0121] To further illustrate the effectiveness and superiority of the RAG-oriented embedded service elastic deployment method described in the present application, the following examples are provided. The embedded model is Bge-large-zh-v1.5, and the embedded service is constructed. On a 1-machine 4-card server with 4 NVIDIA L40S GPUs, the embedded service is automatically analyzed, and the hybrid scaling strategy is combined Figure 5 When the SM occupancy is 30% and the batch size is 8, it is the relatively optimal resource configuration for the service. Based on this configuration, a single L40S GPU can theoretically deploy 3 instances of the embedded service. The present application achieves single-card multi-instance deployment through fine-grained resource allocation, ensuring that each instance meets the quality of service requirements. On this server, a load test of 1000 requests and 500 concurrent requests is performed on the embedded service, and compared with the traditional single-card single-instance, the performance comparison results are shown in Table 1. It can be seen that the present application not only improves the GPU resource utilization from about 30% to more than 90%, but also reduces the waste of GPU resources through resource integration. At the same time, under the same hardware investment, the concurrent processing capacity of the embedded service is improved to 1.5 times that of the original, fully verifying the significant advantages of the present application in resource elastic scheduling and efficiency optimization.

[0122] Table 1 Performance comparison results

[0123]

[0124] To sum up, the application provides a RAG-oriented embedded service elastic deployment method, which effectively solves the problems of resource fragmentation, low utilization, delay fluctuation in high concurrency scenarios and the like in traditional deployment through automatic resource analysis, GPU resource fragmentation optimization algorithm and hybrid expansion and contraction strategy, saves GPU resources, improves GPU resource utilization and response performance of the embedded service. Finally, the performance of the embedded service is optimized to provide efficient and stable vector calculation support for downstream semantic retrieval, intelligent question answering and the like, and improve the overall effect and user experience of the RAG system.

[0125] The embodiments of the application are described in detail above with reference to the drawings, but the application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A method for elastic deployment of embedded services for RAG, characterized in that, Includes the following steps: Obtain an embedded model suitable for the RAG scenario, and build an embedded service that includes model loading and inference execution to handle inference requests; Perform automated resource analysis on embedded services, batch test different resource configuration parameters, calculate performance indicators, and select the parameter combination that meets service quality requirements and maximizes performance indicators as the optimal resource configuration. Inject the optimal resource configuration into the embedded service, monitor changes in service load, execute dynamic batch processing threads based on dual trigger conditions, provide vectorized interfaces to the outside world, compile and generate GPU control shared libraries, configure the system's shared library preload list, and build the embedded service image accordingly. Unified allocation of port resources, combined with GPU resource fragmentation optimization algorithm and three-level candidate GPU scheduling strategy to intelligently allocate GPU resources, and deploy embedded service instances based on container technology; Hybrid scaling is performed based on the embedded inference request load, dynamically adjusting the number of embedded service instances to achieve horizontal scaling. When a sudden load occurs, the GPU resource allocation of existing ready instances is adjusted synchronously in conjunction with the vertical scaling strategy to form a collaborative scheduling; Inference requests are distributed through a polling scheduling mechanism, and data on the operation and resource consumption of embedded service instances are collected in real time. The system's GPU resource utilization and embedded service performance are continuously optimized according to a hybrid scaling strategy, and the optimization results are used for subsequent GPU resource allocation and scaling decisions. The steps for automating resource analysis of embedded services, batch testing of different resource configuration parameters, calculating performance metrics, and selecting the parameter combination that meets service quality requirements and maximizes performance metrics as the optimal resource configuration include: The computational efficiency index, used to evaluate the rationality of resource allocation, is formulated as follows: ， In the formula, This represents the total number of inference texts that the embedded service can successfully process per unit of time. The percentage of streaming multiprocessors; Initialize parameters, including setting the inference latency threshold, streaming multiprocessor ratio, and batch size. Perform inference tests on the embedded service, and record parameters such as streaming multiprocessor ratio, batch size, inference time, and throughput. Resource analysis employs a parameter search strategy with inference latency threshold constraints. This involves adjusting parameters such as streaming multiprocessor usage and batch size, with efficiency as the objective, while meeting inference time requirements. Under the condition of inference delay threshold, the configuration parameter that maximizes the efficiency index is selected as the optimal resource configuration.

2. The method for elastic deployment of embedded services for RAG according to claim 1, characterized in that, Simultaneously monitoring service load changes and executing dynamic batch processing threads based on dual trigger conditions includes the following steps: Create a monitoring thread that runs continuously throughout the monitoring cycle, counting the number of inferences and calculating the number of requests per second in each cycle. Create a dynamic batch processing thread and determine when to start processing based on two conditions: first, the injected batch size parameter is reached; second, the waiting time of requests entering the queue exceeds the waiting threshold. The dynamic batch processing thread checks two conditions in a loop. When either condition is met, it retrieves the current batch request from the global queue, performs batch inference, and returns the inference results through an independent result queue bound to each request.

3. The method for elastic deployment of embedded services for RAG according to claim 2, characterized in that, The steps for uniformly allocating port resources, intelligently allocating GPU resources by combining GPU resource fragmentation optimization algorithms and a three-level candidate GPU scheduling strategy, and deploying embedded service instances based on container technology include: The formula for calculating GPU resource fragmentation score is: ， In the formula, β is the fragmentation weighting coefficient for streaming multiprocessors, and β is the memory fragmentation weighting coefficient. ,and ; This represents the currently allocated number of SM requests, where SM stands for Streaming Multiprocessor. The number of SMs requested for the instance. This represents the total number of SMs for the GPU. This represents the currently allocated video memory capacity. The requested video memory capacity for the instance. This represents the total video memory capacity of the GPU. Build a port resource pool containing a preset continuous range, use mutex locks to protect port allocation operations, automatically allocate ports when service instances are created, and automatically reclaim ports when instances are deleted; A three-tiered candidate GPU scheduling strategy is adopted: First, co-located GPUs that have deployed the same service instance are selected as the first candidate set; if the co-located GPUs do not exist or have insufficient resources, active GPUs that host other services are selected as the second candidate set; if active GPUs do not exist or have insufficient resources, idle GPUs that have never hosted any instance are selected as the third candidate set. For each GPU in the candidate set, first check whether the resource requirements are met. If they are met, perform GPU resource fragmentation scoring, select the GPU in the candidate set that meets the current instance's resource requirements and has the lowest fragmentation score to deploy the instance, and inject the SMP parameters from the optimal resource configuration into the container environment variables.

4. The method for elastic deployment of embedded services for RAG according to claim 3, characterized in that, The steps for horizontal scaling, which involves performing hybrid scaling based on embedded inference request load and dynamically adjusting the number of embedded service instances, include: The expansion threshold is calculated using the following formula: ， The formula for calculating the reduction threshold is: ， In the formula, This represents the current number of embedded service instances that have been successfully deployed and are in normal operation. The maximum number of inference requests that a single instance of the embedded service can handle per unit of time; Monitor the inference request load of the embedded service in real time, perform load threshold checks, and perform horizontal scaling when the load reaches the scaling threshold to add embedded service instances; perform horizontal scaling when the load reaches the scaling threshold to reduce the number of embedded service instances; when the load is normal, keep the current service instances unchanged. Embedded service instances can be dynamically created or destroyed using Docker container management mechanisms.

5. A method for elastic deployment of embedded services for RAG according to claim 4, characterized in that, When the load reaches the scaling threshold, the steps for performing horizontal scaling include: An embedded service instance is created using the container creation command. First, a GPU resource fragmentation optimization algorithm and a three-level candidate GPU scheduling strategy are used to select the GPU that meets the current instance's resource requirements and has the lowest fragmentation score for deployment. At the same time, the already created embedded service image is called, the SMP parameters in the optimal resource configuration are injected into the container environment variables, and an idle port is obtained from the port pool. After the instance health check is passed and it is registered with the service management center, the expansion is completed.

6. The method for elastic deployment of embedded services for RAG according to claim 5, characterized in that, When the load reaches the scaling-down threshold, the steps for performing horizontal scaling-down include: Select the last instance from the embedded service instances managed by the service management center, remove the traffic distribution from the request distribution layer, destroy the container through the container deletion command after the remaining requests are processed, release the port and GPU resources, and cancel the instance message from the service management center to achieve scaling down.

7. A method for elastic deployment of embedded services for RAG according to claim 6, characterized in that, The steps for adjusting GPU resource allocation for existing ready instances in conjunction with vertical scaling strategies when a sudden load occurs include: When a sudden load request arrives, the system triggers horizontal scaling and starts new instances. At the same time, the vertical scaling strategy performs relevant operations on existing ready instances. Within the container, the GPU control shared library is preloaded through a user-space library preloading mechanism. The shared library uses hook technology to hijack the unified computing device architecture kernel to start relevant calls.

Images