Inference acceleration method for realizing machine learning and large language model based on packaging engine

Abstract

The invention provides an inference acceleration method for realizing machine learning and a large language model based on a packaging engine. The inference acceleration method comprises the following steps: receiving a Boolean parameter enable cache in an inference engine; executing condition judgment logic according to an input enable cache value, and determining whether to execute cache query or not: creating a Triton model warehouse under a specified file path, and establishing an independent folder for each model; in a config.pbtxt file, version information, input and output tensor names, data types and dimensions of the model are defined; the method comprises the following steps of: starting a Triton Inference Server by using a command line tool, and appointing a model warehouse path; the method has the remarkable advantages in the aspects of increasing the reasoning speed, optimizing resource use, improving user experience, reducing cost, enhancing expandability and the like, and powerful support is provided for practical application of a large model.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and machine learning, and in particular to a method for accelerating the reasoning of machine learning and large language models based on a packaging engine. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the increasing application of large language models (LLM) in various industries, fields such as finance, medical care, education, and government affairs have ushered in intelligent transformation. In the financial field, large models can accurately assess credit risks and quickly analyze insurance terms, thereby optimizing customer service. In the medical industry, they are used for case analysis and auxiliary diagnosis, greatly improving doctors' work efficiency and diagnostic accuracy. In the field of education, large models can provide students with personalized learning guidance, recommend suitable learning resources, and promote the development of personalized teaching. In government services, large models improve the efficiency of information query and business processing, and improve user experience. In addition, large models also play an important role in technical scenarios such as intelligent customer service, computer vision, speech recognition, and recommendation systems.

[0003] However, the reasoning process of large models often requires huge computing resources, usually contains hundreds of millions of parameters, and has a complex network structure. This makes the inference process computationally intensive and highly dependent on hardware resources such as CPUs and GPUs. Especially in application scenarios such as real-time data analysis and online question-and-answering, reasoning speed often becomes a key requirement. These scenarios require complex reasoning calculations to be completed within a limited time, but traditional reasoning methods often have difficulty balancing speed and accuracy, resulting in increased reasoning delays, affecting user experience, and limiting the widespread use of large models in practical applications.

[0004] At present, although the technical community has proposed inference acceleration solutions such as model quantization, distillation and pruning, they each have their limitations. Model quantization reduces the model size by converting floating-point parameters into low-bit integers, but may lose some precision. Model distillation reduces computing requirements by training a small model to imitate the output of a large model, but it is usually difficult to reach the level of a large model in terms of performance. Model pruning improves computing efficiency by removing redundant neurons or connections, but this process is often accompanied by performance degradation and the need for additional fine-tuning. Although these existing technologies have alleviated the resource pressure of inference to a certain extent, they are still not enough to solve the real-time and efficiency requirements of large models. Therefore, there is an urgent need for a new inference acceleration method to achieve more efficient computing and meet the growing application needs. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes an inference acceleration method for machine learning and large language models based on a packaging engine. The method can support online updating and version management of the model, and facilitate iteration and optimization in a rapidly changing application environment.

[0006] To achieve the above object, the present invention provides a method for accelerating the reasoning of machine learning and large language models based on a packaging engine, comprising the following steps: Step 1: Implement the performInferenceWithCacheCheck method in the inference engine and receive the Boolean parameter enable_cache to control whether to enable the kvcache acceleration function to ensure that the system can flexibly adapt to different inference requirements.

[0007] Step 2: According to the passed enable_cache value, execute the conditional judgment logic to decide whether to perform cache query; if true, perform kvcache query to improve reasoning efficiency.

[0008] Step 3: Create a Triton model repository under the specified file path, ensure that the file structure is clear and meets the requirements, and create a separate folder for each model to facilitate subsequent access and management.

[0009] Step 4: In the config.pbtxt file, accurately define the model version information, input and output tensor names, data types, and dimensions to ensure data compatibility and stability during inference.

[0010] Step 5: Start the Triton Inference Server using the command line tool and specify the model repository path so that the server can load and prepare to process subsequent inference requests.

[0011] Furthermore, the specific method of determining the user configuration in step 2 is: Step 2.1: In the performInferenceWithCacheCheck method, use a conditional statement to determine the Boolean value of the parameter enable_cache.

[0012] Step 2.2: If enable_cache is true, call the kvcache query method to check whether there is a corresponding inference result in the cache.

[0013] Step 2.3: If the cache hits, directly return the result in the cache; if it does not hit, record the current input parameters and continue with subsequent inference calculations.

[0014] Furthermore, the specific process of configuring and creating the Triton model warehouse in step 3 is as follows: Step 3.1: Specify the path of the model repository in the file system, for example, / models / . Make sure the path exists and is writable.

[0015] Step 3.2: Create a separate folder for each model to be deployed under this path, with the naming format of / models / model_name / for easy subsequent management.

[0016] Step 3.3: In each model folder, prepare the model file and its configuration file config.pbtxt, making sure the file format complies with Triton specifications.

[0017] Furthermore, the detailed configuration of the model file in step 4 is as follows: Step 4.1: In the config.pbtxt file, define the basic information of the model, including the platform (such as TensorRT), max_batch_size, input and output names and data types.

[0018] Step 4.2: For each input and output tensor, specify its data type (e.g., FP32, INT64), shape (e.g., [1, 3, 224, 224]), and other relevant parameters to ensure that the model can interpret the data correctly.

[0019] Step 4.3: Ensure that the model version information is clear and set the version number for each model as needed so that subsequent calls can specify a specific version.

[0020] Furthermore, the specific operation of starting the Triton inference server in step 5 is: Step 5.1: Use the command line tool to execute the start command of Triton Inference Server. The example is as follows: Step 5.2: During the startup process, monitor the terminal output to confirm that all models are loaded correctly and there are no errors, ensuring that the server is in a ready state.

[0021] Step 5.3: Verify whether the inference server can receive and respond to inference requests by accessing the specified API interface to ensure the normal operation of the system.

[0022] Furthermore, the specific method for implementing kvcache query in step 2 is: Step 6.1: Store the input parameters and corresponding output results of each inference in kvcache to form a key-value pair mapping.

[0023] Step 6.2: When making a new inference request, first build the query key based on the input parameters and check whether the key exists in the kvcache.

[0024] Step 6.3: If the query result exists, the corresponding output result is returned directly; if the query result does not exist, the request is recorded and the inference calculation is prepared to be performed.

[0025] Furthermore, the dynamic update function of the Triton model is configured in step 3 as follows: Step 7.1: In the Triton model repository, create a monitoring script to regularly check for new model versions and updates.

[0026] Step 7.2: When a new model version is found, the config.pbtxt file is automatically updated and the old version is replaced in the model repository.

[0027] Step 7.3: After updating, restart the Triton inference server to load the new version of the model to ensure that the new model can be used for inference immediately.

[0028] Furthermore, the data compatibility and stability in the reasoning process in step 4 are as follows: Step 8.1: When an inference request arrives, record the source IP address and request time of the request for traffic analysis.

[0029] Step 8.2: Distribute the inference requests to different backend instances based on the real-time traffic situation to ensure load balancing for each instance.

[0030] Step 8.3: When an instance is overloaded, dynamically adjust the request routing strategy to prioritize requests to instances with lower current loads.

[0031] Furthermore, the subsequent reasoning request is processed in step 5 as follows: Step 9.1: When receiving an inference request, immediately return a request receipt confirmation to inform the client that inference is in progress.

[0032] Step 9.2: Put the inference task into the task queue and use the thread pool to asynchronously perform inference calculations to improve system responsiveness.

[0033] Step 9.3: After the inference is completed, the result is sent to the client through the callback mechanism, or is made available for client query through the specified interface.

[0034] Furthermore, step 5 also includes implementing persistent storage of the inference results, as follows: Step 10.1: After the reasoning is completed, the input parameters and output results of each reasoning are recorded in the database to form a complete reasoning log.

[0035] Step 10.2: Assign a unique identifier to each record to facilitate subsequent query and analysis.

[0036] Step 10.3: Regularly clean up expired inference records to maintain efficient operation of the database and ensure effective use of storage space.

[0037] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides an inference acceleration method for machine learning and large language models based on a packaging engine. By using the kvcache efficient caching mechanism, data access delay is reduced, thereby significantly accelerating the inference response time and meeting real-time application requirements. The method optimizes the use of computing resources during the inference process and reduces the demand for hardware such as CPU and GPU, so that it can run efficiently even in a resource-constrained environment.

[0038] 2. The present invention provides an inference acceleration method for machine learning and large language models based on a packaging engine. Combined with the optimization characteristics of Triton, it can improve the inference efficiency of large models while maintaining high precision and ensure the smoothness of user experience. The flexibility of Triton makes this method compatible with a variety of deep learning frameworks, facilitating model integration and deployment.

[0039] 3. The present invention provides an inference acceleration method for machine learning and large language models based on a packaging engine. By accelerating the inference process, the long-term consumption of computing resources is reduced, thereby reducing operating costs, especially when deployed on a large scale. The fast inference response time improves the interactivity of the application and user satisfaction, especially in fields such as finance, medical care, and education that have high real-time requirements.

[0040] 4. The present invention provides a method for accelerating the reasoning of machine learning and large language models based on a packaging engine. The method can support online updating and version management of the model, and facilitate iteration and optimization in a rapidly changing application environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of the steps of the present invention; Figure 2 It is a comparison of KV-Cache inference acceleration; Figure 3 Comparison of time consumption for KV-Cache reasoning acceleration; Figure 4 Comparison of the time consumption per token for KV-Cache reasoning acceleration; Figure 5 It is a verification and comparison data diagram of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.

[0044] Terminology explanation: kvcache: A cache mechanism used to store input parameters and their corresponding output results during the inference process to speed up subsequent inference requests. By looking up the cache, repeated calculations can be avoided, thereby improving efficiency.

[0045] Triton: Triton Inference Server is an open source inference server designed to provide efficient inference services for machine learning models. It supports multiple deep learning frameworks (such as TensorFlow, PyTorch, etc.) and provides dynamic batch processing, model parallelism and other functions to optimize inference performance.

[0046] performInferenceWithCacheCheck: This is the name of a method used to perform a cache check when performing inference. If caching is enabled, this method queries the kvcache to check if there are already results to return.

[0047] config.pbtxt: This is the standard format of Triton model configuration file, which contains basic information about the model, such as the name of the input and output, data type, dimension, etc. It helps the Triton server understand how to process a specific model.

[0048] max_batch_size: The maximum batch size specified in the Triton configuration, which refers to the maximum number of input samples that can be processed simultaneously in one inference request.

[0049] FP32: represents 32-bit floating point numbers, a format used to represent numerical values, commonly used for input and output of deep learning models.

[0050] INT64: represents a 64-bit integer, which is another format for representing numerical values ​​and is often used to store a large range of integer values.

[0051] Dynamic update: refers to the ability to automatically check and update model versions in the Triton model repository. This ensures that the latest model version is used.

[0052] Traffic analysis: The process of monitoring and analyzing request traffic in a system in order to optimize system performance and user experience.

[0053] Load balancing: A technique for distributing requests to multiple backend instances to ensure that each instance has a relatively even workload, thereby improving the overall performance and responsiveness of the system.

[0054] Task queue: A data structure used to store pending inference requests, allowing inference computations to be performed asynchronously, thereby improving the responsiveness of the system.

[0055] Callback mechanism: A programming model that returns the results to the client through a specific method (such as notification or interface call) after the inference is completed.

[0056] like Figure 1 As shown, the present invention is specifically: Step 1: Implement the performInferenceWithCacheCheck method in the inference engine and accept the Boolean parameter enable_cache to control whether to enable the kvcache acceleration function; Step 2: According to the passed enable_cache value, execute the conditional judgment logic to decide whether to perform cache query; if true, perform kvcache query; Step 3: Create a Triton model repository in the specified file path and create a separate folder for each model; Step 4: In the config.pbtxt file, define the model version information, input and output tensor names, data types, and dimensions; Step 5: Use the command line tool to start Triton Inference Server and specify the model repository path.

[0057] The idea of ​​the present invention is as follows: In the application process of business production, the reasoning of large models will be accelerated. The goal is to improve the reasoning efficiency through kvcache cache acceleration and encapsulation combined with the triton reasoning framework on the basis of ensuring the reasoning accuracy of large models, significantly reduce the computational complexity and resource consumption of the reasoning process, and increase the reasoning speed.

[0058] kvcache cache is enabled Supports configuration of the effect of turning on / off KV-Cache on inference efficiency. KV-Cache can be turned on for model inference acceleration. The inference engine can dynamically modify the KV-Cache option after receiving the parameters. KV-Cache can be turned on and off through use_cache. When use_cache is true, the model will initialize past_key_values ​​to store Key and Value vectors during the inference process, and each step of inference will maintain it. If use_cache is false, past_key_values ​​will not take effect. Figure 2-Figure 4 As shown; Triton frame package Based on the existing inference engine extension, the triton framework is supported for inference acceleration. When inferring large models, the dynamic batch processing and concurrent execution features of the triton framework can be used to accelerate inference and improve the inference performance of large models.

[0059] Inference startup customization enabled When the model is encapsulated in a capability state, the page can dynamically enable kvcache inference acceleration.

[0060] The technical implementation process is described as follows: (I) Kvcache cache acceleration customization is enabled The specific implementation is as follows: The performInferenceWithCacheCheck method added in the inference engine supports dynamic setting of whether to enable kvcache. After the judgment is passed, the original inference logic is called to continue execution.

[0061] (II) Triton framework packaging acceleration 1. Triton creates a model warehouse 2. Model file configuration 3. Start the triton inference server 4. The existing inference engine supports the triton framework to improve the efficiency of inference acceleration.

[0062] As a specific implementation method, the system architecture is designed as follows: Overall architecture: The system consists of the client, KVCACHE layer and Triton Inference Server. The client is responsible for sending inference requests and receiving results, KVCACHE is used to cache data, and Triton processes model inference.

[0063] Module division: Client module: Responsible for data collection, request sending and result display, supporting multiple data formats (such as images, text).

[0064] KVCACHE module: Responsible for caching input data and intermediate results, providing a fast access interface, using key-value pair storage to ensure efficient search and update.

[0065] Inference engine module: Triton Inference Server, which supports parallel inference of multiple models and has automatic scaling capabilities.

[0066] Data Flow Management: Input data processing: The client sends an inference request through the REST API or gRPC, and the data is initially verified and preprocessed before being sent, including format conversion and standardization.

[0067] After preprocessing, the data is converted into a format suitable for model input (for example, images are scaled to a specified size).

[0068] Cache strategy: Cache structure: KVCACHE uses a distributed cache structure to store the hash value of the input data as the key and the intermediate inference result as the value.

[0069] Cache strategy: Adopt LRU (least recently used) strategy to manage cache to improve hit rate and save storage space.

[0070] Model loading and inference: Model Management: Preload multiple versions of the model in Triton to accommodate different inference requirements (such as different precision or optimized versions).

[0071] The model's input and output formats, batch size, and other inference parameters are specified through the configuration file.

[0072] Reasoning process: After receiving the inference request, the system first queries KVCACHE to determine whether there is cached data.

[0073] If the cache hits, the cached result is returned directly to reduce inference latency.

[0074] If the cache misses, the system calls Triton for inference, and after the inference is completed, the result is stored in KVCACHE for subsequent use.

[0075] Result caching and updating mechanism: Result cache: After the reasoning is completed, the results and their input data are stored in KVCACHE, and a reasonable expiration time (TTL) is set to ensure the timeliness of the data.

[0076] Dynamic update mechanism: When new input data is detected, the system automatically updates the relevant cache in KVCACHE to ensure that the latest data is available.

[0077] Supports event-based update strategies. For example, when similar data is requested multiple times, the corresponding cache is updated first.

[0078] Performance monitoring and tuning: Monitoring module: Implement real-time performance monitoring and collect indicators such as inference time, cache hit rate, and system load.

[0079] Display real-time performance data through a visual dashboard to help developers identify performance bottlenecks in a timely manner.

[0080] Tuning strategy: Automatically adjust model loading strategies and cache configurations based on monitoring data to optimize inference performance.

[0081] Regularly analyze system operation, generate reports and make recommendations for improvements, such as increasing cache size or optimizing model configuration.

[0082] Security and fault tolerance mechanism: Data security: During data transmission, HTTPS protocol is used to encrypt communication to ensure data security.

[0083] Fault-tolerance mechanism: Set up a retry mechanism. When an inference request fails, the system will automatically retry or roll back to the last stable state to ensure high availability of the system.

[0084] Through the above detailed implementation method, the system can effectively utilize KVCACHE and Triton engines to accelerate large model reasoning and improve overall system performance.

[0085] As a specific implementation example, a technology company developed a deep learning-based image classification application with the goal of processing a large number of image requests in real time. The company used this method as follows: 1. System architecture design Overall architecture: The system consists of user client, KVCACHE layer, Triton Inference Server and data storage layer.

[0086] The user client interacts with the system through the API, KVCACHE is used to store historical reasoning results, and Triton is responsible for model reasoning.

[0087] Component details: User client: Built using the React framework, supports image uploading and displays inference results.

[0088] KVCACHE: Implemented using Redis, setting a reasonable expiration strategy to reduce memory usage.

[0089] Triton Inference Server: Deployed on NVIDIA GPU servers, supports multiple models and is called through a RESTful API.

[0090] 2. Data flow management process User input: When users upload images on the client, the system first verifies the format and size (supports JPEG / PNG, maximum 5MB).

[0091] After successful upload, the client will convert the image to the appropriate Tensor format (224x224 RGB).

[0092] Hash generation and cache query: After the image is processed, an MD5 hash value is generated as a unique identifier.

[0093] The client queries KVCACHE to see if the hash value already exists.

[0094] Cache processing logic: Hit cache: If KVCACHE returns a result, the client directly displays the classification result and records the number of hits.

[0095] Cache miss: If no results are found, the client sends the image data to Triton for inference.

[0096] 3. Model loading and inference mechanism Model configuration and loading: The Triton configuration file defines the model input and output specifications, uses the ResNet model optimized by TensorRT, and supports batch reasoning (batch size is set to 4).

[0097] Model preloading mechanism ensures fast response during inference.

[0098] Reasoning process: The client sends the image data to Triton via an HTTP POST request.

[0099] After Triton processes the request, it performs inference and returns the results, including the classification label and confidence.

[0100] The system records the inference time and compares it with the set target time (for example, 200 milliseconds).

[0101] 4. Result caching and update strategy Result storage: After Triton inference is completed, the classification results (label, confidence, image hash) are stored in KVCACHE, and the TTL is set to 10 minutes to support fast repeated requests.

[0102] Dynamic Update and Cleanup: The LRU (Least Recently Used) algorithm is used for cache elimination to ensure that frequently used results remain in the cache.

[0103] Regularly check the cache status, count the hit rate, and optimize the TTL settings.

[0104] 5. Performance monitoring and optimization Monitoring Tools: Use Prometheus to collect performance metrics, including inference latency, cache hit rate, and request volume.

[0105] Configure Grafana to display the dashboard and monitor the system health status in real time.

[0106] Performance optimization: By analyzing the monitoring data, we found that the main bottleneck of inference latency is model loading.

[0107] Take steps such as adding Triton instances to leverage NVIDIA GPUs for parallel processing.

[0108] 6. Security and fault-tolerant design Data transmission security: All API interfaces use HTTPS encryption to ensure user data security.

[0109] Fault Tolerance: The system is designed with a retry mechanism. If Triton reasoning fails, the system will retry after 1 second, and the maximum number of retries is set to 3 times.

[0110] The log system records all failed requests and retries for easy later analysis and repair.

[0111] 7. Results and Evaluation Through the above design and implementation, the system achieves the following effects: Improved inference speed: The average inference time is reduced from 350 milliseconds to 150 milliseconds.

[0112] Cache hit rate: reaches 90%, greatly reducing the computational burden of repeated reasoning.

[0113] Improved user experience: Basic real-time response has been achieved, and user satisfaction surveys show that 94% of users say the system responds quickly.

[0114] Through this case, the comprehensive implementation path and results of the reasoning acceleration system based on KVCACHE and Triton are demonstrated, providing a powerful reference for subsequent related projects.

[0115] The design proposed in this paper to achieve large model reasoning acceleration based on kvcache and triton packaging engine uses kvcache cache and triton packaging engine in technical implementation method, which is better than the traditional direct reasoning acceleration engine method. In terms of business use, it can better and more flexibly support enterprise data intelligence business, especially in the use of large model intelligent question and answer reasoning, which can help enterprises reduce costs and increase efficiency and enhance their competitiveness. Figure 5 shown.

[0116] This method has multiple beneficial effects, as follows: First, through the efficient caching mechanism of kvcache, this method can greatly improve the inference speed. When processing large-scale data, kvcache effectively reduces the latency of data access and optimizes the use of memory and storage. This significantly shortens the inference response time and meets the needs of real-time application scenarios, such as online question-answering and real-time data analysis.

[0117] Secondly, this method effectively optimizes the use of computing resources. Large models usually have very high requirements for hardware resources such as CPU and GPU, and kvcache reduces the consumption of computing resources by caching intermediate results and duplicate data. This is particularly important in resource-constrained environments, allowing low-power devices to run complex models and expand the boundaries of applications.

[0118] In addition, combined with the Triton engine's multiple optimization techniques, this method not only maintains the high accuracy of the model, but also improves the efficiency of reasoning. Triton supports strategies such as dynamic batch processing and model parallelism, allowing multiple requests to be processed simultaneously, thereby further improving throughput. This can ensure stable service quality under high concurrency for large-scale application scenarios such as intelligent customer service and financial risk control.

[0119] In terms of user experience, fast reasoning responses significantly improve interactivity and user satisfaction. When users use financial analysis tools, medical auxiliary diagnosis systems, or personalized learning platforms, they can experience instant feedback, which enhances the practicality and attractiveness of the system.

[0120] By reducing computing costs, enterprises can achieve significant economic benefits in large-scale deployments. Reducing the need for high-performance computing hardware reduces overall operating costs, allowing enterprises to be more flexible when investing in technology.

[0121] This method also has good scalability and can support online updates and version management of the model, making it easy to quickly respond to changes in market demand. In a fast-iterating application environment, developers can easily deploy new model versions while maintaining high availability and stability of the service.

[0122] Finally, Triton’s flexibility and compatibility allow the method to support multiple deep learning frameworks, facilitating model integration and deployment. This enables different teams and projects to quickly adapt and implement their respective technical needs, driving innovation and development across the industry.

[0123] In summary, this method shows significant advantages in improving reasoning speed, optimizing resource usage, improving user experience, reducing costs and enhancing scalability, providing strong support for the practical application of large models.

[0124] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.

Claims

1. A method for accelerating the reasoning of machine learning and large language models based on a packaging engine, characterized in that: The following steps are involved: Step 1: Implement the performInferenceWithCacheCheck method in the inference engine and accept the Boolean parameter enable_cache to control whether to enable the kvcache acceleration function; Step 2: Based on the passed enable_cache value, execute the conditional judgment logic to decide whether to execute the cache query; If true, perform a kvcache query; Step 3: Create a Triton model repository in the specified file path and create a separate folder for each model; Step 4: In the config.pbtxt file, define the model version information, input and output tensor names, data types, and dimensions; Step 5: Use the command line tool to start Triton Inference Server and specify the model repository path.

2. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The specific method for determining the user configuration in step 2 is: Step 2.1: In the performInferenceWithCacheCheck method, use the conditional statement to determine the Boolean value of the parameter enable_cache; Step 2.2: If enable_cache is true, call the kvcache query method to check whether there is a corresponding inference result in the cache; Step 2.3: If the cache hits, directly return the result in the cache; if it does not hit, record the current input parameters and continue with subsequent inference calculations.

3. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The specific process of configuring and creating the Triton model warehouse in step 3 is: Step 3.1: Specify the path to the model repository in the file system; Step 3.2: Create a separate folder for each model to be deployed under the path in step 3.1, with the naming format of / models / model_name / ; Step 3.3: In each model folder, prepare the model file and its configuration file config.pbtxt, making sure the file format complies with Triton specifications.

4. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The detailed configuration of the model file in step 4 is as follows: Step 4.1: In the config.pbtxt file, define the basic information of the model, including platform, max_batch_size, input and output names and data types; Step 4.2: Specify the data type, shape, and related parameters for each input and output tensor; Step 4.3: Ensure that the model version information is clear and set the version number for each model as needed.

5. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The specific operations to start the Triton inference server in step 5 are: Step 5.1: Use the command line tool to execute the start command of Triton Inference Server; Step 5.2: During the startup process, monitor the terminal output to confirm that all models are loaded correctly and there are no errors; Step 5.3: Verify whether the inference server can receive and respond to inference requests by accessing the specified API interface.

6. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The specific method to implement kvcache query in step 2 is: Step 6.1: Store the input parameters and corresponding output results of each inference in kvcache to form a key-value pair mapping; Step 6.2: When making a new inference request, first build the query key based on the input parameters and check whether the query key exists in the kvcache; Step 6.3: If the query result exists, directly return the corresponding output result; If the query result does not exist, the request is recorded and the inference calculation is prepared to be performed.

7. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: In step 3, the dynamic update function of the Triton model is configured as follows: Step 7.1: In the Triton model repository, create a monitoring script to regularly check for new model versions and updates; Step 7.2: When a new model version is found, the config.pbtxt file is automatically updated and the old version is replaced in the model repository; Step 7.3: After updating, restart the Triton inference server to load the new version of the model.

8. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The data compatibility and stability in the reasoning process in step 4 are as follows: Step 8.1: When an inference request arrives, record the source IP address and request time of the request for traffic analysis; Step 8.2: Distribute the inference requests to different backend instances based on the real-time traffic situation to ensure load balancing for each instance; Step 8.3: When an instance is overloaded, dynamically adjust the request routing strategy to prioritize requests to instances with lower current loads.

9. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: The subsequent inference request is processed in step 5 as follows: Step 9.1: When receiving an inference request, immediately return a request receipt confirmation to inform the client that the inference is in progress; Step 9.2: Put the inference task into the task queue and use the thread pool to asynchronously perform the inference calculation to improve the system responsiveness; Step 9.3: After the inference is completed, the result is sent to the client through the callback mechanism, or is made available for client query through the specified interface.

10. The method for accelerating the reasoning of machine learning and large language models based on a packaging engine according to claim 1, characterized in that: Step 5 also includes the implementation of persistent storage of inference results, as follows: Step 10.1: After the reasoning is completed, the input parameters and output results of each reasoning are recorded in the database to form a complete reasoning log; Step 10.2: Assign a unique identifier to each record for subsequent query and analysis; Step 10.3: Regularly clean up expired reasoning records to keep the database running efficiently and ensure the utilization of storage space.

Images