A method and system for realizing resource use and concurrent inference of a large model all-in-one machine

Abstract

The application provides a method and system for realizing resource use and concurrent reasoning of a large model all-in-one machine, which comprises collecting model basic information in combination with the use scene of the large model all-in-one machine; formulating a differentiated resource allocation strategy according to an evaluation result, planning isolated resources and shared resources; evaluating and classifying data storage; optimizing a model reasoning engine; defining core monitoring indexes; constructing a multi-dimensional index evaluation system; and completing dynamic optimization of model resources and parameters by referring to the evaluation result of the model indexes. Through systematic resource planning and management design, the application constructs an efficient intelligent resource scheduling system, realizes intelligent perception and precise deployment of resources of the large model all-in-one machine, and effectively supports accurate evaluation of the rationality of resource use, effectively responds to complex and changeable application scene requirements, significantly improves the resource utilization rate and operation efficiency of the large model all-in-one machine, and brings an innovative solution and significant application value to the field of the large model all-in-one machine.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of resource use and concurrent reasoning of model all-in-one machine, and particularly relates to a method and system for realizing resource use and concurrent reasoning of large model all-in-one machine. BACKGROUND

[0002] With the vigorous development of artificial intelligence technology, large model all-in-one machine has been widely applied in natural language processing, computer vision, intelligent decision-making and many other fields due to its integrated and high-performance characteristics. In practical applications, large model all-in-one machine needs to support concurrent reasoning of multiple large models at the same time to meet the business needs in diversified scenarios. However, current large model all-in-one machine faces severe challenges in resource use and concurrent reasoning.

[0003] In terms of resource use, due to the complex structure and computationally intensive nature of large models, the demand for CPU, memory, GPU, storage and other resources varies greatly and dynamically among different models, and existing technologies are difficult to accurately assess the resource demand of each model, resulting in "one-size-fits-all" or over-reservation in resource allocation, causing some models to occupy too many resources during reasoning but not fully utilized, while other models cannot function properly due to insufficient resources.

[0004] In terms of concurrent reasoning, as application scenarios continue to expand, higher requirements are placed on the concurrent reasoning capability of large model all-in-one machine. However, traditional reasoning techniques are difficult to adapt to high-concurrency scenarios, with slow data read-write speed, idle computing resources and other issues. To address these issues, some research attempts to adopt a single resource management strategy or optimize a particular technology, such as only optimizing data format or only increasing thread count to improve concurrency. However, these methods lack systematicness and overallness, and cannot fundamentally solve the coordination problem of resource use and concurrent reasoning, making it difficult to meet the demand for high-reliability and high-efficiency concurrent reasoning in actual business.

[0005] Therefore, there is an urgent need for a method and system that can solve the problems of resource waste, unreasonable allocation and resource insufficiency in the concurrent reasoning process of large models, and achieve high-reliability concurrent reasoning. SUMMARY

[0006] To overcome the shortcomings of existing technologies, the present application proposes a method and system for realizing resource use and concurrent reasoning of large model all-in-one machine, which realizes systematic optimization of large model all-in-one machine resources and high-reliability concurrent reasoning through building a comprehensive resource demand assessment system, innovative resource management and allocation mechanism, multi-dimensional data optimization strategy and efficient concurrent reasoning technology, combined with a perfect monitoring and dynamic adjustment mechanism. Compared with existing technologies, the present application has significant innovation value and advantages in resource utilization, reasoning efficiency and scenario adaptation, providing a new solution for the development and application of large model all-in-one machine.

[0007] To achieve the above object, the application provides a method for realizing resource use and concurrent reasoning of large model integrated machine, comprising the following steps:

[0008] Step S1: In combination with the use scene of the large model integrated machine, collect model basic information, and accurately evaluate the resource requirements of all models by using a hybrid resource evaluation method;

[0009] Step S2: In combination with the actual resource situation of the large model integrated machine, develop a differentiated resource allocation strategy according to the evaluation results, and reasonably plan isolated resources and shared resources;

[0010] Step S3: Evaluate the classification data storage, use an adaptive compression framework, and intelligently optimize the integrated machine storage space and read-write speed;

[0011] Step S4: Optimize the model reasoning engine, complete model service path analysis and dependency arrangement, and containerize deployment of stable and reliable model reasoning services;

[0012] Step S5: Define core monitoring indicators, develop a hierarchical, collection and processing strategy for the indicators, and monitor various indicators in the model reasoning process in real time;

[0013] Step S6: Build a multi-dimensional indicator evaluation system, use a baseline comparison and weighted analysis comprehensive scoring model to evaluate the indicators of the model service;

[0014] Step S7: Refer to the model indicator evaluation results, complete dynamic optimization of model resources and parameters according to the model priority and resource requirement difference.

[0015] Further, the step S1 specifically comprises:

[0016] Step S11: Clearly define the use scene of the large model integrated machine, analyze the specific requirements such as business process, data characteristics, user request, service quality and the like under each scene;

[0017] Step S12: Collect information such as model architecture, algorithm logic, input-output characteristics, historical running records and the like;

[0018] Step S13: Build a hybrid evaluation model based on the rule engine-gradient boosting tree collaborative architecture, take the specific scene requirements, model structure parameters, input-output data characteristics, historical resource use data as the input features of the model, and evaluate the resource requirements of all models;

[0019] The core algorithm of the rule engine comprises:

[0020] According to the model parameter quantity dynamic calculation GPU demand: parameter quantity ≥1000 billion, GPU quantity = max(1, int(parameter quantity / 200)), maximum limit 16 cards;

[0021] According to the input type correction resource demand: image scene GPU quantity ×1.5, audio scene ×1.2;

[0022] According to the QoS constraint adjustment: delay requirement <100ms, GPU quantity ×2;

[0023] CPU core number = max(8, int(total computing power / 10)), memory = max(32GB, int(GPU memory ×1.5));

[0024] Gradient boosting tree uses LightGBM implementation, target function is mean square error, training target is resource demand prediction error ≤15%;

[0025] 1) The core algorithm of the rule engine is:

[0026] def rule_based_estimation(model_params, scenario, gpu_model="H100"):

[0027] # GPU model configuration mapping table (memory unit: GB, computing power unit: TFLOPS)

[0028] GPU_CONFIG = {"H100": {"memory": 80, "flops": 600}, "A100": {"memory": 80, "flops": 312}, "A6000": {"memory": 48, "flops": 156}, "V100": {"memory": 32, "flops": 125}, "T4": {"memory": 16, "flops": 80}}

[0029] # Get the current GPU configuration

[0030] gpu_config = GPU_CONFIG.get(gpu_model, GPU_CONFIG["H100"])

[0031] # Parameter level rule (unit: billion parameters)

[0032] if model_params["params"]>= 1000:

[0033] gpu_count = max(1, int(model_params["params"] / 200)) # Dynamic calculation

[0034] gpu_count = min(gpu_count, 16) # Maximum 16 GPUs limit

[0035] elif model_params["params"]>= 100:

[0036] gpu_count = max(1, int(model_params["params"] / 50))

[0037] else:

[0038] gpu_count = 1

[0039] # Input complexity correction (image / speech scenarios)

[0040] if "image" in scenario["input_types"]:

[0041] gpu_count = int(gpu_count * 1.5)

[0042] if "audio" in scenario["input_types"]:

[0043] gpu_count = int(gpu_count * 1.2)

[0044] # QoS Constraints (Latency-Sensitive Scenarios)

[0045] if scenario["latency_sla"]<100:

[0046] gpu_count = int(gpu_count * 2)

[0047] # Calculate total resource requirements

[0048] total_gpu_memory = gpu_count * gpu_config["memory"]

[0049] total_flops = gpu_count * gpu_config["flops"]

[0050] # CPU and memory ratio based on computing power

[0051] cpu_cores = max(8, int(total_flops / 10)) # 1 CPU core per 10 TFLOPS

[0052] memory = max(32, int(total_gpu_memory * 1.5)) # Memory is 1.5 times the GPU memory.

[0053] return {"gpu_model": gpu_model, "gpu_count": gpu_count, "total_gpu_memory": total_gpu_memory, "cpu_cores": cpu_cores, "memory": memory, "total_flops": total_flops}

[0054] Further, step S2 specifically includes:

[0055] Step S21: Statistically analyze the hardware resource configuration of the large-scale integrated machine, and analyze the current utilization rate and load of each resource;

[0056] Step S22: Formulate resource allocation principles, including multiple allocation principles such as high priority, on-demand allocation, dynamic balancing, and efficiency priority. The specific methods for handling resource competition are as follows:

[0057] Priority preemption: High-priority models configured with preemption_policy=Always can preempt CPU time slices from low-priority models;

[0058] Quota management: Limit namespace resources through ResourceQuota, such as limits.cpu=96c, limits.memory=1024GiB;

[0059] Step S23: Based on the model priority and resource demand assessment results, determine specific planning schemes for isolated and shared resources, including:

[0060] Configure GPU memory hard isolation via nvidia-container-cli;

[0061] Set CPU quotas using cgroup v2;

[0062] Adjust memory OOM priority; Implement multiple resource isolation and sharing mechanisms.

[0063] Furthermore, step S3 specifically includes:

[0064] Step S31: Evaluate and classify data in terms of data type, data format, access mode, data lifecycle management, etc.

[0065] Step S32: Build an adaptive compression framework, integrate multiple efficient data compression algorithms, and design intelligent selection strategies to meet the needs of data compression in different scenarios.

[0066] Step S33: Identify high-frequency access data and low-frequency access data, distinguish real-time data, short-term data, and long-term archived data, and build a data hierarchical storage architecture to meet the needs of high-frequency data fast read and write and low-frequency data long-term archiving.

[0067] Further, the step S4 specifically includes:

[0068] Step S41: Optimize the memory management of the model inference engine, use asynchronous stream processing mechanism to improve the efficiency of model concurrent inference, which involves the following key steps:

[0069] Use aiohttp to handle network requests and use asyncio to manage coroutines to solve the performance bottleneck of traditional synchronous IO and improve request throughput;

[0070] Divide the inference process into four stages: word segmentation, feature extraction, model inference, and post-processing, and execute each stage in an independent GPU stream to reduce latency;

[0071] Step S42: Analyze the dependency relationship of the model inference service, determine the model service startup order, and use containerized orchestration to deploy the model inference service.

[0072] Further, the step S5 specifically includes:

[0073] Step S51: Define the core monitoring index system, including system resource index set, model inference performance index set, and business quality index set;

[0074] Step S52: Build a three-level hierarchical collection strategy, configure differentiated collection frequency, and use TSDB time series database to compress and store index data, wherein:

[0075] First-level collection: Real-time key indicators, collection frequency 5 seconds;

[0076] Second-level collection: System performance indicators, collection frequency 10-30 seconds;

[0077] Third-level collection: Business statistics indicators, collection frequency 60 seconds.

[0078] Further, the step S6 specifically includes:

[0079] Step S61: define evaluation dimensions and weights, set model index grading standards, and construct a multi-dimensional index evaluation system;

[0080] Step S62: establish baseline standards, calculate the comprehensive score of multiple dimensions using a weighted analysis method, and map to the corresponding grade interval.

[0081] Further, the step S7 specifically includes:

[0082] Step S71: intuitively analyze system resource index scores to determine unreasonable resource allocation and usage;

[0083] Step S72: according to the comprehensive score of the service index, further identify low-score dimensions, use correlation analysis method to find the correlation between model services and resource usage, model parameters, and determine the optimization entry point; specifically as shown in Table 1;

[0084] Table 1

[0085]

[0086] The algorithm for correlation analysis is:

[0087]

[0088] wherein and are the observation values of two indexes, is the mean value of , and is the mean value of , and the value of r is between [-1, 1], when |r| is greater than or equal to 0.7, it is a strong correlation, and the associated index is adjusted first.

[0089] Step S73: dynamically adjust resource usage using various techniques such as vertical expansion and horizontal expansion, and timely optimize model services, specifically including:

[0090] CPU resource adjustment: if the utilization rate is greater than 80% or less than 20% for 10 consecutive minutes, then increase or decrease 2 cores each time, with a maximum adjustment range of 50% of the current configuration;

[0091] GPU memory adjustment: if the memory utilization rate is greater than 90% and the delay is greater than the threshold, then expand by 2GB steps, up to 90% of the physical memory; if the memory utilization rate is less than 30% for 1 hour and the request delay meets the SLA, then shrink by 2GB steps;

[0092] Horizontal expansion: based on the HPA mechanism, when QPS is greater than the threshold and lasts for 10 minutes, automatically increase the number of inference container replicas, the expansion step is 50% of the current number of replicas; when QPS is less than the threshold and lasts for 1 hour, automatically reduce 1 inference container replica, but at least keep 1 inference container.

[0093] A system for realizing efficient use of resources and concurrent inference of large model all-in-one machine, for realizing any one of the methods, characterized in that the system comprises:

[0094] Resource demand evaluation module: for combining the actual application scene of the large model all-in-one machine, through multi-dimensional data collection and hybrid evaluation method, accurately quantifying the hardware resources and computing demand required for model running, providing scientific basis for resource allocation;

[0095] Resource management and allocation module: for combining the real-time resource state of the large model all-in-one machine according to the output result of the resource demand evaluation module, realizing dynamic scheduling, elastic allocation and efficient reuse of computing power resources, and ensuring the performance and stability during multi-model concurrent inference;

[0096] Storage optimization processing module: for the storage data characteristics of the large model all-in-one machine, through intelligent compression, hierarchical storage, efficient read-write optimization and other technologies, realizing storage space occupation minimization and access efficiency maximization, supporting the storage performance requirements of multi-model concurrent inference;

[0097] Concurrent inference module: for realizing efficient parallel processing of multiple models and multiple requests on the large model all-in-one machine, through request scheduling, model parallelism, batch processing optimization and other technologies to improve system throughput and resource utilization, while ensuring that the inference service meets the business SLA requirements;

[0098] Performance monitoring module: for building a comprehensive monitoring system for the large model all-in-one machine, through multi-dimensional data collection and real-time analysis, reasonably evaluating various indicators of the model, and providing data support for resource adjustment and model optimization decision-making.

[0099] Further, the resource demand evaluation module comprises:

[0100] Rule engine unit: built-in GPU configuration mapping table and resource calculation rules, dynamically calculating the basic resource demand according to the model parameter quantity, input type and QoS requirement;

[0101] Gradient boosting tree unit: using LightGBM algorithm, training prediction model based on historical resource usage data, correcting the output of the rule engine, and ensuring that the prediction error is controlled within 15%.

[0102] Further, the resource management and allocation module comprises:

[0103] Priority manager: Maintains model priority queue, supports preemption_policy configuration, and implements priority-based resource preemption;

[0104] Quota controller: Implements resource quota enforcement through cgroup v2 and namespace technology combined with ResourceQuota object;

[0105] Shared pool manager: Manages dynamic allocation of shared resource pool using time division multiplexing algorithm;

[0106] Isolation controller: Implements GPU memory hard isolation through nvidia-container-cli and CPU core binding through CPU affinity setting.

[0107] Further, the performance monitoring module comprises:

[0108] Data collector: Implements three-level hierarchical collection, including real-time collector (5-second frequency), performance collector (10-30 second frequency), and business collector (60-second frequency);

[0109] Time series database: Stores monitoring data using TSDB technology, supports data compression and multi-level retention policy;

[0110] Indicator evaluation engine: Implements multi-dimensional weighted scoring algorithm and correlation analysis algorithm to generate comprehensive evaluation report;

[0111] Dynamic optimization executor: Automatically executes vertical and horizontal expansion strategies based on evaluation results and correlation analysis.

[0112] Compared with the prior art, the present application has the following advantages:

[0113] 1. The present application provides a method and system for realizing resource use and concurrent inference of large model all-in-one machine, which accurately evaluates model resource demand, combines resource isolation restriction and shared reuse technology, effectively solves the problem of unreasonable resource allocation, improves the resource utilization rate of large model all-in-one machine, and reduces the running cost.

[0114] 2. The present application provides a method and system for realizing resource use and concurrent inference of large model all-in-one machine, which adopts efficient data format and compression algorithm, optimizes storage space and read-write speed, reduces data processing time, and provides guarantee for fast inference of large model.

[0115] 3. The present application provides a method and system for realizing resource use and concurrent inference of large model all-in-one machine, which utilizes asynchronous IO and multi-threading technology to realize concurrent inference, improves the concurrent processing capacity and inference efficiency of the model, and can meet the high-concurrency inference demand in diversified scenarios.

[0116] 4. The application provides a method and system for realizing resource use and concurrent reasoning of a large model all-in-one machine, establishing a perfect core monitoring index system and a dynamic adjustment strategy, so that the system can be adjusted in time according to actual operation, and the applicability and reliability of the large model all-in-one machine in different scenarios are enhanced, and the overall performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0117] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the description of the specific embodiments or prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0118] Figure 1 A schematic diagram of steps in the method for realizing efficient use of resources and concurrent reasoning of a large model all-in-one machine is shown.

[0119] Figure 2 A structural block diagram of the system for realizing efficient use of resources and concurrent reasoning of a large model all-in-one machine is shown.

[0120] Figure 3 A schematic diagram of storage optimization processing in the method for realizing efficient use of resources and concurrent reasoning of a large model all-in-one machine is shown.

[0121] Figure 4 A schematic diagram of model concurrent reasoning in the method for realizing efficient use of resources and concurrent reasoning of a large model all-in-one machine is shown. DETAILED DESCRIPTION

[0122] The technical solutions of the present application will be described more clearly and completely by combining the drawings, preferred embodiments of the present application.

[0123] The present application will be further described in conjunction with the drawings and specific embodiments, as shown. Figure 1 According to one embodiment of the present application, a method for realizing efficient use of resources and concurrent reasoning of a large model all-in-one machine is provided, which comprises the following steps:

[0124] In combination with the use scenarios of the large model all-in-one machine, model basic information is collected, a hybrid resource evaluation method is adopted, and all resource requirements of the models are accurately evaluated;

[0125] Specifically, the combination of the use scenarios of the large model all-in-one machine, the collection of model basic information, the adoption of the hybrid resource evaluation method, and the accurate evaluation of all resource requirements of the models comprise the following steps:

[0126] Explicitly use the scene of the large model all-in-one machine, analyze the business process, data characteristics, user request, service quality and other specific requirements under each scene;

[0127] Collect information such as model architecture, algorithm logic, input and output characteristics, and historical running records;

[0128] Construct a hybrid evaluation model based on the rule engine-gradient boosting tree collaborative architecture, and use specific scene requirements, model structure parameters, input and output data characteristics, and historical resource usage data as input features of the model to evaluate the resource requirements of all models;

[0129] As a specific implementation:

[0130] Collect the basic information of each model, as shown in Table 2 below:

[0131] Table 2

[0132]

[0133] Evaluate the resource requirements of the model, as shown in Table 3 below:

[0134] Table 3

[0135]

[0136] According to the evaluation results, develop a differentiated resource allocation strategy, and reasonably plan isolated resources and shared resources in combination with the actual resource situation of the large model all-in-one machine;

[0137] Specifically, the combination of the actual resource situation of the large model all-in-one machine, the differentiated resource allocation strategy according to the evaluation results, and the reasonable planning of isolated resources and shared resources include the following steps:

[0138] Statistical analysis of the hardware resource configuration of the large model all-in-one machine, analysis of the utilization rate and load of the current resources;

[0139] Develop resource allocation principles, including high priority, on-demand allocation, dynamic balance, efficiency first, and other allocation principles;

[0140] Based on the model priority and resource demand evaluation results, determine the specific planning scheme of isolated resources and shared resources;

[0141] As a specific implementation:

[0142] Query the actual hardware configuration of the large model all-in-one machine, which has 192-core CPU, 2015GB video memory, 16 GPU cards, and each GPU card has 96GB video memory. Adopt the resource allocation principle of on-demand allocation, and allocate resources as shown in Table 4:

[0143] Table 4

[0144]

[0145] The evaluation classification data storage uses an adaptive compression framework to intelligently optimize the storage space and read-write speed of the all-in-one machine.

[0146] As shown in Figure 3 , four main branches extend outward from the center node, namely data type, data format, access mode, and life cycle. These four branches constitute the core dimensions of data classification, and the life cycle branch further extends downward into two sub-nodes: compression strategy and hierarchical storage. Compression strategy is further divided into two technical paths: lossless compression and lossy compression, while hierarchical storage includes two levels: high-speed storage layer and ordinary storage layer. The entire diagram presents a clear hierarchical relationship, gradually refining from macro data classification to specific storage and compression strategies, forming a complete data management system framework.

[0147] Specifically, the evaluation classification data storage uses an adaptive compression framework to intelligently optimize the storage space and read-write speed of the all-in-one machine, including the following steps:

[0148] According to data type, data format, access mode, data life cycle management, and other aspects, data evaluation and classification are completed.

[0149] An adaptive compression framework is constructed, integrating multiple efficient data compression algorithms, and designing intelligent selection strategies to meet the data compression needs of different scenarios.

[0150] High-frequency access data and low-frequency access data are identified, real-time data, short-term data, and long-term archival data are distinguished, and a data hierarchical storage architecture is constructed to meet the needs of high-frequency data fast read-write and low-frequency data long-term archiving.

[0151] As a specific implementation:

[0152] Data evaluation and classification, taking the Qwen3-235B-A22 model as an example, the results are shown in Table 5 below:

[0153] Table 5

[0154]

[0155] The data adaptive compression framework integrates multiple data compression algorithms, which can support the compression of text, pictures, voice, and other data. The specific compression algorithms are shown in Table 6 below:

[0156] Table 6

[0157]

[0158] Optimize the model inference engine, complete the model service path analysis and dependency arrangement, and containerize the deployment of stable and reliable model inference services;

[0159] As shown in Figure 4 The flow starts from the left "inference request", the request first enters the "request queue" for buffering processing. After the queue, the system adopts a three-way parallel processing architecture: the upper path passes through the "memory pre-allocation" module, including three sub-functions of fixed memory pool, shared weight cache and modulus check; the middle path passes through the "asynchronous scheduler" for task allocation; the lower path passes through the "request batch processing" module, covering the request accumulation pool, asynchronous data flow allocation and pipeline execution three processing links. The three parallel paths finally converge to the "memory reuse" module, and the whole processing flow ends with "result return" on the right. The arrows in the figure clearly indicate the data flow direction, and the plus symbol represents the convergence point of the parallel processing path, which presents an efficient asynchronous concurrent processing architecture

[0160] Specifically, the optimized model inference engine completes the model service path analysis and dependency arrangement, and containerizes the deployment of stable and reliable model inference services, including the following steps:

[0161] Optimize the memory management of the model inference engine, adopt an asynchronous flow processing mechanism, and improve the efficiency of model concurrent inference;

[0162] Analyze the dependency relationship of the model inference service, determine the starting order of the model service, and use containerized arrangement to deploy the model inference service;

[0163] As a specific implementation:

[0164] Since the services of private model A and private model B both depend on the use of Qwen3-235B-A22 model service, and there is no dependency relationship with Qwen2.5-72B model service, the deployment order of the model is: first, use 0-7 card resources to deploy Qwen3-235B-A22 model service, then use 8-11 card resources to deploy Qwen2.5-72B model service, then use 10 card to deploy private model A, and finally use 11 card to deploy private model B.

[0165] Define core monitoring indicators, develop indicator stratification, collection and processing strategies, and monitor various indicators in the model inference process in real time;

[0166] Specifically, define core monitoring indicators, develop indicator stratification, collection and processing strategies, and monitor various indicators in the model inference process in real time, including the following steps:

[0167] Define the core monitoring indicator system, including the system resource indicator set, the model inference performance indicator set, and the business quality indicator set;

[0168] A three-level hierarchical collection strategy is constructed, a differentiated collection frequency is configured, and TSDB time series database is used to compress and store index data.

[0169] As a specific embodiment:

[0170] The core monitoring index system and the differentiated collection frequency are shown in Table 7 as follows:

[0171] Table 7

[0172]

[0173] A multi-dimensional index evaluation system is constructed, a baseline comparison and weighted analysis comprehensive scoring model are used, and index evaluation is performed on the model service.

[0174] Specifically, a multi-dimensional index evaluation system is constructed, a baseline comparison and weighted analysis comprehensive scoring model are used, and index evaluation is performed on the model service, including the following steps:

[0175] Define evaluation dimensions and weights, set model index classification standards, and construct a multi-dimensional index evaluation system.

[0176] Establish a baseline standard, use a weighted analysis method to calculate the comprehensive score of multiple dimensions, and map to the corresponding grade interval.

[0177] As a specific embodiment:

[0178] Define evaluation dimensions and weights, as shown in Table 8 as follows:

[0179] Table 8

[0180]

[0181] Define the score weight and grade interval of the index, as shown in Table 9 as follows:

[0182] Table 9

[0183]

[0184] Referring to the model index evaluation results, according to the model priority and resource demand difference, the dynamic optimization of model resources and parameters is completed.

[0185] Specifically, referring to the model index evaluation results, according to the model priority and resource demand difference, the dynamic optimization of model resources and parameters is completed, including the following steps:

[0186] Intuitively analyze the system resource index score to determine unreasonable resource allocation and use;

[0187] According to the service index comprehensive score, further identify low score dimensions, adopt correlation analysis method, find the correlation of model service and resource use, model parameter, determine the optimization entry point;

[0188] Adopt vertical expansion, horizontal expansion and other technologies to dynamically adjust resource use, and optimize model service in time;

[0189] As a specific embodiment:

[0190] For large model inference scenarios, the most important thing to focus on is the related indicators of inference performance. When the indicator data of inference performance is found to be abnormal, the correlation analysis is used to learn the related system resources or model parameters, and then the optimization entry point is determined. The algorithm of correlation analysis is as follows:

[0191] The algorithm of correlation analysis is as follows:

[0192]

[0193] Wherein and are the observation values of two indicators, is the mean of , is the mean of , and the value of r is between [-1, 1]. When |r|≥0.7, it is a strong correlation, and the related indicators are adjusted first.

[0194] Based on the correlation analysis method, it can be known that the correlation between GPU memory utilization and inference delay is large.

[0195] By observing the indicator data and analyzing the comprehensive score, it is found that when the system resources execute the inference task, the resource utilization rate will jump obviously, and the remaining time will be stably maintained in the normal interval. The response of model inference request is fast, which meets the business SLA requirement, and the throughput is high, which reflects the high performance of the system. The accuracy rate and success rate of inference result are high, which meets the business use.

[0196] According to another embodiment of the application, as shown in Figure 2 , a large model all-in-one machine resource efficient use and concurrent inference method is provided, which comprises: a resource demand evaluation module, a resource management and allocation module, a storage optimization processing module, a concurrent inference module and a performance monitoring module.

[0197] The resource demand evaluation module is used to combine the actual application scene of the large model all-in-one machine, accurately quantize the hardware resources and calculation demand required by the model running through the multi-dimensional data collection and hybrid evaluation method, and provide a scientific basis for resource allocation.

[0198] The resource management and allocation module is configured to realize dynamic scheduling, elastic allocation and efficient reuse of computing power resources according to the output result of the resource demand evaluation module and in combination with the real-time resource state of the large model all-in-one machine, so as to ensure the performance and stability during concurrent inference of multiple models.

[0199] The storage optimization processing module is configured to realize minimization of storage space occupation and maximization of access efficiency by means of intelligent compression, hierarchical storage, efficient read-write optimization and other technologies in combination with the storage data characteristics of the large model all-in-one machine, so as to support the storage performance requirements of concurrent inference of multiple models.

[0200] The concurrent inference module is configured to realize efficient parallel processing of multiple models and multiple requests on the large model all-in-one machine, so as to improve system throughput and resource utilization by means of request scheduling, model parallelism, batch processing optimization and other technologies, while ensuring that the inference service meets the business SLA requirements.

[0201] The performance monitoring module is configured to build a comprehensive monitoring system for the large model all-in-one machine, so as to reasonably evaluate various indexes of the model by means of multi-dimensional data acquisition and real-time analysis, and provide data support for resource adjustment and model optimization decisions.

[0202] To sum up, by means of the technical solutions of the present application, the present application can accurately evaluate model resource requirements, combine resource isolation restriction and sharing reuse technology, effectively solve the problem of unreasonable resource allocation, improve the resource utilization rate of the large model all-in-one machine, and reduce the operating cost. The present application adopts efficient data formats and compression algorithms, optimizes storage space and read-write speed, reduces data processing time, and provides a guarantee for fast inference of large models. The present application uses asynchronous IO and multi-threading technology to realize concurrent inference, improves the concurrent processing capacity and inference efficiency of the model, and can meet the high-concurrency inference requirements in diversified scenarios. The present application establishes a perfect core monitoring index system and dynamic adjustment strategy, so that the system can be adjusted in time according to the actual running situation, enhances the applicability and reliability of the large model all-in-one machine in different scenarios, and improves the overall performance.

[0203] The above specific embodiments only describe the preferred embodiments of the present application, and do not limit the protection scope of the present application. Various modifications, substitutions and improvements of the technical solutions of the present application made by those skilled in the art according to the description and drawings provided by the present application, without departing from the design concept and spirit of the present application, should all belong to the protection scope of the present application. The protection scope of the present application is determined by the claims.

Claims

1. A method for implementing resource utilization and concurrent inference in a large-scale integrated model, characterized in that, Includes the following steps: Step S1: Based on the usage scenarios of the large model all-in-one machine, collect basic model information and use a hybrid resource evaluation method to accurately evaluate the resource requirements of all models; Step S2: Based on the actual resource situation of the large-scale integrated machine and the evaluation results, formulate a differentiated resource allocation strategy and plan for isolated and shared resources; Step S3: Evaluate the categorized data storage and use an adaptive compression framework to intelligently optimize the all-in-one machine's storage space and read / write speed; Step S4: Optimize the model inference engine, complete model service path resolution and dependency orchestration, and containerize and deploy stable and reliable model inference services; Step S5: Define core monitoring metrics, formulate strategies for metric stratification, collection, and processing, and monitor various metrics in real time during the model inference process; Step S6: Construct a multi-dimensional indicator evaluation system and use a comprehensive scoring model of baseline comparison and weighted analysis to evaluate the model service. Step S7: Refer to the model indicator evaluation results and, based on the differences in model priority and resource requirements, perform dynamic optimization of model resources and parameters.

2. The method for implementing resource utilization and concurrent inference of a large-scale integrated machine according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Define the application scenarios for the large-scale all-in-one machine and analyze the specific requirements under each scenario; Step S12: Collect information on model architecture, algorithm logic, input / output characteristics, and historical execution records; Step S13: Construct a hybrid evaluation model based on a rule engine-gradient boosting tree collaborative architecture, using specific scenario requirements, model structure parameters, input and output data characteristics, and historical resource usage data as input features of the model, and evaluate the resource requirements of all models. The core algorithms of the rule engine include: The GPU requirement is dynamically calculated based on the number of model parameters: when the number of parameters is ≥100 billion, the number of GPUs = max(1, int(number of parameters / 200)), with a maximum limit of 16 cards; Adjust resource requirements based on input type: GPU count ×1.5 for image scenes, ×1.2 for audio scenes; Adjust according to QoS constraints: When latency requirement is <100ms, multiply the number of GPUs by 2; CPU core count = max(8, int(total computing power / 10)), memory = max(32GB, int(GPU memory × 1.5)); The gradient boosting tree is implemented using LightGBM, with the objective function being the mean squared error and the training objective being a resource demand prediction error ≤15%.

3. The method for implementing resource utilization and concurrent inference of a large-scale integrated model as described in claim 1, characterized in that, Step S2 specifically includes: Step S21: Statistically analyze the hardware resource configuration of the large-scale integrated machine, and analyze the current utilization rate and load of each resource; Step S22: Formulate resource allocation principles, including high priority, on-demand allocation, dynamic balancing, and efficiency priority. The specific methods for handling resource competition are as follows: Priority preemption: High-priority models are configured with preemption. policy =Always can preempt CPU time slices from low-priority models; Quota management: Limit namespace resources through ResourceQuota; Step S23: Based on the model priority and resource demand assessment results, determine specific planning schemes for isolated and shared resources, including: Configure GPU memory hard isolation via nvidia-container-cli; Set CPU quotas using cgroup v2; Adjust memory OOM priority; Implement multiple resource isolation and sharing mechanisms.

4. The method for implementing resource utilization and concurrent inference of a large-scale integrated model as described in claim 1, characterized in that, Step S3 specifically includes: Step S31: Complete the evaluation and classification of data based on multiple aspects such as data type, data format, access mode, and data lifecycle management; Step S32: Construct an adaptive compression framework, integrate multiple efficient data compression algorithms, and design an intelligent selection strategy to meet the data compression needs of different scenarios; Step S33: Identify high-frequency access data and low-frequency access data, distinguish between real-time data, short-term data and long-term archived data, and construct a hierarchical data storage architecture to meet the needs of fast reading and writing of high-frequency data and long-term archiving of low-frequency data.

5. The method for implementing resource utilization and concurrent inference of a large-scale integrated model as described in claim 1, characterized in that, Step S4 specifically includes: Step S41: Optimize the memory management of the model inference engine by adopting an asynchronous stream processing mechanism to improve the efficiency of concurrent model inference. Key steps involved include: Use aiohttp to handle network requests and asyncio to manage coroutines, thus solving the performance bottleneck of traditional synchronous I / O and improving request throughput. The reasoning process is divided into four stages: word segmentation, feature extraction, model reasoning, and post-processing. Each stage is executed in an independent GPU stream to reduce latency. Step S42: Analyze the dependencies of the model inference service, determine the startup order of the model service, and deploy the model inference service using containerized orchestration.

6. The method for implementing resource utilization and concurrent inference of a large-scale integrated machine according to claim 1, characterized in that, Step S5 specifically includes: Step S51: Define the core monitoring indicator system, including the system resource indicator set, the model inference performance indicator set, and the business quality indicator set; Step S52: Construct a three-level hierarchical data acquisition strategy, configure differentiated acquisition frequencies, and use the TSDB time-series database to compress and store indicator data, wherein: Level 1 data collection: Real-time key indicators, collected every 5 seconds; Level 2 data acquisition: System performance indicators, acquisition frequency 10-30 seconds; Level 3 data collection: Business statistical indicators, collected every 60 seconds.

7. The method for implementing resource utilization and concurrent inference of a large-scale integrated machine according to claim 1, characterized in that, Step S6 specifically includes: Step S61: Define evaluation dimensions and weights, set model indicator grading standards, and construct a multi-dimensional indicator evaluation system; Step S62: Establish baseline standards, use weighted analysis to calculate the comprehensive score of multiple dimensions, and map it to the corresponding grade intervals.

8. The method for implementing resource utilization and concurrent inference of a large-scale integrated machine according to claim 1, characterized in that, Step S7 specifically includes: Step S71: Visually analyze the system resource index scores to identify unreasonable resource allocation and usage. Step S72: Based on the comprehensive score of service indicators, further identify low-scoring dimensions, use correlation analysis to discover the correlation between model service and resource usage and model parameters, and determine the optimization entry point; The algorithm for correlation analysis is as follows: ； in, and These are the values ​​of the two indicators at the i-th observation, where n is the number of observations. yes The mean, yes The mean of r is between [-1, 1]. When |r| ≥ 0.7, the correlation is strong, and the correlation index should be adjusted first. Step S73: Dynamically adjust resource usage using vertical and horizontal scaling techniques to optimize model services in a timely manner, specifically including: CPU resource adjustment: If the utilization rate is greater than 80% or less than 20% for 10 consecutive minutes, then 2 cores will be added or removed each time, with a maximum adjustment of 50% of the current configuration; GPU memory adjustment: If the memory utilization is greater than 90% and the latency is greater than the threshold, the memory will be expanded in 2GB increments, up to a maximum of 90% of the physical memory; if the memory utilization is less than 30% for one hour and the requested latency meets the SLA, the memory will be reduced in 2GB increments. Horizontal scaling: Based on the HPA mechanism, when the QPS is greater than the threshold and lasts for 10 minutes, the number of inference container replicas is automatically increased, with a scaling step of 50% of the current number of replicas; when the QPS is less than the threshold and lasts for 1 hour, the number of inference container replicas is automatically reduced by 1, but at least 1 inference container is retained.

9. A system for achieving efficient resource utilization and concurrent inference of a large-scale model all-in-one machine, used to implement the method for achieving resource utilization and concurrent inference of a large-scale model all-in-one machine as described in any one of claims 1-8, characterized in that, The system includes: Resource Requirements Assessment Module: This module is used to accurately quantify the hardware resources and computing requirements required for model operation by combining the actual application scenarios of large-scale integrated machine with multi-dimensional data collection and hybrid assessment methods, so as to provide a scientific basis for resource allocation. Resource Management and Allocation Module: Based on the output of the resource demand assessment module and the real-time resource status of the large model integrated machine, this module enables dynamic scheduling, elastic allocation, and efficient reuse of computing resources, ensuring performance and stability during concurrent inference of multiple models. Storage optimization module: Designed to address the storage data characteristics of large model all-in-one machines, it minimizes storage space usage and maximizes access efficiency through technologies such as intelligent compression, hierarchical storage, and efficient read / write optimization, supporting the storage performance requirements of concurrent inference for multiple models; Concurrent Inference Module: Used to achieve efficient parallel processing of multiple models and multiple requests on the large model appliance. It improves system throughput and resource utilization through request scheduling, model parallelism and batch processing optimization techniques, while ensuring that the inference service meets the business SLA requirements. Performance monitoring module: Used to build a comprehensive monitoring system for large-scale integrated models. Through multi-dimensional data collection and real-time analysis, it reasonably evaluates various indicators of the model and provides data support for resource adjustment and model optimization decisions.

10. The system according to claim 9, characterized in that, The resource demand assessment module includes: The rules engine unit has a built-in GPU configuration mapping table and resource calculation rules, which dynamically calculate the basic resource requirements based on the number of model parameters, input type, and QoS requirements. Gradient boosting tree unit: The LightGBM algorithm is used to train the prediction model based on historical resource usage data, and the output of the rule engine is corrected to ensure that the prediction error is controlled within 15%. The resource management and allocation module includes: Priority Manager: Maintains a model priority queue, supports preemption_policy configuration, and enables priority-based resource preemption; Quota controller: It uses cgroup v2 and namespace technology, combined with the ResourceQuota object, to enforce resource quotas. Shared Pool Manager: Employs a time-division multiplexing algorithm to manage the dynamic allocation of shared resource pools; Isolation Controller: Implements hardware isolation of GPU memory through nvidia-container-cli, and CPU core binding through CPU affinity settings; The performance monitoring module includes: Data collector: Implements three-level hierarchical data collection, including real-time collector, performance collector, and business collector; Time-series database: Uses TSDB technology to store monitoring data, and supports data compression and multi-level retention strategies; Indicator evaluation engine: Implements multi-dimensional weighted scoring algorithms and correlation analysis algorithms to generate comprehensive evaluation reports; Dynamic optimization actuator: Based on evaluation results and correlation analysis, automatically execute vertical and horizontal expansion strategies.

Images